gdb: improve disassembler styling when Pygments raises an exception
While working on another issue relating to GDB's use of the Python
Pygments package for disassembly styling I noticed an issue in the
case where the Pygments package raises an exception.
The intention of the current code is that, should the Pygments package
raise an exception, GDB will disable future attempts to call into the
Pygments code. This was intended to prevent repeated errors during
disassembly if, for some reason, the Pygments code isn't working.
Since the Pygments based styling was added, GDB now supports
disassembly styling using libopcodes, but this is only available for
some architectures. For architectures not covered by libopcodes
Pygments is still the only option.
What I observed is that, if I disable the libopcodes styling, then
setup GDB so that the Pygments based styling code will indicate an
error (by returning None), GDB does, as expected, stop using the
Pygments based styling. However, the libopcodes based styling will
instead be used, despite this feature having been disabled.
The problem is that the disassembler output is produced into a
string_file buffer. When we are using Pygments, this buffer is
created without styling support. However, when Pygments fails, we
recreate the buffer with styling support.
The problem is that we should only recreate the buffer with styling
support only if libopcodes styling is enabled. This was an oversight
in commit:
commit
4cbe4ca5da5cd7e1e6331ce11f024bf3c07b9744
Date: Mon Feb 14 14:40:52 2022 +0000
gdb: add support for disassembler styling using libopcodes
This commit:
1. Factors out some of the condition checking logic into two new
helper functions use_ext_lang_for_styling and
use_libopcodes_for_styling,
2. Reorders gdb_disassembler::m_buffer and gdb_disassembler::m_dest,
this allows these fields to be initialised m_dest first, which means
that the new condition checking functions can rely on m_dest being
set, even when called from the gdb_disassembler constructor,
3. Make use of the new condition checking functions each time
m_buffer is initialised,
4. Add a new test that forces the Python disassembler styling
function to return None, this will cause GDB to disable use of
Pygments for styling, and
5. When we reinitialise m_buffer, and re-disassemble the
instruction, call reset the in-comment flag. If the instruction
being disassembler ends in a comment then the first disassembly pass
will have set the in-comment flag to true. This shouldn't be a
problem as we will only be using Pygments, and thus performing a
re-disassembly pass, if libopcodes is disabled, so the in-comment
flag will never be checked, even if it is set incorrectly. However,
I think that having the flag set correctly is a good thing, even if
we don't check it (you never know what future uses might come up).