libopcodes: extend the styling within the i386 disassembler
The i386 disassembler is pretty complex. Most disassembly is done
indirectly; operands are built into buffers within a struct instr_info
instance, before finally being printed later in the disassembly
process.
Sometimes the operand buffers are built in a different order to the
order in which they will eventually be printed.
Each operand can contain multiple components, e.g. multiple registers,
immediates, other textual elements (commas, brackets, etc).
When looking for how to apply styling I guess the ideal solution would
be to move away from the operands being a single string that is built
up, and instead have each operand be a list of "parts", where each
part is some text and a style. Then, when we eventually print the
operand we would loop over the parts and print each part with the
correct style.
But it feels like a huge amount of work to move from where we are
now to that potentially ideal solution. Plus, the above solution
would be pretty complex.
So, instead I propose a .... different solution here, one that works
with the existing infrastructure.
As each operand is built up, piece be piece, we pass through style
information. This style information is then encoded into the operand
buffer (see below for details). After this the code can continue to
operate as it does right now in order to manage the set of operand
buffers.
Then, as each operand is printed we can split the operand buffer into
chunks at the style marker boundaries, with each chunk being printed
with the correct style.
For encoding the style information I use a single character, currently
\002, followed by the style encoded as a single hex digit, followed
again by the \002 character.
This of course relies on there not being more than 16 styles, but that
is currently true, and hopefully will remain true for the foreseeable
future.
The other major concern that has arisen around this work is whether
the escape character could ever be encountered in output naturally
generated by the disassembler. If this did happen then the escape
characters would be stripped from the output, and the wrong styling
would be applied.
However, I don't believe that this is currently a problem.
Disassembler content comes from a number of sources. First there's
content that copied directly from the i386-dis.c file, this is things
like register names, and other syntax elements (brackets, commas,
etc). We can easily check that the i386-dis.c file doesn't contain
our special character.
The next source of content are immediate operands. The text for these
operands is generated by calls into libc. By selecting a
non-printable character we can be confident that this is not something
that libc will generate as part of an immediate representation.
The other output that appears to be from the disassembler is operands
that contain addresses and (possibly) symbol names. It is quite
possible that a symbol name might contain any special character we
could imagine, so is this a problem?
I don't think it is, we don't actually print address and symbol
operands through the disassembler, instead, the disassembler calls
back to the user (objdump, gdb, etc) to print the address and symbol
on its behalf. This content is printed directly to the output stream,
it does not pass through the i386 disassembler output buffers. As a
result, we never check this particular output for styling escape
characters.
In some (not very scientific) benchmarking on my machine,
disassembling a reasonably large (142M) shared library, I'm not seeing
any significant slow down in disassembler speed with this change.
Most instructions are now being fully syntax highlighted when I
disassemble using the --disassembler-color=extended-color option. I'm
sure that there are probably still a few corner cases that need fixing
up, but we can come back to them later I think.
When disassembler syntax highlighting is not being used, then there
should be no user visible changes after this commit.