[gdb/tdep] Fix avx512 -m32 support in gdbserver
PR27257 reports a problem that can be reproduced as follows:
- use x86_64 machine with avx512 support
- compile a hello world with -m32 to a.out
- start a gdbserver session with a.out
- use gdb to connect to the gdbserver session
This makes us run into:
...
Listening on port 2346
Remote debugging from host ::1, port 34940
src/gdbserver/regcache.cc:257: \
A problem internal to GDBserver has been detected.
Unknown register zmm16h requested
...
The problem is that i387_xsave_to_cache in gdbserver/i387-fp.cc can't find a
register zmm16h in the register cache.
To understand how this happens, first some background.
SSE has 16 128-bit wide xmm registers.
AVX extends the SSE registers set as follows:
- it extends the 16 existing 128-bit wide xmm registers to 256-bit wide ymm
registers.
AVX512 extends the AVX register set as follows:
- it extends the 16 existing 256-bit wide ymm registers to 512-bit wide zmm
registers.
- it adds 16 additional 512-bit wide zmm registers (with corresponding ymm and
xmm subregisters added as well)
However, in 32-bit mode, there are only 8 xmm/ymm/zmm registers.
The problem we're running into is that gdbserver/i387-fp.cc uses these
constants to describe the size of the register file:
...
static const int num_avx512_zmmh_low_registers = 16;
static const int num_avx512_zmmh_high_registers = 16;
static const int num_avx512_ymmh_registers = 16;
static const int num_avx512_xmm_registers = 16;
...
which are all incorrect for the 32-bit case.
Fix this by replacing the constants with variables that have the appropriate
values in 64-bit and 32-bit mode.
Tested on x86_64-linux with native and unix/-m32.