[powerpc] crash trying to allocate memory in inferior
Our testsuite noticed a crash when trying to call a function which
requires GDB to allocate memory in the inferior. Typically, this
happens when one of the parameters is a string. For instance, our
testcase tries:
(gdb) call debug.trace (me, "You")
[1] 32737 segmentation fault /path/to/gdb
What happens is that GDB sees the string, and thus tries to allocate
memory for it in the inferior:
> /* Allocate NBYTES of space in the inferior using the inferior's
> malloc and return a value that is a pointer to the allocated
> space. */
>
> struct value *
> value_allocate_space_in_inferior (int len)
> {
> struct objfile *objf;
> struct value *val = find_function_in_inferior ("malloc", &objf);
And find_function_in_inferior first searches the symtab in case
we have debug info. But, in our case (bareboard powerpc), we don't,
so it gets "malloc"'s address from the minimal symbols, and builds
a value whose type is a TYPE_CODE_PTR, not a TYPE_CODE_FUNC.
As a result, when we later try to make the call to malloc, we end up
inside the powerpc tdep code that has:
> do_ppc_sysv_return_value (struct gdbarch *gdbarch, struct type *func_type,
[...]
> if (func_type
> && TYPE_CALLING_CONVENTION (func_type) == DW_CC_GDB_IBM_OpenCL)
The problem is that func_type is not a TYPE_CODE_FUNC, and thus
the type-specific kind is not TYPE_SPECIFIC_FUNC, and so we do
TYPE_CALLING_CONVENTION is an invalid access.
Interestingly, the other call to TYPE_CALLING_CONVENTION is correctly
preceded by a check of the type's TYPE_CODE (making sure that it is
TYPE_CODE_FUNC).
gdb/ChangeLog:
* ppc-sysv-tdep.c (do_ppc_sysv_return_value): Do not check
FUNC_TYPE's calling convention if FUNC_TYPE is not a function.