dmalloc directly calls into $(LD) to generate a shared library our of
the static one.
To detect what command it should run, ./configure tries various
incantations of ld with various command line options until one does not
fail. One of those is (basically):
$(LD) --whole-archive -o contest.o.t contest.a
This makes ./configure conclude what the command to link a shared
library in the Makefile should be, and thus stores that in a variable:
shlinkargs='$(LD) --whole-archive -o $@'
... which is then AC_SUBST()ed into Makefile.in with a rule like:
$(SHLIB): $(LIBRARY)
@shlinkargs@ $(LIRARY)
which once substiuted, gives:
$(SHLIB): $(LIBRARY)
$(LD) --whole-archive -o $@ $(LIRARY)
However, when SSP is enabled, the __stack_chk_fail_local and co symbols
are provided by additional libraries or object files, and that is the
responsibility of gcc to pass those when linking. But as dmalloc
directly calls ld, it misses those.
Changing dmalloc to use $(CC) is not trivial, however.
First, we can't pass LD=$(TARGET_CC), otherwise the whole package
explodes [0]: indeed --whole-archive is unknown to gcc, so it must be
passed as -Wl,--whole archive instead. So we'd need to add a new test
that uses $(CC), like so:
$(CC) -Wl,--whole-archive -o contest.o.t contest.a
However, in that case, gcc does pass additional libs/objs (like, for
eample, the SSP ones) to the linker. But then those are also included
in the whole-archive section. This causes the linker to add all the
symbols form those libs/objs, even those not needed for SSP; on some
archs, like PPC, that may require floating point symbols (__muldiv3 et
al.) which are in another library, and thus the linker can't find them.
The proper solution wouild be to add -Wl,--no-whole-archive, but that
would have to be added _after_ the library we want to link, i.e.e we
should be able to evntually run:
$(CC) -Wl,--whole-archive -o $@ $(LIRARY) -Wl,--no-whole-archive
That would require that we introduce a new variable that is added
_after_ the $(LIBRARY), e.g. @shlinkargs_post@ or so...
This is a bigger endeavour than we want to pursue...
Since dmalloc is a debugging utility, it is not supposed to go into
production builds, and during debugging, it would not be surprising that
it needs to poke around arrays to debug them.
So, we go the easier route: disable SSP altogether.
[0] with lots of nice colors, but that's not the point, is it?
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
3481674ee3