kumquat-buildroot/package/dmalloc/dmalloc.mk
Yann E. MORIN 3481674ee3 package/dmalloc: don't use SSP
dmalloc directly calls into $(LD) to generate a shared library our of
the static one.

To detect what command it should run, ./configure tries various
incantations of ld with various command line options until one does not
fail. One of those is (basically):
    $(LD) --whole-archive -o contest.o.t contest.a

This makes ./configure conclude what the command to link a shared
library in the Makefile should be, and thus stores that in a variable:
    shlinkargs='$(LD) --whole-archive -o $@'

... which is then AC_SUBST()ed into Makefile.in with a rule like:

    $(SHLIB): $(LIBRARY)
        @shlinkargs@ $(LIRARY)

which once substiuted, gives:

    $(SHLIB): $(LIBRARY)
        $(LD) --whole-archive -o $@ $(LIRARY)

However, when SSP is enabled, the __stack_chk_fail_local and co symbols
are provided by additional libraries or object files, and that is the
responsibility of gcc to pass those when linking. But as dmalloc
directly calls ld, it misses those.

Changing dmalloc to use $(CC) is not trivial, however.

First, we can't pass LD=$(TARGET_CC), otherwise the whole package
explodes [0]: indeed --whole-archive is unknown to gcc, so it must be
passed as -Wl,--whole archive instead. So we'd need to add a new test
that uses $(CC), like so:
    $(CC) -Wl,--whole-archive -o contest.o.t contest.a

However, in that case, gcc does pass additional libs/objs (like, for
eample, the SSP ones) to the linker. But then those are also included
in the whole-archive section. This causes the linker to add all the
symbols form those libs/objs, even those not needed for SSP; on some
archs, like PPC, that may require floating point symbols (__muldiv3 et
al.) which are in another library, and thus the linker can't find them.

The proper solution wouild be to add -Wl,--no-whole-archive, but that
would have to be added _after_ the library we want to link, i.e.e we
should be able to evntually run:

    $(CC) -Wl,--whole-archive -o $@ $(LIRARY) -Wl,--no-whole-archive

That would require that we introduce a new variable that is added
_after_ the $(LIBRARY), e.g. @shlinkargs_post@ or so...

This is a bigger endeavour than we want to pursue...

Since dmalloc is a debugging utility, it is not supposed to go into
production builds, and during debugging, it would not be surprising that
it needs to poke around arrays to debug them.

So, we go the easier route: disable SSP altogether.

[0] with lots of nice colors, but that's not the point, is it?

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-02-15 22:46:35 +01:00

82 lines
2.2 KiB
Makefile

################################################################################
#
# dmalloc
#
################################################################################
DMALLOC_VERSION = 5.6.5
DMALLOC_SOURCE = dmalloc-$(DMALLOC_VERSION).tgz
DMALLOC_SITE = http://dmalloc.com/releases
DMALLOC_LICENSE = ISC
DMALLOC_LICENSE_FILES = LICENSE.txt
DMALLOC_INSTALL_STAGING = YES
DMALLOC_CFLAGS = $(TARGET_CFLAGS)
# dmalloc uses $(LD) to link, and thus misses the object files or libs that
# are needed to provide the __stack_chk_fail_local and co. symbols. Changing
# to use $(CC) is really more complex that we'd like. Since dmalloc is
# involved in debugging memory allocation, it is not expected to be a
# production library, so we do not care that much that it has SSP.
DMALLOC_CFLAGS += -fno-stack-protector
ifeq ($(BR2_STATIC_LIBS),y)
DMALLOC_CONF_OPTS += --disable-shlib
else
DMALLOC_CONF_OPTS += --enable-shlib
DMALLOC_CFLAGS += -fPIC
endif
ifeq ($(BR2_INSTALL_LIBSTDCPP),y)
DMALLOC_CONF_OPTS += --enable-cxx
else
DMALLOC_CONF_OPTS += --disable-cxx
endif
ifeq ($(BR2_TOOLCHAIN_HAS_THREADS),y)
DMALLOC_CONF_OPTS += --enable-threads
else
DMALLOC_CONF_OPTS += --disable-threads
endif
# dmalloc has some assembly function that are not present in thumb1 mode:
# Error: lo register required -- `str lr,[sp,#4]'
# so, we desactivate thumb mode
ifeq ($(BR2_ARM_INSTRUCTIONS_THUMB),y)
DMALLOC_CFLAGS += -marm
endif
ifeq ($(BR2_TOOLCHAIN_HAS_GCC_BUG_63261),y)
DMALLOC_CFLAGS += -O0
endif
DMALLOC_CONF_ENV = \
CFLAGS="$(DMALLOC_CFLAGS)" \
ac_cv_page_size=12 \
ac_cv_strdup_macro=yes \
ac_cv_strndup_macro=yes
# both DESTDIR and PREFIX are ignored..
define DMALLOC_INSTALL_STAGING_CMDS
$(TARGET_MAKE_ENV) $(MAKE) includedir="$(STAGING_DIR)/usr/include" \
bindir="$(STAGING_DIR)/usr/bin" \
libdir="$(STAGING_DIR)/usr/lib" \
shlibdir="$(STAGING_DIR)/usr/lib" \
infodir="$(STAGING_DIR)/usr/share/info/" \
-C $(@D) install
endef
ifeq ($(BR2_STATIC_LIBS),)
define DMALLOC_INSTALL_SHARED_LIB
cp -dpf $(STAGING_DIR)/usr/lib/libdmalloc*.so $(TARGET_DIR)/usr/lib
endef
endif
define DMALLOC_INSTALL_TARGET_CMDS
$(DMALLOC_INSTALL_SHARED_LIB)
cp -dpf $(STAGING_DIR)/usr/bin/dmalloc $(TARGET_DIR)/usr/bin/dmalloc
endef
$(eval $(autotools-package))