When nodejs is build, a qemu wrapper script is used to execute some
programs built for the target in user-mode emulation. However, when the
target and build machines are similar (e.g. x86_74), running those
programs fails, with errors such as:
cd ../../tools/v8_gypfiles; python ../../deps/v8/tools/run.py ../../out/Release/v8-qemu-wrapper ../../out/Release/bytecode_builtins_list_generator ../../out/Release/obj.host/gen/generate-bytecode-output-root/builtins-generated/bytecodes-builtins-list.h
../../out/Release/bytecode_builtins_list_generator: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by ../../out/Release/bytecode_builtins_list_generator)
../../out/Release/bytecode_builtins_list_generator: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by ../../out/Release/bytecode_builtins_list_generator)
../../out/Release/bytecode_builtins_list_generator: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by ../../out/Release/bytecode_builtins_list_generator)
../../out/Release/bytecode_builtins_list_generator: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by ../../out/Release/bytecode_builtins_list_generator)
Return code is 1
So the question is: why the heck does Qemu use the host C library?
To answer this question, we first have to look at how the -L option of
Qemu is implemented. This option is documented as such:
-L path QEMU_LD_PREFIX set the elf interpreter prefix to 'path'
The v8-qemu-wrapper script makes this option point to $(STAGING_DIR),
so that the ELF interpreter used is the one in $(STAGING_DIR).
However, contrary to what the option documentation says, this option
does much more than setting the ELF interpreter prefix: it is going to
affect how *all* system calls manipulating files (open, etc.) are
going to work.
When this option is passed, the function init_paths() in
https://git.qemu.org/?p=qemu.git;a=blob;f=util/path.c is called at
initialization time, and essentially its sets the global "base"
variable to point to the directory passed as -L argument.
Then, for every single syscall that manipulates a path, this path will
be passed through the path() function in the same file. This function
will first attempt to resolve the path with "base" as a prefix, and if
not, return the unprefixed path.
After adding some traces into this function, I was able to understand
what happens:
(1) -L$(STAGING_DIR) is passed, causing "base" to point to
$(STAGING_DIR)
(2) The target ELF interpreter from $(STAGING_DIR) is properly invoked
(3) When this ELF interpreter then resolves the libc.so.6 library, it
first looks for /etc/ld.so.cache.
(4) Qemu first looks for /etc/ld.so.cache with the -L prefix, i.e
$(STAGING_DIR)/etc/ld.so.cache, but it does not exist. So, the Qemu
system call emulation falls back to /etc/ld.so.cache, which means
the target ELF interpreter reads the /etc/ld.so.cache of the host
system.
(5) This /etc/ld.so.cache of the host system says that libc.so.6 is in
/lib/x86_64-linux-gnu/
(6) The target ELF interpreter therefore tries to use
/lib/x86_64-linux-gnu/libc.so.6. The Qemu system call emulation
first tries $(STAGING_DIR)/lib/x86_64-linux-gnu/libc.so.6, but
this library does not exist (it is in
$(STAGING_DIR)/lib/libc.so.6), so the Qemu system call emulation
falls back to /lib/x86_64-linux-gnu/libc.so.6 of the host system,
which exist... but is too old compared to the target C library.
Indeed, results from ld.so.cache take precedence over the simple
resolution of library paths in /usr/lib and /lib.
We see 3 possible ideas to resolve this problem:
(A) Change the behavior of Qemu to not fallback to unprefixed paths:
when -L is passed, all path-related system calls should see the
paths prefixed by the -L option.
Issue with this is that this change is unlikely to get accepted by
Qemu upstream. And there might be some side effects we have not
really identified.
(B) Create an empty $(STAGING_DIR)/etc/ld.so.cache. We have tested
this solution and it works: it gets used instead of the host
/etc/ld.so.cache. Because $(STAGING_DIR)/etc/ld.so.cache is empty,
there's no libc.so.6 match, so the target ELF interpreter goes
through its normal library location resolution logic, which falls
back to trying in /usr/lib and /lib, which works as those paths
ends up being prefixed with $(STAGING_DIR) by Qemu.
(C) Pass LD_LIBRARY_PATH pointing to $(STAGING_DIR)/lib and
$(STAGING_DIR)/usr/lib in the Qemu wrapper. This works because
LD_LIBRARY_PATH paths have precedence over paths given by
ld.so.cache.
This is the solution already used by the GOI qemu wrapper in
package/gobject-introspection/g-ir-scanner-qemuwrapper.in.
We chose to go with the third option, because it has been proven to work
for the GOI wrapper, and has been reported to solve #14366. Even though
the first option would be the best, it is also the one that has the
least chances to land any time soon (if ever); the second has not been
exercised, and the impact is not fully understood either (e.g what about
non-glibc toolchains?).
Fixes: #14366
Signed-off-by: Jens Maus <mail@jens-maus.de>
[yann.morin.1998@free.fr:
- add whole analsys done by Thomas in:
https://lore.kernel.org/buildroot/20221031213926.50d3c778@windsurf/
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
278d1db56b