Which Library Did You Say Will Be Loaded?

"Fun" with otool -L on OS X

Aug 6, 2021

Introduction

This is just a short blog on a “gotcha” that initially caught me while developing dynamic library code on OS X.

The Problem

Suppose that you are developing a dynamic library which can act as a runtime replacement for an existing runtime library (e.g. LOMP as a, currently incomplete and risky, replacement for the CLANG or GCC OpenMP runtime).

You want to have an existing executable which has been dynamically linked against the compiler’s runtime use your runtime instead.

For instance, you want to run benchmarks such as the EPCC OpenMP micro-benchmarks with the same executable but testing the performance of each runtime library.

Since, as a software developer, you are, justifiably paranoid

, you also want to be sure that the code really is using your library, not the system one you are replacing.

Linux Solution

On Linux getting the code to use your library can be achieved by putting the directory in which your library (which has the same name as the compiler’s) can be found into the LD_LIBRARY_PATH envirable. The dynamic linker searches for libraries in the directories in $LD_LIBRARY_PATH before looking in the directories built into the image, so will find yours and stop searching.

Checking what is going on can be achieved by using ldd.

Here you can see me compiling a trivial OpenMP code (using one of the system compilers), running it, checking which OpenMP runtime library it is using, then using LD_LIBRARY_PATH to use the LOMP runtime instead (and out of real paranoia, checking that by asking LOMP to announce itself)

$ armclang -fopenmp hello_omp.c
$ OMP_NUM_THREADS=2 ./a.out
Hello from thread 0
Hello from thread 1
# Check the libraries used
$ ldd ./a.out
... elided ...
        libomp.so => /lustre/software/aarch64/tools/arm-compiler/20.1//arm-linux-compiler-20.1_Generic-AArch64_SUSE-12_aarch64-linux/lib/libomp.so (0x0000400019bc3000)
... elided ...

# Run using LOMP as the OpenMP runtime
$ LD_LIBRARY_PATH=`echo ~/lomp/build_aarch64/src`:${LD_LIBRARY_PATH} OMP_NUM_THREADS=2 ./a.out
Hello from thread 0
Hello from thread 1

# Check the libraries used
$ LD_LIBRARY_PATH=`echo ~/lomp/build_aarch64/src`:${LD_LIBRARY_PATH} ldd ./a.out
... elided ...
        libomp.so => /home/br-jcownie/lomp/build_aarch64/src/libomp.so (0x000040002dfeb000)
... elided ...

# Prove that I'm using LOMP by asking it to announce itself
$ LD_LIBRARY_PATH=`echo ~/lomp/build_aarch64/src`:${LD_LIBRARY_PATH} OMP_NUM_THREADS=2 LOMP_DEBUG=1 ./a.out
LOMP:runtime version 0.1 (SO version 1) compiled at 12:04:35 on Jul 20 2021
LOMP:from Git commit 5db9696 for aarch64 by LLVM:11:0:0
LOMP:with configuration -march=armv8.1a;DEBUG=10;LOMP_WARN_API_STUBS=1;LOMP_WARN_ARCH_FEATURES=1;LOMP_HAVE_LIBATOMIC=\
1;LOMP_HAVE_LIBNUMA=1
Hello from thread 0
Hello from thread 1
$

MacOS Solution

Asking the web shows that MacOS does not use LD_LIBRARY_PATH, but rather DYLD_LIBRARY_PATH

, and does not have an ldd command, but, instead provides the same information using the otool -L option.

“So, that’s all good then”.

Doing the same things as above, here’s what we see :-

$ clang -fopenmp hello_omp.c
$ OMP_NUM_THREADS=2 ./a.out
Hello from thread 0
Hello from thread 1
$ otool -L ./a.out
./a.out:
	/opt/homebrew/opt/llvm/lib/libomp.dylib (compatibility version 5.0.0, current version 5.0.0)
... elided ...

# Now set DYLD_LIBRARY_PATH and check
$ DYLD_LIBRARY_PATH=`echo ~/lomp/build/src`:${DYLD_LIBRARY_PATH} otool -L ./a.out
./a.out:
	/opt/homebrew/opt/llvm/lib/libomp.dylib (compatibility version 5.0.0, current version 5.0.0)
... elided ...

# Did we mis-spell the path? Try running anyway 
$ DYLD_LIBRARY_PATH=`echo ~/lomp/build/src`:${DYLD_LIBRARY_PATH} LOMP_DEBUG=1 OMP_NUM_THREADS=2 ./a.out 
LOMP:runtime version 0.1 (SO version 1) compiled at 11:41:47 on Aug  4 2021 LOMP:from Git commit 700e26b for aarch64 by LLVM:12:0:1 LOMP:with configuration LOMP_COMPILE_OPTIONS-NOTFOUND;DEBUG=10;LOMP_WARN_API_STUBS=1;LOMP_WARN_ARCH_FEATURES=1 
Hello from thread 0 
Hello from thread 1

The final execution proves that we do get the runtime we wanted, but otool -L told us the code would run with compiler’s runtime! How helpful…

What Is Going On?

We are hitting a piece of MacOS security enforcement!

By default modern instances of MacOS implement “System Integrity Protection” (SIP), and one of the things SIP does is to “sanitize” critical environment variables (such as DYLD_LIBRARY_PATH) when executing system commands (such as otool). This is done to prevent important executables from loading untrusted dynamic libraries. (Imagine what you could do if you loaded your own version of the system library into a process running with all privileges!)

Here, though, the effect is to make otool -L lie about the libraries that will actually be loaded by the process when it is not running under otool, which makes it positively misleading, rather than useful.

Another Approach

Instead of using ldd or otool, we can directly ask the dynamic linker to show us which libraries it loads. As ever, this is achieved in a slightly different way on OS X than on Linux.

Linux

On Linux the dynamic linker supports the LD_TRACE_LOADED_OBJECTS envirable, which, when set to 1, asks the dynamic linker to output information about all loaded code, and then exit before starting the program. In effect this is almost exactly like running ldd.

LD_LIBRARY_PATH=`echo ~/lomp/build_aarch64/src`:${LD_LIBRARY_PATH} LD_TRACE_LOADED_OBJECTS=1 ./a.out
... elided ...
        libomp.so => /home/br-jcownie/lomp/build_aarch64/src/libomp.so (0x00004000314e3000)
... elided ...

OS X

Here the magic envirable is DYLD_PRINT_LIBRARIES, which causes the dynamic loader to print information about each library which is loaded if the envirable is set to 1.

One difference, though, is that the dynamic linker continues on to execute the code, so this isn’t exactly the same as ldd or otool.

$ DYLD_LIBRARY_PATH=`echo ~/lomp/build/src`:${DYLD_LIBRARY_PATH} OMP_NUM_THREADS=2 DYLD_PRINT_LIBRARIES=1 ./a.out
dyld: loaded: <66F0786F-2994-39D6-9FCA-55CAC207B9B8> /Users/jcownie/tmp/./a.out
dyld: loaded: <0368D8A6-C4DA-3B2C-8683-30AB51F7E8D0> /Users/jcownie/lomp/build/src/libomp.dylib
... elided ...
LOMP:runtime version 0.1 (SO version 1) compiled at 11:41:47 on Aug  4 2021
LOMP:from Git commit 700e26b for aarch64 by LLVM:12:0:1
LOMP:with configuration LOMP_COMPILE_OPTIONS-NOTFOUND;DEBUG=10;LOMP_WARN_API_STUBS=1;LOMP_WARN_ARCH_FEATURES=1
Hello from thread 0
Hello from thread 1
$

Above you can see (both from the dyld trace and the behaviour) that the libomp we wanted is being loaded

Conclusions

OS X is not Linux; although the shell may be the same there are significant differences.
Be careful when people say “The MacOS equivalent of Linux’ X is MacOS’ Y”. The equivalence may not be complete and may be positively misleading!
If you think you want an ldd equivalent on OS X you’re likely to be less misled if you use DYLD_PRINT_LIBRARIES instead of a command.

Acknowledgements

This work used the Isambard 2 UK National Tier-2 HPC Service operated by GW4 and the UK Met Office, and funded by EPSRC (EP/T022078/1)

I once spent a long time debugging a code and fixed a variety of bugs before realising that because of a messed up environment I was never running my fixed versions of the code…

E.g. see this StackOverflow question and highest voted answer: “As you've noted, DYLD_LIBRARY_PATH behaves like LD_LIBRARY_PATH on other *nix.” (Though to be fair the next answer disagrees: “DYLD_LIBRARY_PATH does not behave like LD_LIBRARY_PATH.”

CPU fun