LD_PRELOAD for real world heap access tracking

Instrumenting vs Overloading

It is important to understand the difference between these two ways to track memory allocation : edLeak is instrumenting allocation functions, meaning that allocation calls are measured and then the original allocation function is called. Most memory tracking tools overload the allocation function with other allocators implementations.

These two ways have different aims: Instrumenting minimises the behavior changes of the running code while Overloading allows to check much more aspects of the allocations. This also means that Instrumenting is more constrained by the allocator implementation (i.e. glibc or uclibc in our case) while Overloading is more independent from it.

Hooking what ?

The usual suspects to track heap usage are malloc/calloc/realloc/free. However some less common functions should also be hooked depending on the targets that you need to analyze. Here is a - hopefully - exhaustive list:

• malloc / free : Allocate and free memory.
• calloc : Allocate and initialize to zero.
• realloc : Allocate or re-allocate some memory.
• memalign / valloc / posix_memalign : Allocates some memory with alignment constraints.

 Moreover if C++ is used then multiple new and delete variants must be hooked :

• new / delete : Allocate and delete an object.
• new (std::nothrow) : does not through exception when the new operator runs out of memory.
• new[] / delete[] : Allocate and delete objects arrays.
• new[] (std::nothrow) : does not through exception when running out of memory.

Bootstrap

The bootstrap was already re-written several times in edLeak before a good solution was found: Since the hooks are preloaded they are  called as soon as the library is loaded which happens before \"main\" is reached and before constructor functions are called. During that phase, all hooks must be initialized. The main issue is that the only place where this can be done is when the first hook is call. So the following rules must be respected:

1. As long as the initialization is not completed, then all hooks must operate as pass-through : They just redirect the calls to libc without any action.
2. constructor attributes should not be used for initialization because they will be called after the first hooks. All allocations that are done during the executable load time will not be tracked.
3. global static variables must not be used : They will be initialized after the first hooks are called. However static variables inside functions/methods are the only way to ensure that the static variables will be initialized at the correct time. Such variables are guaranteed to be initialized when the function/method is called for the first time.
4. Pass-through must be disabled only once the full initialization is done since it is probable that this step will call allocation functions. if pass-through is disabled too early then the bootstrap code calls allocation functions before they are initialized.

This bootstrap is implemented via a 3 state state machine in edLeak:

• Start : Initial state when no hook has been called.
• Starting : Transitioning state, entered as soon as the first hook is called.
• Started : Nominal state; entered once initialization is completed. When entering this state the hooks change from pass-through to hooking mode.

Redirecting to libc

Redirecting a call to its original implementation is quite well documented but some code is provided so that all useful information is here. This is done very easily via a specific dlsym handle:

typedef void *(*malloc_t)(size_t);

malloc_t   AllocFunc;
AllocFunc = (malloc_t) dlsym(RTLD_NEXT, \"malloc\");

This handle was introduced in linux but is also supported on other OSes. malloc of libc can then be called via the AllocFunc function pointer. Be carreful when hooking a C++ function: The name must be the mangled name.

Also calloc is special on glibc : since dlsym is calling calloc on glibc, dlsym cannot be used to retrieve the address of calloc. An other exported symbol must be used : __libc_calloc.

Deadlocks, Recursive calls, and threads

If a lock is used to protect mutliple calls of the hooks (which is the case in edLeak), then some extra problems occur: Most glibc and uclibc also protect some of their code with their own mutexes. This is not a problem as long as no hook leads to recursive calls to the libc. Otherwise this leads to deadlocks. More specifically the following cases were identified in edLeak until now:

• dladdr can cause deadlocks on glibc (). So dladdr must not be used in hooks. Since this function is the easy way to do name resolution, name resolution must be defered later : edLeak does this during the first query of the hooks results (When a new slice is generated).
• Creating threads also leads to deadlocks in glibc so the bootstrap code must not create any thread.

Other tricks

Some other things must be done correctly to get a reliable set of heap hooks.

The first one is memory alignment. malloc and other allocation functions must ensure that the result is 8 byte aligned. Several frameworks such as glib fully rely on this. Obviously memalign must respect the requested alignment. This means that the hook must take into account the size of the data added on the begining of the memory allocated from libc. Otherwise libc will allocate memory correctly aligned, but the hook will beak it when adding an offset to the returned pointer.

The second one concerns realloc : Depending of its input parameters, this function can do different things:

• If size is 0 then the provided pointer is fred (realloc acts as free).
• If the provided pointer is  NULL then new memory is allocated (realloc acts as malloc).

The third one concerns free : free can be called with a NULL pointer. This should not be considered as an error.