Modern Userland Exec

2014-02-18 Bruce Ediger


In what amounts to a grand tradition stretching back decades, Linux has the execve() system call. The man page for execve() says: "execve() executes the program pointed to by filename...", and goes on to say something interesting: "the text, data, bss, and stack of the calling process are overwritten by that of the program loaded".

The enterprising person reading that man page contemplates it in light of mmap() and the man page of and sees that this overwriting doesn't necessarily constitute something that the Linux kernel must have a monopoly over, unlike filesystem access, or any number of other things. "Heck, I can do that myself", the enterprising person might think, "and learn something in the process".

Original Grugq's Userland Exec

The Grugq's userland exec worked by compiling an ELF shared object that has no references to any libc code. He did this with Diet libc. The example program given by The Grugq used the function dlopen() to read in and relocate that ELF shared object with no external references. Transferring flow-of-control to the ul_exec() function inside the shared object allows the shared object code to unmap from the process' address space any pages from the example program, text, bss or data. At that point, code in the shared object can read in a new ELF executable, create memory maps that would have overlapped or overwritten the original example program, and copy code and data from the new ELF exectable into newly-mapped memory. Userland exec also maps in /lib/ld-*.so, the dynamic linker, creates a new stack, jumps into the dynamic linker in the same fashion that the kernel does, and a new process is born, overlaying the old one.

Modern Userland Exec

Lots of particulars about Glibc, Linux and dynamically-linked executables have changed since 2004. Linus introduced the "Virtual Dynamic Shared Object" (VDSO). The number and type of ELF auxillary vectors passed to a new process by the Linux kernel have changed. The top-of-stack doesn't always appear at the same address, due to "Address Space Layour Randomization" (ASLR). The mmap system call doesn't necessarily map "anonymous" memory at higher addresses with each invocation. Diet libc and Glibc interact strangely via "weak references". All of these things necessitate changes to userland exec to get it to work.

My 32-bit refresh of The Grugq's userland exec differs in these particulars:

My 64-bit userland exec is a complete rewrite, as Diet libc doesn't work as well with x86_64 CPUs. It does much of the same work as the 32-bit version, but with it's own, position-independent code version of system calls.

Development Environment

I did the 32-bit development under 32-bit Arch linux, so a 3.2, 3.3 or 3.4 linux kernel, Glibc 2.15 and 2.16, and Gcc 4.7.x, along with Diet libc 0.32.

For the 64-bit version, I used Gcc 4.8.x, Glibc 2.19 and linux 3.12.9.


For the 32-bit version, you will need to have Diet libc 0.32 installed to compile and exercize the modern userland exec. The link to v0.32 source code is broken on that page: you'll have to scrounge around for the source.

The 64-bit version only requires GNU CC. You can also use clang as a C compiler.

Since Details of the glibc dynamic linker change, you may need to do some futzing around to get either of these to work at all, much less work well.

32-bit userland exec source code.

64-bit userland exec source code.


A simple make creates everything.


There really isn't an "installation". When compiled, the whole thing consists of example, an ELF executable, and, a shared object dynamically loaded by example at run time.


My personal criteria was that if the program example could get vim to start correctly it was done. You can accomplish this by invoking the following in the ul/ directory (32-bit):

$ ./example /usr/bin/vim /etc/hosts

For the 64-bit example:

$ PATH=$PATH:. example ./ /usr/bin/vim /etc/hosts

Running vim should demonstrate that the whole thing works. For more information you can do this:

$ ./example $(which cat) /proc/self/maps
08048000-08053000 r-xp 00000000 00:00 0 
08053000-08054000 r--p 00000000 00:00 0 
08054000-08055000 rw-p 00000000 00:00 0 
08640000-08682000 rw-p 00000000 00:00 0          [heap]
b7277000-b743c000 r--p 00000000 08:03 37994      /usr/lib/locale/locale-archive
b743c000-b75db000 r-xp 00000000 08:03 139405     /lib/
b75db000-b75dc000 ---p 0019f000 08:03 139405     /lib/
b75dc000-b75de000 r--p 0019f000 08:03 139405     /lib/
b75de000-b75df000 rw-p 001a1000 08:03 139405     /lib/
b75df000-b75e3000 rw-p 00000000 00:00 0 
b773f000-b775f000 r-xp 00000000 00:00 0 
b775f000-b7760000 r--p 00000000 00:00 0 
b7760000-b7761000 rw-p 00000000 00:00 0 
b7786000-b7789000 rw-p 00000000 00:00 0 
b778f000-b7790000 rw-p 00000000 00:00 0 
b7796000-b779d000 r-xp 00000000 08:04 929848     /home/bediger/src/csrc/ul/
b779d000-b779e000 rw-p 00006000 08:04 929848     /home/bediger/src/csrc/ul/
b779e000-b77a0000 rw-p 00000000 00:00 0 
b77a0000-b77a1000 r-xp 00000000 00:00 0          [vdso]
b77b0000-b77b1000 rw-p 00000000 00:00 0 
b77c2000-b77c3000 rw-p 00000000 00:00 0 
bfa3b000-bfa5b000 rwxp 00000000 00:00 0          [stack]
bfa5b000-bfa5c000 rw-p 00000000 00:00 0 

Compared to invoking cat /proc/self/maps, you don't see mappings for /bin/cat or /lib/ example and userland exec don't map pages directly from a file like the kernel does. This gets around any filesystems mounted "noexec", and lets you use the example program to run programs off filesystems mounted "noexec".

For an excess of fun, you can try this with the 64-bit version:

$ PATH=$PATH:. example ./ example ./ example ./ $(which cat) /proc/self/maps

That actually works, because example reads in the executable to be userland exec'ed, rather than memory mapping it. It can read itself in just as easily as it can read in any other ELF executable file. It does leave anonymously-mapped pieces of memory containing all over the address space.