[Hooking] Unpacking by hooking?

Lets try something totally crazy. Lets try dynamic unpacking without total unpacking control, without breakpoints, without any kind of debugging whatsoever. Lets merge our unpacking process with the packer itself binding them into one unique work flow that collects information while packer is executing. Similar to what we do with debugging just without the debugger. How do we do this? Can we for that matter?

We can, with the little help from the TitanEngine's hooking library. The idea is to have our unpacker as a library which will be injected to the packed file during its execution. Such library would place hooks inside the packer code which redirect the control flow to our unpacker where ever data collection or execution handling is needed. Those places are usually spots where packer processes the import table, relocations, jumps to the original entry point or just switches execution from one layer to another.

Benefits of such approach? Even though its slightly harder to create and test such unpacker most notable benefit of unpacking by hooking is total immunity to various anti-debugging tricks used to detect the unpacking process. Only detection applicable to this unpacking scenario is anti-hooking and memory checksumming. First is hardly ever used in modern protections due to large number of false positives it gives which are triggered by the operating system itself, security software and various window skinning applications. And the second one is rarely present and when it is it only covers specific memory regions that correspond to single protection layer. In conclusion this method of implementing the unpacking process should give less things to worry about.

Implementing this kind of hooks requires a building custom functions that process the hook events. This is necessary to maintain the packed program work flow and is exactly why we preserve the register state with PUSHAD and if there is a jump affected by our hook even EFLAGS with PUSHFD. These ASM instructions are embedded in our C code and with the help of naked pre-processor instruction they become the prologue and epilogue of the function. To apply the hooks we use the DLL_PROCESS_ATTACH event. For example if we were to hook the UPX code which loads libraries the hook code flow would look like this:

Since our hooks are 5 bytes we need to "borrow" as much instructions we need to insert the hook. In this case we are "borrowing" three instructions. These instructions will be executed right after our inserted function is called. This is done to preserve the packer work flow. As you can see from this diagram we are using hooks instead of breakpoints. Therefore these hooks will be placed on at-least three places. When UPX calls LoadLibraryA, GetProcAddress and finally once is jumps to the entry point. Most basic sample UPX unpacker is limited to working on executables which don't import functions by ordinals and use the old jump to entry point method. Quite limited but enough for our technique proof-of-concept.

Debugging this kind of unpackers can be rather tricky. This video shows a quick and easy way to do it:

Since we are creating a hook library unpacker we also need a loader which will execute the unpacking target and inject the unpacker library in it. This can be done in number of ways but we decided to do via debug - detach method. Once both unpacker hook library and the loader are made our unpacker is complete. We hope you got the idea on how to use this technique to build your own hooking unpackers from our short blog. Until next week...

upxHooks
(package contains the unpacker with source and the samples used)

reflink: http://blog.reversinglabs.com/2010/06/unpacking-by-hooking/