By Amit Malik on Jul 15 2012
Some people queried me about ShellDetect specifically, how it works or how they can use it for shellcode analysis and detection. Shelldetect is the result of a short term research programme, I was actually trying to build a tool that can detect shellcode using the dynamic analysis apart from applying GetPC heuristics or some similar kind of technologies because my main objective was not just detection but the analysis as well. Dynamic analysis provides much flexibility, we can get exact decoded shellcode and if it is a drive by download then we can get embedded urls as well.
Another important thing is to detect the shellcode from a raw stream because if you are an analyst then you must be getting couple of malicious files like media (avi,jpeg etc.) files, pcap files etc. In some cases it becomes important that if the file is really malicious or not.
Shelldetect serves both the purposes. It will detect the shellcode from a raw stream and also dump the decoded shellcode on the disk for analysis. At the core it is using a simple but effective detection algorithm, it will execute the stream byte by byte and try to identify the actual executable code by hooking the loadlibrary. When it identifies the valid executable code then it will alert about shellcode detected and dumps the decoded shellcode on the disk. In 99% cases it will not execute the entire shellcode or infect the analysis system but as it is executing the stream so there is always a possibility of shellcode execution so we recommend using it inside virtual machines.
The schandler.exe is the main analysis engine while the python programme is equally important and communicate with schandler.exe. Currently the time for detection is directly proportional to the number of invalid instructions inside a stream and ofcourse it is also the area of improvement. The detection rate is 100% for the real life shellcodes.
Shelldetect only works for Windows shellcodes.
Let’s check some shellcodes against shelldetect.
Below snapshot is of a blackhole exploit kit shellcode, we will feed this stream to shelldetect.
Below snapshot shows the result of shelldetect.
The stream was direct shellcode so shelldetect reported the results without any delay; in the snapshot we can see the decoded shellcode and also the malicious url.
Let’s feed a random stream to shelldetect and see how shelldetect react to it.
As you can see that shelldetect detected the shellcode from a random stream, actually from implementation point of view it is not that easy because any random stream can assemble to random instructions which can effect the analysis process in uncertain ways, for example any input/output instruction can hang a process permanently while some instructions are very noisy and can kill analysis process. But shelldect can handle such type of errors and reports the exact result on the console.
You can download the shelldetect from here.