Comprehensive shellcode detection using runtime heuristics

A promising method for the detection of previously unknown code injection attacks is the identification of the shellcode that is part of the attack vector using payload execution. Existing systems based on this approach rely on the self-decrypting behavior of polymorphic code and can identify only that particular class of shellcode. Plain, and more importantly, metamorphic shellcode do not carry a decryption routine nor exhibit any self-modifications and thus both evade existing detection systems. In this paper, we present a comprehensive shellcode detection technique that uses a set of runtime heuristics to identify the presence of shellcode in arbitrary data streams. We have identified fundamental machine-level operations that are inescapably performed by different shellcode types, based on which we have designed heuristics that enable the detection of plain and metamorphic shellcode regardless of the use of self-decryption. We have implemented our technique in Gene, a code injectio...

• Type: Journal
• Year: 2010
• Where: ACSAC
• Authors: Michalis Polychronakis, Kostas G. Anagnostakis, Evangelos P. Markatos
• Keywords: ACSAC 2010, Code Injection Attack, Security Privacy, Shellcode, Shellcode Detection