Malicious Shellcode Detection with Virtual Memory Snapshots

Abstract—Malicious shellcodes are segments of binary code disguised as normal input data. Such shellcodes can be injected into a target process’s virtual memory. They overwrite the process’s return addresses and hijack control flow. Detecting and filtering out such shellcodes is vital to prevent damage. In this paper, we propose a new malicious shellcode detection methodology in which we take snapshots of the process’s virtual memory before input data are consumed, and feed the snapshots to a malicious shellcode detector. These snapshots are used to instantiate a runtime environment that emulates the target process’s input data consumption to monitor shellcodes’ behaviors. The snapshots can also be used to examine the system calls that shellcodes invoke, these system call parameters, and the process’s execution flow. We implement a prototype system in Debian Linux with kernel version 2.6.26. Our extensive experiments with real traces and thousands of malicious shellcod...

• Type: Journal
• Year: 2010
• Where: INFOCOM
• Authors: Boxuan Gu, Xiaole Bai, Zhimin Yang, Adam C. Champion, Dong Xuan
• Keywords: Communications, INFOCOM 2010, Input Data, Malicious Shellcodes, Virtual Memory