Polyglot: automatic extraction of protocol message format using dynamic binary analysis

Protocol reverse engineering, the process of extracting the application-level protocol used by an implementation, without access to the protocol specification, is important for many network security applications. Recent work [17] has proposed protocol reverse engineering by using clustering on network traces. That kind of approach is limited by the lack of semantic information on network traces. In this paper we propose a new approach using program binaries. Our approach, shadowing, uses dynamic binary analysis and is based on a unique intuition—the way that an implementation of the protocol processes the received application data reveals a wealth of information about the protocol message format. We have implemented our approach in a system called Polyglot and evaluated it extensively using real-world implementations of five different protocols: DNS, HTTP, IRC, Samba and ICQ. We compare our results with the manually crafted message format, included in Wireshark, one of the state-o...

• Type: Conference
• Year: 2007
• Where: CCS
• Authors: Juan Caballero, Heng Yin, Zhenkai Liang, Dawn Xiaodong Song
• Keywords: Application-level Protocol, CCS 2007, Network Traces, Protocol Reverse Engineering, Security Privacy