On evolving buffer overflow attacks using genetic programming

In this work, we employed genetic programming to evolve a "white hat" attacker; that is to say, we evolve variants of an attack with the objective of providing better detectors. Assuming a generic buffer overflow exploit, we evolve variants of the generic attack, with the objective of evading detection by signature-based methods. To do so, we pay particular attention to the formulation of an appropriate fitness function and partnering instruction set. Moreover, by making use of the intron behavior inherent in the genetic programming paradigm, we are able to explicitly obfuscate the true intent of the code. All the resulting attacks defeat the widely used 'Snort' Intrusion Detection System. Categories and Subject Descriptors K.6.5 [Security and Protection]: Unauthorized access; I.2.8 [Artificial Intelligence]: Problem Solving, Control Methods, and Search; I.2.2 [Automatic Programming]; General Terms: Algorithms, Design, Security.

• Type: Conference
• Year: 2006
• Where: GECCO
• Authors: Hilmi Günes Kayacik, Malcolm I. Heywood, A. Nur Zincir-Heywood
• Keywords: GECCO 2006, Generic Buffer Overflow, Genetic Programming, Genetic Programming Paradigm, Optimization