Constructing malware normalizers using term rewriting

A malware mutation engine is able to transform a malicious program to create a different version of the program. Such mutation engines are used at distribution sites or in self-propagating malware in order to create variation in the distributed programs. Program normalization is a way to remove variety introduced by mutation engines, and can thus simplify the problem of detecting variant strains. This paper introduces the "normalizer construction problem" (NCP), and formalizes a restricted form of the problem called "NCP=", which assumes a model of the engine is already known in the form of a term rewriting system. It is shown that even this restricted version of the problem is undecidable. A procedure is provided that can, in certain cases, automatically solve NCP= from the model of the engine. This procedure is analyzed in conjunction with term rewriting theory to create a list of distinct classes of normalizer construction problems. These classes yield a list of ...

• Type: Journal
• Year: 2008
• Where: VIROLOGY
• Authors: Andrew Walenstein, Rachit Mathur, Mohamed R. Chouchane, Arun Lakhotia
• Keywords: Malware, Mutation Engines, Normalizer Construction Problems, VIROLOGY 2008