We present a mechanism for autonomous self-adaptation of a network-based intrusion detection system (IDS). The system is composed of a set of cooperating agents, each of which is based on an existing network behavior analysis method. The self adaptation mechanism is based on the insertion of a small number of challenges, i.e. known instances of past legitimate or malicious behavior. The response of individual system components to these challenges is used to measure and eventually optimize the system performance in terms of accuracy. In this work we show how to choose the challenges in a way such that the IDS attaches more importance to the detection of attacks that cause much damage. Categories and Subject Descriptors: C.2.0 [ComputerCommunication Networks]: Security and Protection. General Terms: Security, Management, Measurement.