This paper aims at proposing a methodology for evaluating current IDS capabilities of detecting attacks targeting the networks and their services. This methodology tries to be as realistic as possible and reproducible, i.e. it works with real attacks and real traffic in controlled environments. It especially relies on a database containing attack traces specifically created for that evaluation purpose. By confronting IDS to these attack traces, it is possible to get a statistical evaluation of IDS, and to rank them according to their detection capabilities without false alarms. For illustration purposes, this paper shows the results obtained with 3 public IDS. It also shows how the attack traces database impacts the results got for the same IDS. Keywords. Statistical evaluation of IDS, attack traces, ROC curves, KDD'99 I Motivation I.1 Problematics Internet is becoming the universal communication network, conveying all kinds of information, ranging from the simple transfer of bina...