Advantages, Opportunities and Limits of Empirical Evaluations: Evaluating Adaptive Systems

While empirical evaluations are a common research method in some areas of Artificial Intelligence (AI), others still neglect this approach. This article outlines both the opportunities and the limits of empirical evaluations for AI techniques exemplified by the evaluation of adaptive systems. Using the so called layered evaluation approach, we demonstrate that empirical evaluations are able to identify errors in AI systems that would otherwise remain undiscovered. To encourage new evaluations we implemented an online database of studies that are concerned with empirical evaluations of adaptive systems (EASy-D). 1 Advantages: Why Evaluations are needed Some areas of AI apply empirical methods regularly. E.g., planning and search algorithms are benchmarked in standard domains, and machine learning algorithms are usually tested with real data sets. However, looking at some applied areas such as user modeling, empirical studies are rare. E.g., only a quarter of the articles published in U...

• Type: Journal
• Year: 2002
• Where: KI
• Authors: Stephan Weibelzahl, Gerhard Weber
• Keywords: Adaptive Systems, AI Techniques, Artificial Intelligence, Empirical Evaluation, KI 2002