Torture Tests: A Quantitative Analysis for the Robustness of Knowledge-Based Systems

Abstract. The overall aim of this paper is to provide a general setting for quantitative quality measures of Knowledge-Based System behavior which is widely applicable to many Knowledge-Based Systems. We propose a general approach that we call "degradation studies": an analysis of how system output degrades as a function of degrading system input, such as incomplete or incorrect inputs. Such degradation studies avoid a number of problems that have plagued earlier attempts at defining such quality measures because they do not require a comparison between different (and often incomparable) systems, and they are entirely independent of the internal workings of the particular Knowledge-Based System at hand. To show the feasibility of our approach, we have applied it in a specific casestudy. We have taken a large and realistic vegetation-classification system, and have analyzed its behavior under various varieties of missing input. This casestudy shows that degradation studies can...

• Type: Conference
• Year: 2000
• Where: EKAW
• Authors: Perry Groot, Frank van Harmelen, Annette ten Teije
• Keywords: Degradation Studies, EKAW 2000, Information Management, Knowledge-based Systems, Quality Measures