Torture Tests A Quantitative Analysis for the Robustness of KnowledgeBased Systems.pdf

Torture tests: a quantitative analysis for the robustness of Knowledge-Based Systems 1 1 2 Perry Groot , Frank van Harmelen , and Annette ten Teije 1 Division of Mathematics and Computer Science, Faculty of Sciences, Vrije Universiteit, De Boelelaan 1081A, 1081 HV Amsterdam, the Netherlands. E-mail: fperry,frankhg @cs.vu.nl. 2 Department of Computer Science, Universiteit Utrecht, PO Box 80.089, Padualaan 14, 3508 TB Utrecht, the Netherlands. E-mail: annette@cs.uu.nl. Abstract. The overall aim of this paper is to provide a general setting for quan- titative quality measures of Knowledge-Based System behavior which is widely applicable to many Knowledge-Based Systems. We propose a general approach that we call “degradation studies”: an analysis of how system output degrades as a function of degrading system input, such as incomplete or incorrect inputs. Such degradation studies avoid a number of problems that have plagued earlier attempts at defining such quality measures because they do not require a compar- ison between different (and often incomparable) systems, and they are entirely independent of the internal workings of the particular Knowledge-Based System at hand. To show the feasibility of our approach, we have applied it in a specific case- study. We have taken a large and realistic vegetation-classification system, and have analyzed its behavior under various varieties of missing input. This case- study shows that degradation studies can reveal interesting and surprising prop- erties of the system under study. 1 Motivation When asked about the essential differences between Knowledge-Based Systems (KBSs) and “conven