Evaluation Of Learning Performance In Backpropagation Neural Networks And Decision-tree Classifier Systems

Learning accuracy is often used as a measurement for evaluating as well as comparing the performance of automated learning mechanisms such as artificial neural networks (ANN) and decision-tree classifier systems (DTCS). Experience in applying machine learning techniques to real problems shows that, in many cases, learning accuracy measurements are incomplete measurements of learning success. The work presented in this paper describes the measurement of learning accuracy in both artificial neural networks and decision-tree classifier systems. When an ANN and a DTCS are applied to the same domain, the learning accuracy of the resulting ANN may exceed that of the DTCS. However, experience indicates that the ANN may produce results that are less conservative than those produced by the DTCS. Examples are provided to show how these measurements can differ between the two types of learning strategies and why the practitioner's interpretation of accuracy results must be based both on the learning algorithm used and on the domain being learned. Alternative ways of evaluating accuracy are suggested that may be more suitable to some application domains than the standard learning accuracy measurement. Experimental results from 'real' domains are used to illustrate these concepts.