What Size Test Set Gives Good Error Rate Estimates?

—We address the problem of determining what size test set guarantees statistically significant results in a character recognition task, as a function of the expected error rate. We provide a statistical analysis showing that if, for example, the expected character error rate is around 1 percent, then, with a test set of at least 10,000 statistically independent handwritten characters (which could be obtained by taking 100 characters from each of 100 different writers), we guarantee, with 95 percent confidence,

• Type: Journal
• Year: 1998
• Where: PAMI
• Authors: Isabelle Guyon, John Makhoul, Richard M. Schwartz, Vladimir Vapnik
• Keywords: Character Recognition Task, Error Rate, Expected Character Error, PAMI 1998