Bayesian performance comparison of text classifiers

D Zhang, J Wang, Emine Yilmaz, X Wang, Y Zhou

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

5 Scopus citations

Abstract

How can we know whether one classifier is really better than the other? In the area of text classification, since the publication of Yang and Liu's seminal SIGIR-1999 paper, it has become a standard practice for researchers to apply nullhypothesis significance testing (NHST) on their experimental results in order to establish the superiority of a classifier. However, such a frequentist approach has a number of inherent deficiencies and limitations, e.g., the inability to accept the null hypothesis (that the two classifiers perform equally well), the difficulty to compare commonly-used multivariate performance measures like F1 scores instead of accuracy, and so on. In this paper, we propose a novel Bayesian approach to the performance comparison of text classifiers, and argue its advantages over the traditional frequentist approach based on t-test etc. In contrast to the existing probabilistic model for F1 scores which is unpaired, our proposed model takes the correlation between classifiers into account and thus achieves greater statistical power. Using several typical text classification algorithms and a benchmark dataset, we demonstrate that the our approach provides rich information about the difference between two classifiers' performances.

Original languageAmerican English
Title of host publicationSIGIR 2016 - Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval
PublisherAssociation for Computing Machinery, Inc
Pages15-24
Number of pages10
ISBN (Electronic)9781450342902
DOIs
StatePublished - 2016
Event39th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2016 - Pisa, Italy
Duration: Jul 17 2016Jul 21 2016

Publication series

NameSIGIR 2016 - Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval

Conference

Conference39th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2016
Country/TerritoryItaly
CityPisa
Period07/17/1607/21/16

Keywords

  • Bayesian inference
  • Hypothesis testing
  • Performance evaluation
  • Text classification

Fingerprint

Dive into the research topics of 'Bayesian performance comparison of text classifiers'. Together they form a unique fingerprint.
  • Dynamic User Interests

    Liang, S. (CoI), Ren, Z. (CoI), Zhao, Y. (CoI), Yilmaz, E. (CoI), Kanoulas, E. (CoI), Ma, J. (CoI), De Rijke, M. (CoI) & Hobby, M. (CoI)

    08/1/1507/1/19

    Project: Research

Cite this