research product . 2018

Is it worth it? Budget-related evaluation metrics for model selection

Klubicka, Filip; Salton, Giancarlo; Kelleher, John D.;
Open Access
  • Published: 01 Jan 2018
  • Publisher: Dublin Institute of Technology
  • Country: Ireland
Projects that set out to create a linguistic resource often do so by using a machine learning model that pre-annotates or filters the content that goes through to a human annotator, before going into the final version of the resource. However, available budgets are often limited, and the amount of data that is available exceeds the amount of annotation that can be done. Thus, in order to optimize the benefit from the invested human work, we argue that the decision on which predictive model one should employ depends not only on generalized evaluation metrics, such as accuracy and F-score, but also on the gain metric. The rationale is that, the model with the high...
free text keywords: model evaluation, gain, budget, linguistic resource creation, idiom identification, idiom dictionary, F-score, Computational Engineering, Digital Humanities, Other Computer Engineering
Related Organizations
Social Science and Humanities
Digital Humanities and Cultural Heritage
Download from
Any information missing or wrong?Report an Issue