publication . Conference object . 2009

A study of inter-annotator agreement for opinion retrieval

Adam Bermingham; Alan F. Smeaton;
Open Access
  • Published: 28 Jul 2009
  • Publisher: ACM Press
  • Country: Ireland
Evaluation of sentiment analysis, like large-scale IR evalu- ation, relies on the accuracy of human assessors to create judgments. Subjectivity in judgments is a problem for rel- evance assessment and even more so in the case of senti- ment annotations. In this study we examine the degree to which assessors agree upon sentence-level sentiment anno- tation. We show that inter-assessor agreement is not con- tingent on document length or frequency of sentiment but correlates positively with automated opinion retrieval per- formance. We also examine the individual annotation cate- gories to determine which categories pose most di±culty for annotators.
Persistent Identifiers
free text keywords: Computational linguistics, Machine learning, Information retrieval, Information retrieval, Natural language processing, computer.software_genre, computer, IR evaluation, Artificial intelligence, business.industry, business, Computational linguistics, Annotation, Subjectivity, Sentiment analysis, Computer science
Related Organizations
Funded by
SFI| CSET CLARITY: Bringing Information to Life
  • Funder: Science Foundation Ireland (SFI)
  • Project Code: 07/CE/I1147
  • Funding stream: SFI Centre for Science Engineering and Technology (CSET)
Digital Humanities and Cultural Heritage

[1] A. Bermingham, A. Smeaton, J. Foster, and D. Hogan. DCU at the TREC 2008 Blog Track. In The Seventeenth Text REtrieval Conference (TREC 2008) Proc., 2008. [OpenAIRE]

[2] A. F. Hayes and K. Krippendor®. Answering the call for a standard reliability measure for coding data. In Communication Methods and Measures, 2007.

[3] I. Ounis, C. MacDonald, and I. Soboro®. Overview of the TREC-2008 Blog Track. In The Text REtrieval Conference (TREC 2008) Proc. NIST, 2008.

[4] Y. Seki, D. K. Evans, L. Ku, L. Sun, H. Chen, and N. Kando. Overview of multilingual opinion analysis task at NTCIR-7. 2008.

[5] J. Wiebe, T. Wilson, and C. Cardie. Annotating expressions of opinions and emotions in language. Language Resources and Evaluation, 1(2):0, 2005.

Any information missing or wrong?Report an Issue