Please use this identifier to cite or link to this item: http://hdl.handle.net/10316/27407
Title: Learning from multiple annotators: distinguishing good from random labelers
Authors: Rodrigues, Filipe 
Pereira, Francisco 
Ribeiro, Bernardete 
Keywords: Multiple annotators; Crowdsourcing; Latent variable models; Expectation–Maximization; Logistic Regression
Issue Date: 1-Sep-2013
Publisher: Elsevier
Citation: RODRIGUES, Filipe; PEREIRA, Francisco; RIBEIRO, Bernardete - Learning from multiple annotators: distinguishing good from random labelers. "Pattern Recognition Letters". ISSN 0167-8655. Vol. 34 Nº. 12 (2013) p. 1428-1436
Abstract: With the increasing popularity of online crowdsourcing platforms such as Amazon Mechanical Turk (AMT), building supervised learning models for datasets with multiple annotators is receiving an increasing attention from researchers. These platforms provide an inexpensive and accessible resource that can be used to obtain labeled data, and in many situations the quality of the labels competes directly with those of experts. For such reasons, much attention has recently been given to annotator-aware models. In this paper, we propose a new probabilistic model for supervised learning with multiple annotators where the reliability of the different annotators is treated as a latent variable. We empirically show that this model is able to achieve state of the art performance, while reducing the number of model parameters, thus avoiding a potential overfitting. Furthermore, the proposed model is easier to implement and extend to other classes of learning problems such as sequence labeling tasks.
URI: http://hdl.handle.net/10316/27407
ISSN: 0167-8655
DOI: 10.1016/j.patrec.2013.05.012
Rights: openAccess
Appears in Collections:I&D CISUC - Artigos em Revistas Internacionais
FCTUC Eng.Informática - Artigos em Revistas Internacionais

Files in This Item:
File Description SizeFormat 
Learning from Multiple Annotators.pdf1.45 MBAdobe PDFView/Open
Show full item record

SCOPUSTM   
Citations

33
checked on Jun 25, 2019

WEB OF SCIENCETM
Citations

22
checked on Jun 25, 2019

Page view(s)

106
checked on Jul 17, 2019

Download(s) 20

580
checked on Jul 17, 2019

Google ScholarTM

Check

Altmetric


Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.