Deep Learning from Crowds

    Research output: Chapter in Book/Report/Conference proceedingArticle in proceedingsResearchpeer-review

    114 Downloads (Pure)

    Abstract

    Over the last few years, deep learning has revolutionized the field of machine learning by dramatically improving the stateof-the-art in various domains. However, as the size of supervised artificial neural networks grows, typically so does the need for larger labeled datasets. Recently, crowdsourcing has established itself as an efficient and cost-effective solution for labeling large sets of data in a scalable manner, but it often requires aggregating labels from multiple noisy contributors with different levels of expertise. In this paper, we address the problem of learning deep neural networks from crowds. We begin by describing an EM algorithm for jointly learning the parameters of the network and the reliabilities of the annotators. Then, a novel general-purpose crowd layer is proposed, which allows us to train deep neural networks end-to-end, directly from the noisy labels of multiple annotators, using only backpropagation. We empirically show that the proposed approach is able to internally capture the reliability and biases of different annotators and achieve new state-of-the-art results for various crowdsourced datasets across different settings, namely classification, regression and sequence labeling.
    Original languageEnglish
    Title of host publicationProceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18)
    PublisherAAAI Press
    Publication date2018
    Pages1611-1618
    ISBN (Electronic)978-1-57735-800-8
    Publication statusPublished - 2018
    EventThe Thirty-Second AAAI Conference on Artificial Intelligence (AAAI), 2018 -
    Duration: 2 Feb 20187 Feb 2018
    Conference number: 32

    Conference

    ConferenceThe Thirty-Second AAAI Conference on Artificial Intelligence (AAAI), 2018
    Number32
    Period02/02/201807/02/2018

    Cite this

    Rodrigues, F., & Pereira, F. C. (2018). Deep Learning from Crowds. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18) (pp. 1611-1618). AAAI Press.