Large scale topic modeling made practical

Bjarne Ørum Wahlgreen, Lars Kai Hansen

    Research output: Chapter in Book/Report/Conference proceedingArticle in proceedingsResearchpeer-review

    Abstract

    Topic models are of broad interest. They can be used for query expansion and result structuring in information retrieval and as an important component in services such as recommender systems and user adaptive advertising. In large scale applications both the size of the database (number of documents) and the size of the vocabulary can be significant challenges. Here we discuss two mechanisms that can make scalable solutions possible in the face of large document databases and large vocabularies. The first issue is addressed by a parallel distributed implementation, while the vocabulary problem is reduced by use of large and carefully curated term set. We demonstrate the performance of the proposed system and in the process break a previously claimed ’world record’ announced April 2010 both by speed and size of problem. We show that the use of a WordNet derived vocabulary can identify topics at par with a much larger case specific vocabulary.
    Original languageEnglish
    Title of host publication2011 IEEE International Workshop on Machine Learning for Signal Processing (MLSP)
    PublisherIEEE
    Publication date2011
    ISBN (Print)978-1-4577-1621-8
    ISBN (Electronic)978-1-4577-1622-5
    DOIs
    Publication statusPublished - 2011
    Event2011 IEEE International Workshop on Machine Learning for Signal Processing - Beijing, China
    Duration: 1 Jan 2011 → …
    http://mlsp2011.conwiz.dk/

    Conference

    Conference2011 IEEE International Workshop on Machine Learning for Signal Processing
    CountryChina
    CityBeijing
    Period01/01/2011 → …
    Internet address
    SeriesMachine Learning for Signal Processing
    ISSN1551-2541

    Cite this

    Wahlgreen, B. Ø., & Hansen, L. K. (2011). Large scale topic modeling made practical. In 2011 IEEE International Workshop on Machine Learning for Signal Processing (MLSP) IEEE. Machine Learning for Signal Processing https://doi.org/10.1109/MLSP.2011.6064628