A Scalable Smart Meter Data Generator Using Spark

Nadeem Iftikhar, Xiufeng Liu, Sergiu Danalachi, Finn Nordbjerg, Jens Vollesen

    Research output: Chapter in Book/Report/Conference proceedingArticle in proceedingsResearchpeer-review

    151 Downloads (Pure)

    Abstract

    Today, smart meters are being used worldwide. As a matter of fact smart meters produce large volumes of data. Thus, it is important for smart meter data management and analytics systems to process petabytes of data. Benchmarking and testing of these systems require scalable data, however, it can be challenging to get large data sets due to privacy and/or data protection regulations. This paper presents a scalable smart meter data generator using Spark that can generate realistic data sets. The proposed data generator is based on a supervised machine learning method that can generate data of any size by using small data sets as seed. Moreover, the generator can preserve the characteristics of data with respect to consumption patterns and user groups. This paper evaluates the proposed data generator in a cluster based environment in order to validate its effectiveness and scalability.
    Original languageEnglish
    Title of host publicationOTM Confederated International Conferences "On the Move to Meaningful Internet Systems"
    PublisherSpringer
    Publication date2017
    Pages21-36
    DOIs
    Publication statusPublished - 2017
    EventOn the Move to Meaningful Internet Systems. OTM 2017 - Rhodos, Greece
    Duration: 23 Oct 201727 Oct 2017

    Conference

    ConferenceOn the Move to Meaningful Internet Systems. OTM 2017
    CountryGreece
    CityRhodos
    Period23/10/201727/10/2017
    SeriesLecture Notes in Computer Science
    Volume10573
    ISSN0302-9743

    Cite this

    Iftikhar, N., Liu, X., Danalachi, S., Nordbjerg, F., & Vollesen, J. (2017). A Scalable Smart Meter Data Generator Using Spark. In OTM Confederated International Conferences "On the Move to Meaningful Internet Systems" (pp. 21-36). Springer. Lecture Notes in Computer Science, Vol.. 10573 https://doi.org/10.1007/978-3-319-69462-7_2
    Iftikhar, Nadeem ; Liu, Xiufeng ; Danalachi, Sergiu ; Nordbjerg, Finn ; Vollesen, Jens. / A Scalable Smart Meter Data Generator Using Spark. OTM Confederated International Conferences "On the Move to Meaningful Internet Systems". Springer, 2017. pp. 21-36 (Lecture Notes in Computer Science, Vol. 10573).
    @inproceedings{934b7d64a6be4e1084de7caf8c14d80d,
    title = "A Scalable Smart Meter Data Generator Using Spark",
    abstract = "Today, smart meters are being used worldwide. As a matter of fact smart meters produce large volumes of data. Thus, it is important for smart meter data management and analytics systems to process petabytes of data. Benchmarking and testing of these systems require scalable data, however, it can be challenging to get large data sets due to privacy and/or data protection regulations. This paper presents a scalable smart meter data generator using Spark that can generate realistic data sets. The proposed data generator is based on a supervised machine learning method that can generate data of any size by using small data sets as seed. Moreover, the generator can preserve the characteristics of data with respect to consumption patterns and user groups. This paper evaluates the proposed data generator in a cluster based environment in order to validate its effectiveness and scalability.",
    author = "Nadeem Iftikhar and Xiufeng Liu and Sergiu Danalachi and Finn Nordbjerg and Jens Vollesen",
    year = "2017",
    doi = "10.1007/978-3-319-69462-7_2",
    language = "English",
    series = "Lecture Notes in Computer Science",
    publisher = "Springer",
    pages = "21--36",
    booktitle = "OTM Confederated International Conferences {"}On the Move to Meaningful Internet Systems{"}",

    }

    Iftikhar, N, Liu, X, Danalachi, S, Nordbjerg, F & Vollesen, J 2017, A Scalable Smart Meter Data Generator Using Spark. in OTM Confederated International Conferences "On the Move to Meaningful Internet Systems". Springer, Lecture Notes in Computer Science, vol. 10573, pp. 21-36, On the Move to Meaningful Internet Systems. OTM 2017, Rhodos, Greece, 23/10/2017. https://doi.org/10.1007/978-3-319-69462-7_2

    A Scalable Smart Meter Data Generator Using Spark. / Iftikhar, Nadeem; Liu, Xiufeng; Danalachi, Sergiu; Nordbjerg, Finn; Vollesen, Jens.

    OTM Confederated International Conferences "On the Move to Meaningful Internet Systems". Springer, 2017. p. 21-36 (Lecture Notes in Computer Science, Vol. 10573).

    Research output: Chapter in Book/Report/Conference proceedingArticle in proceedingsResearchpeer-review

    TY - GEN

    T1 - A Scalable Smart Meter Data Generator Using Spark

    AU - Iftikhar, Nadeem

    AU - Liu, Xiufeng

    AU - Danalachi, Sergiu

    AU - Nordbjerg, Finn

    AU - Vollesen, Jens

    PY - 2017

    Y1 - 2017

    N2 - Today, smart meters are being used worldwide. As a matter of fact smart meters produce large volumes of data. Thus, it is important for smart meter data management and analytics systems to process petabytes of data. Benchmarking and testing of these systems require scalable data, however, it can be challenging to get large data sets due to privacy and/or data protection regulations. This paper presents a scalable smart meter data generator using Spark that can generate realistic data sets. The proposed data generator is based on a supervised machine learning method that can generate data of any size by using small data sets as seed. Moreover, the generator can preserve the characteristics of data with respect to consumption patterns and user groups. This paper evaluates the proposed data generator in a cluster based environment in order to validate its effectiveness and scalability.

    AB - Today, smart meters are being used worldwide. As a matter of fact smart meters produce large volumes of data. Thus, it is important for smart meter data management and analytics systems to process petabytes of data. Benchmarking and testing of these systems require scalable data, however, it can be challenging to get large data sets due to privacy and/or data protection regulations. This paper presents a scalable smart meter data generator using Spark that can generate realistic data sets. The proposed data generator is based on a supervised machine learning method that can generate data of any size by using small data sets as seed. Moreover, the generator can preserve the characteristics of data with respect to consumption patterns and user groups. This paper evaluates the proposed data generator in a cluster based environment in order to validate its effectiveness and scalability.

    U2 - 10.1007/978-3-319-69462-7_2

    DO - 10.1007/978-3-319-69462-7_2

    M3 - Article in proceedings

    T3 - Lecture Notes in Computer Science

    SP - 21

    EP - 36

    BT - OTM Confederated International Conferences "On the Move to Meaningful Internet Systems"

    PB - Springer

    ER -

    Iftikhar N, Liu X, Danalachi S, Nordbjerg F, Vollesen J. A Scalable Smart Meter Data Generator Using Spark. In OTM Confederated International Conferences "On the Move to Meaningful Internet Systems". Springer. 2017. p. 21-36. (Lecture Notes in Computer Science, Vol. 10573). https://doi.org/10.1007/978-3-319-69462-7_2