A Scalable Smart Meter Data Generator Using Spark

Nadeem Iftikhar; Xiufeng Liu; Sergiu Danalachi; Finn Nordbjerg; Jens Vollesen

doi:10.1007/978-3-319-69462-7_2

A Scalable Smart Meter Data Generator Using Spark

Nadeem Iftikhar, Xiufeng Liu, Sergiu Danalachi, Finn Nordbjerg, Jens Vollesen

CITIES - Centre for IT-Intelligent Energy Systems

University College of Northern Denmark

Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review

478 Downloads (Pure)

Abstract

Today, smart meters are being used worldwide. As a matter of fact smart meters produce large volumes of data. Thus, it is important for smart meter data management and analytics systems to process petabytes of data. Benchmarking and testing of these systems require scalable data, however, it can be challenging to get large data sets due to privacy and/or data protection regulations. This paper presents a scalable smart meter data generator using Spark that can generate realistic data sets. The proposed data generator is based on a supervised machine learning method that can generate data of any size by using small data sets as seed. Moreover, the generator can preserve the characteristics of data with respect to consumption patterns and user groups. This paper evaluates the proposed data generator in a cluster based environment in order to validate its effectiveness and scalability.

Original language	English
Title of host publication	OTM Confederated International Conferences "On the Move to Meaningful Internet Systems"
Publisher	Springer
Publication date	2017
Pages	21-36
DOIs	https://doi.org/10.1007/978-3-319-69462-7_2
Publication status	Published - 2017
Event	On the Move to Meaningful Internet Systems - Rhodos, Greece Duration: 23 Oct 2017 → 27 Oct 2017

Conference

Conference	On the Move to Meaningful Internet Systems
Country/Territory	Greece
City	Rhodos
Period	23/10/2017 → 27/10/2017

Series	Lecture Notes in Computer Science
Volume	10573
ISSN	0302-9743

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.1007/978-3-319-69462-7_2

CoopIS2017 draftAccepted author manuscript, 881 KB

OpenUrl availability

Full text

Cite this

@inproceedings{934b7d64a6be4e1084de7caf8c14d80d,

title = "A Scalable Smart Meter Data Generator Using Spark",

abstract = "Today, smart meters are being used worldwide. As a matter of fact smart meters produce large volumes of data. Thus, it is important for smart meter data management and analytics systems to process petabytes of data. Benchmarking and testing of these systems require scalable data, however, it can be challenging to get large data sets due to privacy and/or data protection regulations. This paper presents a scalable smart meter data generator using Spark that can generate realistic data sets. The proposed data generator is based on a supervised machine learning method that can generate data of any size by using small data sets as seed. Moreover, the generator can preserve the characteristics of data with respect to consumption patterns and user groups. This paper evaluates the proposed data generator in a cluster based environment in order to validate its effectiveness and scalability.",

author = "Nadeem Iftikhar and Xiufeng Liu and Sergiu Danalachi and Finn Nordbjerg and Jens Vollesen",

year = "2017",

doi = "10.1007/978-3-319-69462-7_2",

language = "English",

series = "Lecture Notes in Computer Science",

publisher = "Springer",

pages = "21--36",

booktitle = "OTM Confederated International Conferences {"}On the Move to Meaningful Internet Systems{"}",

note = "On the Move to Meaningful Internet Systems, OTM 2017 ; Conference date: 23-10-2017 Through 27-10-2017",

}

Iftikhar, N, Liu, X, Danalachi, S, Nordbjerg, F & Vollesen, J 2017, A Scalable Smart Meter Data Generator Using Spark. in OTM Confederated International Conferences "On the Move to Meaningful Internet Systems". Springer, Lecture Notes in Computer Science, vol. 10573, pp. 21-36, On the Move to Meaningful Internet Systems, Rhodos, Greece, 23/10/2017. https://doi.org/10.1007/978-3-319-69462-7_2

A Scalable Smart Meter Data Generator Using Spark. / Iftikhar, Nadeem; Liu, Xiufeng; Danalachi, Sergiu et al.
OTM Confederated International Conferences "On the Move to Meaningful Internet Systems". Springer, 2017. p. 21-36 (Lecture Notes in Computer Science, Vol. 10573).

Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review

TY - GEN

T1 - A Scalable Smart Meter Data Generator Using Spark

AU - Iftikhar, Nadeem

AU - Liu, Xiufeng

AU - Danalachi, Sergiu

AU - Nordbjerg, Finn

AU - Vollesen, Jens

PY - 2017

Y1 - 2017

N2 - Today, smart meters are being used worldwide. As a matter of fact smart meters produce large volumes of data. Thus, it is important for smart meter data management and analytics systems to process petabytes of data. Benchmarking and testing of these systems require scalable data, however, it can be challenging to get large data sets due to privacy and/or data protection regulations. This paper presents a scalable smart meter data generator using Spark that can generate realistic data sets. The proposed data generator is based on a supervised machine learning method that can generate data of any size by using small data sets as seed. Moreover, the generator can preserve the characteristics of data with respect to consumption patterns and user groups. This paper evaluates the proposed data generator in a cluster based environment in order to validate its effectiveness and scalability.

AB - Today, smart meters are being used worldwide. As a matter of fact smart meters produce large volumes of data. Thus, it is important for smart meter data management and analytics systems to process petabytes of data. Benchmarking and testing of these systems require scalable data, however, it can be challenging to get large data sets due to privacy and/or data protection regulations. This paper presents a scalable smart meter data generator using Spark that can generate realistic data sets. The proposed data generator is based on a supervised machine learning method that can generate data of any size by using small data sets as seed. Moreover, the generator can preserve the characteristics of data with respect to consumption patterns and user groups. This paper evaluates the proposed data generator in a cluster based environment in order to validate its effectiveness and scalability.

U2 - 10.1007/978-3-319-69462-7_2

DO - 10.1007/978-3-319-69462-7_2

M3 - Article in proceedings

T3 - Lecture Notes in Computer Science

SP - 21

EP - 36

BT - OTM Confederated International Conferences "On the Move to Meaningful Internet Systems"

PB - Springer

T2 - On the Move to Meaningful Internet Systems

Y2 - 23 October 2017 through 27 October 2017

ER -

A Scalable Smart Meter Data Generator Using Spark

Abstract

Conference

UN SDGs

Access to Document

OpenUrl availability

Fingerprint

Cite this