MapReduce-based Dimensional ETL Made Easy

Xiufeng Liu, Christian Thomsen, Torben Bach Pedersen

Research output: Contribution to journalConference articleResearchpeer-review

Abstract

This paper demonstrates ETLMR, a novel dimensional Extract– Transform–Load (ETL) programming framework that uses Map- Reduce to achieve scalability. ETLMR has built-in native support of data warehouse (DW) specific constructs such as star schemas, snowflake schemas, and slowly changing dimensions (SCDs). This makes it possible to build MapReduce-based dimensional ETL flows very easily. The ETL process can be configured with only few lines of code. We will demonstrate the concrete steps in using ETLMR to load data into a (partly snowflaked) DW schema. This includes configuration of data sources and targets, dimension processing schemes, fact processing, and deployment. In addition, we also present the scalability on large data sets.
Original languageEnglish
JournalProceedings of the VLDB Endowment
Volume5
Issue number12
Pages (from-to)1882-1885
ISSN2150-8097
Publication statusPublished - 2012
Externally publishedYes
Event38th International Conference on Very Large Data Bases - Istanbul, Turkey
Duration: 27 Aug 201231 Aug 2012
Conference number: 38

Conference

Conference38th International Conference on Very Large Data Bases
Number38
CountryTurkey
CityIstanbul
Period27/08/201231/08/2012

Fingerprint Dive into the research topics of 'MapReduce-based Dimensional ETL Made Easy'. Together they form a unique fingerprint.

Cite this