ETLMR: A Highly Scalable Dimensional ETL Framework Based on MapReduce

Xiufeng Liu, Christian Thomsen, Torben Bach Pedersen

Research output: Chapter in Book/Report/Conference proceedingArticle in proceedingsResearchpeer-review


Extract-Transform-Load (ETL) flows periodically populate data warehouses (DWs) with data from different source systems. An increasing challenge for ETL flows is processing huge volumes of data quickly. MapReduce is establishing itself as the de-facto standard for large-scale data-intensive processing. However, MapReduce lacks support for high-level ETL specific constructs, resulting in low ETL programmer productivity. This paper presents a scalable dimensional ETL framework, ETLMR, based on MapReduce. ETLMR has built-in native support for operations on DW-specific constructs such as star schemas, snowflake schemas and slowly changing dimensions (SCDs). This enables ETL developers to construct scalable MapReduce-based ETL flows with very few code lines. To achieve good performance and load balancing, a number of dimension and fact processing schemes are presented, including techniques for efficiently processing different types of dimensions. The paper describes the integration of ETLMR with aMapReduce framework and evaluates its performance on large realistic data sets. The experimental results show that ETLMR achieves very good scalability and compares favourably with other MapReduce data warehousing tools.
Original languageEnglish
Title of host publicationDataWarehousing and Knowledge Discovery. Proceedings
EditorsAlfredo Cuzzocrea, Umeshwar Dayal
Publication date2011
ISBN (Print)978-3-642-23543-6
ISBN (Electronic)978-3-642-23544-3
Publication statusPublished - 2011
Externally publishedYes
Event13th International Conference on Data Warehousing and Knowledge Discovery - Toulouse, France
Duration: 29 Aug 20112 Sep 2011
Conference number: 13


Conference13th International Conference on Data Warehousing and Knowledge Discovery
SeriesLecture Notes in Computer Science

Fingerprint Dive into the research topics of 'ETLMR: A Highly Scalable Dimensional ETL Framework Based on MapReduce'. Together they form a unique fingerprint.

Cite this