Data Skew Profiling using HPCC Systems

Harsh Mishra, S Jayant, Arjuna Chala, Dan Camper, G Shobha, Jyoti Shetty

Research output: Chapter in Book/Report/Conference proceedingChapter

Abstract

Over the last few decades, there has been a tremendous increase in the volume of data available for analysis in various domains. Although processing power has scaled up as well, it is well known that the rate of increase of data far supersedes the higher processing capabilities of modern processors. The natural consequence to the advent of big data was distribution of data across multiple nodes to facilitate not only storage but also parallel processing. The advent of the age of large volumes of data came to be known as the era of big data. The distribution of data among various machines posed a fundamental problem in big data as well as distributed computing: The impact of data skew. We worked on a project to profile data skew on a multi-computing cluster. This paper summarizes our efforts and findings. We use HPCC Systems, a modern big data management and analysis tool. In this project, we analyze the impact of differently skewed data distributions on the most common database operations, namely, NORMALIZE, DENORMALIZE, JOIN, SORT, TABLE, and PROJECT using a set of queries, and analyzing their runtimes.

Original languageAmerican English
Title of host publicationICBDE 2019 - 2019 International Conference on Big Data and Education
Pages66-69
Number of pages4
ISBN (Electronic)9781450361866
DOIs
StatePublished - Mar 30 2019

Publication series

NameACM International Conference Proceeding Series

    Fingerprint

Keywords

  • Big data
  • Clusters
  • HPCC
  • Multicomputing
  • Skew

Cite this

Mishra, H., Jayant, S., Chala, A., Camper, D., Shobha, G., & Shetty, J. (2019). Data Skew Profiling using HPCC Systems. In ICBDE 2019 - 2019 International Conference on Big Data and Education (pp. 66-69). (ACM International Conference Proceeding Series). https://doi.org/10.1145/3322134.3322142