Data Skew Profiling using HPCC Systems

Harsh Mishra, S Jayant, Arjuna Chala, Dan Camper, G Shobha, Jyoti Shetty

Research output: Chapter in Book/Report/Conference proceedingChapterpeer-review


Over the last few decades, there has been a tremendous increase in the volume of data available for analysis in various domains. Although processing power has scaled up as well, it is well known that the rate of increase of data far supersedes the higher processing capabilities of modern processors. The natural consequence to the advent of big data was distribution of data across multiple nodes to facilitate not only storage but also parallel processing. The advent of the age of large volumes of data came to be known as the era of big data. The distribution of data among various machines posed a fundamental problem in big data as well as distributed computing: The impact of data skew. We worked on a project to profile data skew on a multi-computing cluster. This paper summarizes our efforts and findings. We use HPCC Systems, a modern big data management and analysis tool. In this project, we analyze the impact of differently skewed data distributions on the most common database operations, namely, NORMALIZE, DENORMALIZE, JOIN, SORT, TABLE, and PROJECT using a set of queries, and analyzing their runtimes.

Original languageAmerican English
Title of host publicationICBDE 2019 - 2019 International Conference on Big Data and Education
Number of pages4
ISBN (Electronic)9781450361866
StatePublished - Mar 30 2019

Publication series

NameACM International Conference Proceeding Series


  • Big data
  • Clusters
  • HPCC
  • Multicomputing
  • Skew


Dive into the research topics of 'Data Skew Profiling using HPCC Systems'. Together they form a unique fingerprint.
  • Automated Data Skew Profiler

    Mishra, H. & Jayant, S.


    Project: Research

Cite this