Data Skew Profiling using HPCC Systems

Over the last few decades, there has been a tremendous increase in the volume of data available for analysis in various domains. Although processing power has scaled up as well, it is well known that the rate of increase of data far supersedes the higher processing capabilities of modern processors. The natural consequence to the advent of big data was distribution of data across multiple nodes to facilitate not only storage but also parallel processing. The advent of the age of large volumes of data came to be known as the era of big data. The distribution of data among various machines posed a fundamental problem in big data as well as distributed computing: The impact of data skew. We worked on a project to profile data skew on a multi-computing cluster. This paper summarizes our efforts and findings. We use HPCC Systems, a modern big data management and analysis tool. In this project, we analyze the impact of differently skewed data distributions on the most common database operations, namely, NORMALIZE, DENORMALIZE, JOIN, SORT, TABLE, and PROJECT using a set of queries, and analyzing their runtimes.