An (Updated) Overview of Data Provenance: Concepts, Challenges and Opportunities

Data provenance, also known as data lineage, tracks the origins, transformations, and movements of data throughout its lifecycle. This concept was popularized within data management in the early 2000s by Buneman, Khanna, and Tan to address the need to understand and verify data modifications in complex systems. Since then, data provenance has seen significant development, research, and application in various data systems, including relational and NoSQL databases, workflow systems, and scientific computing platforms. Moreover, there is a rise of explainable AI systems, regulatory frameworks demanding transparency (GDPR), and cloud architectures requiring lineage tracking. This growth reflects both industrial and academic interest in the field. Initially, data provenance focused on the relational model, with semantics expressed through why-provenance, where-provenance and, how-provenance. However, more recently, there has been a shift towards broader applications in machine learning pipelines, big data frameworks, knowledge graphs, and streaming systems, expanding beyond traditional relational databases. It has been (almost) 10 years since a tutorial on data provenance was presented at SIGMOD. To catch up with the most recent concepts, this tutorial delves into a comprehensive survey of data provenance. Specifically, we cover topics related to provenance and queries, the role of provenance in databases and we examine the concept of provenance and explainability.