Lee McDonald

Moving Towards Real-Time OLAP - With First Principle Thinking and Guided by Access Patterns

In the rapidly evolving landscape of data analytics, moving towards Real-Time OLAP (Online Analytical Processing) is not just a choice but a necessity. This talk will leverage first principles thinking to break down the complexities of OLAP systems into their fundamental components. By doing so, we strip away legacy biases and pivot towards more efficient, purpose-built solutions. We will ground the audience’s thinking on how understanding access patterns—such as data size, shape, and velocity—can optimize our database designs and meet the demanding needs of real-time analytics. Our journey will spotlight ClickHouse, a game-changing columnar database designed for ultra-fast query performance and real-time data ingestion. We’ll explore its core features like efficient columnar storage, advanced data compression, and vectorized query execution, which collectively revolutionize how we handle and analyze large datasets in real-time. Additionally, we’ll highlight practical implementations such as materialized views and seamless integration with data streams, positioning ClickHouse as an undisputed leader in real-time OLAP. To illustrate the applications and benefits, we’ll re-architect a streaming pipeline for live sports analytics, showing how ClickHouse can transform user engagement and performance metrics into real-time insights. By understanding and aligning with access patterns, this new architecture allows for instant data processing and querying, providing unparalleled scalability and performance. This talk aims to equip you with actionable insights and methodologies to revolutionize your OLAP systems, keeping you ahead in the data-driven world.