Tech on the Rocks

In this episode, we chat with David Jayatillake, VP of AI at Cube, about semantic layers and their crucial role in making AI work reliably with data.

We explore how semantic layers act as a bridge between raw data and business meaning, and why they're more practical than pure knowledge graphs.

David shares insights from his experience at Delphi Labs, where they achieved 100% accuracy in natural language data queries by combining semantic layers with AI, compared to just 16% accuracy with direct text-to-SQL approaches.

We discuss the challenges of building and maintaining semantic layers, the importance of proper naming and documentation, and how AI can help automate their creation.

Finally, we explore the future of semantic layers in the context of AI agents and enterprise data systems, and learn about Cube's upcoming AI-powered features for 2025.

00:00 Introduction to AI and Semantic Layers
05:09 The Evolution of Semantic Layers Before and After AI
09:48 Challenges in Implementing Semantic Layers
14:11 The Role of Semantic Layers in Data Access
18:59 The Future of Semantic Layers with AI
23:25 Comparing Text to SQL and Semantic Layer Approaches
27:40 Limitations and Constraints of Semantic Layers
30:08 Understanding LLMs and Semantic Errors
35:03 The Importance of Naming in Semantic Layers
37:07 Debugging Semantic Issues in LLMs
38:07 The Future of LLMs as Agents
41:53 Discovering Services for LLM Agents
50:34 What's Next for Cube and AI Integration

Summary

In this episode, Apurva Mehta, co-founder and CEO of Responsive, recounts his extensive journey in stream processing—from his early work at LinkedIn and Confluent to his current venture at Responsive.

He explains how stream processing evolved from simple event ingestion and graph indexing to powering complex, stateful applications such as search indexing, inventory management, and trade settlement.

Apurva clarifies the often-misunderstood concept of “real time,” arguing that low latency (often in the one- to two-second range) is more accurate for many applications than the instantaneous response many assume. He delves into the challenges of state management, discussing the limitations of embedded state stores like RocksDB and traditional databases (e.g., Postgres) when faced with high update rates and complex transactional requirements.

The conversation also covers the trade-offs between SQL-based streaming interfaces and more flexible APIs, and how Responsive is innovating by decoupling state from compute—leveraging remote state solutions built on object stores (like S3) with specialized systems such as SlateDB—to improve elasticity, cost efficiency, and operational simplicity in mission-critical applications.

Chapters

00:00 Introduction to Apurva Mehta and Streaming Background
08:50 Defining Real-Time in Streaming Contexts
14:18 Challenges of Stateful Stream Processing
19:50 Comparing Streaming Processing with Traditional Databases
26:38 Product Perspectives on Streaming vs Analytical Systems
31:10 Operational Rigor and Business Opportunities
38:31 Developers' Needs: Beyond SQL
45:53 Simplifying Infrastructure: The Cost of Complexity
51:03 The Future of Streaming Applications

Click here to view the episode transcript.

From GPU computing pioneer to Kubernetes architect, Brian Grant takes us on a fascinating journey through his career at the forefront of systems engineering. In this episode, we explore his early work on GPU compilers in the pre-CUDA era, where he tackled unique challenges in high-performance computing when graphics cards weren't yet designed for general computation. Brian then shares insights from his time at Google, where he helped develop Borg and later became the original lead architect of Kubernetes. He explains key architectural decisions that shaped Kubernetes, from its extensible resource model to its approach to service discovery, and why they chose to create a rich set of abstractions rather than a minimal interface. The conversation concludes with Brian's thoughts on standardization challenges in cloud infrastructure and his vision for moving beyond infrastructure as code, offering valuable perspective on both the history and future of distributed systems.

Links:
Brian Grant LI

Chapters

00:00 Introduction and Background
03:11 Early Work in High-Performance Computing
06:21 Challenges of Building Compilers for GPUs
13:14 Influential Innovations in Compilers
31:46 The Future of Compilers
33:11 The Rise of Niche Programming Languages
34:01 The Evolution of Google's Borg and Kubernetes
39:06 Challenges of Managing Applications in a Dynamically Scheduled Environment
48:12 The Need for Standardization in Application Interfaces and Management Systems
01:00:55 Driving Network Effects and Creating Cohesive Ecosystems
Click here to view the episode transcript.

In this episode, we chat with Ben, founder of Espresso AI, about his journey from building Excel Python integrations to optimizing data warehouse compute costs.

We explore his experience at companies like Uber and Google, where he worked on everything from distributed systems to ML and storage infrastructure.

We learn about the evolution of his latest venture, which started as a C++ compiler optimization project and transformed into a system for optimizing Snowflake workloads using ML.

Ben shares insights about applying LLMs to SQL optimization, the challenges of verified code transformation, and the importance of formal verification in ML systems. Finally, we discuss his practical approach to choosing ML models and the critical lesson he learned about talking to users before building products.

Chapters

00:00 Ben's Journey: From Startups to Big Tech
13:00 The Importance of Timing in Entrepreneurship
19:22 Consulting Insights: Learning from Clients
23:32 Transitioning to Big Tech: Experiences at Uber and Google
30:58 The Future of AI: End-to-End Systems and Data Utilization
35:53 Transitioning Between Domains: From ML to Distributed Systems
44:24 Espresso's Mission: Optimizing SQL with ML
51:26 The Future of Code Optimization and AI

Click here to view the episode transcript.

In this episode, we chat with Roy Ben-Alta, co-founder of Oakminer AI and former director at Meta AI Research, about his fascinating journey through the evolution of data infrastructure and AI. We explore his early days at AWS when cloud adoption was still controversial, his experience building large language models at Meta, and the challenges of training and deploying AI systems at scale. Roy shares valuable insights about the future of data warehouses, the emergence of knowledge-centric systems, and the critical role of data engineering in AI. We'll also hear his practical advice on building AI companies today, including thoughts on model evaluation frameworks, vendor lock-in, and the eternal "build vs. buy" decision. Drawing from his extensive experience across Amazon, Meta, and now as a founder, Roy offers a unique perspective on how AI is transforming traditional data infrastructure and what it means for the future of enterprise software.

Chapters

00:00 Introduction to Roy Benalta and AI Background
04:07 Warren Buffett Experience and MBA Insights
06:45 Lessons from Amazon and Meta Leadership
09:15 Early Days of AWS and Cloud Adoption
12:12 Redshift vs. Snowflake: A Data Warehouse Perspective
14:49 Navigating Complex Data Systems in Organizations
31:21 The Future of Personalized Software Solutions
32:19 Building Large Language Models at Meta
39:27 Evolution of Data Platforms and Infrastructure
50:50 Engineering Knowledge and LLMs
58:27 Build vs. Buy: Strategic Decisions for Startups