Real-Time Data Transformation and Analytics with dbt Labs

Streaming Audio: Apache Kafka® & Real-Time Data

02/22/23 • 43 min

dbt is known as being part of the Modern Data Stack for ELT processes. Being in the MDS, dbt Labs believes in having the best of breed for every part of the stack. Oftentimes folks are using an EL tool like Fivetran to pull data from the database into the warehouse, then using dbt to manage the transformations in the warehouse. Analysts can then build dashboards on top of that data, or execute tests.
It’s possible for an analyst to adapt this process for use with a microservice application using Apache Kafka® and the same method to pull batch data out of each and every database; however, in this episode, Amy Chen (Partner Engineering Manager, dbt Labs) tells Kris about a better way forward for analysts willing to adopt the streaming mindset: Reusable pipelines using dbt models that immediately pull events into the warehouse and materialize as materialized views by default.

dbt Labs is the company that makes and maintains dbt. dbt Core is the open-source data transformation framework that allows data teams to operate with software engineering’s best practices. dbt Cloud is the fastest and most reliable way to deploy dbt.
Inside the world of event streaming, there is a push to expand data access beyond the programmers writing the code, and towards everyone involved in the business. Over at dbt Labs they’re attempting something of the reverse— to get data analysts to adopt the best practices of software engineers, and more recently, of streaming programmers. They’re improving the process of building data pipelines while empowering businesses to bring more contributors into the analytics process, with an easy to deploy, easy to maintain platform. It offers version control to analysts who traditionally don’t have access to git, along with the ability to easily automate testing, all in the same place.
In this episode, Kris and Amy explore:

How to revolutionize testing for analysts with two of dbt’s core functionalities
What streaming in a batch-based analytics world should look like
What can be done to improve workflows
How to democratize access to data for everyone in the business

EPISODE LINKS

02/22/23 • 43 min

Streaming Audio: Apache Kafka® & Real-Time Data - Real-Time Data Transformation and Analytics with dbt Labs

Transcript

Kris Jenkins (00:00):

In this week's Streaming Audio, we're speaking to Amy Chen of dbt Labs, and I had great fun recording this one. Their perspective on the world we're all trying to get to is really interesting. Much like us over here at Confluent, they're deeply immersed in this idea of building data pipelines and thinking about all the different fronts on which you can try and improve that process. But we're each coming at it from very different beginnings.

Kris Jenkins (00:28

Episode Comments

0.0

out of 5

No ratings yet

Join the conversation

Post

Generate a badge

Get a badge for your website that links back to this episode

Select type & size

Copy