E01: Data Orchestration w/ Nick Schrock
09/13/22 • 21 min
Get an update from Nick Schrock on the dagster project. This episode discusses the world of data orchestration at large, when and how to implement dagster, and the broader competitive landscape.
Highlights:
[0:34] The Dagster project and the problems it solves
[3:05] Software defined assets and how they work
[4:55] Example of how this can be implemented
[6:30] Promotion of Dagster pipelines from development to production
[10:34] Dagster cloud overview
[13:15] When to adopt a data orchestrator
[14:23] Will data orchestration be subsumed into larger data platforms or remain standalone?
[18:10] The competitive landscape of data orchestration
--
👉 Don't forget to join us at Data Council 2025 in Oakland, CA - where 1000+ data & AI geeks, founders & investors join each other in building the future of data. This year, we're featuring top speakers from companies like DataDog, Meta, OpenAI, DuckDB, Notion and more!
Purchase tickets here for the biggest data & AI event of the year!
Get an update from Nick Schrock on the dagster project. This episode discusses the world of data orchestration at large, when and how to implement dagster, and the broader competitive landscape.
Highlights:
[0:34] The Dagster project and the problems it solves
[3:05] Software defined assets and how they work
[4:55] Example of how this can be implemented
[6:30] Promotion of Dagster pipelines from development to production
[10:34] Dagster cloud overview
[13:15] When to adopt a data orchestrator
[14:23] Will data orchestration be subsumed into larger data platforms or remain standalone?
[18:10] The competitive landscape of data orchestration
--
👉 Don't forget to join us at Data Council 2025 in Oakland, CA - where 1000+ data & AI geeks, founders & investors join each other in building the future of data. This year, we're featuring top speakers from companies like DataDog, Meta, OpenAI, DuckDB, Notion and more!
Purchase tickets here for the biggest data & AI event of the year!
Next Episode
E02: Metadata Architecture w/ Shirshanka Das
Shirshanka Das spent a decade leading data architecture at LinkedIn where he was the co-creator of DataHub, Apache Gobblin, Databus, Espresso and other high scale data infrastructure projects. Today he is the co-founder and CEO of Acryl Data, listen and learn what problems they are seeking to solve.
Highlights:
[1:10] Leaving LinkedIn and starting Acryl
[4:33] Datahub growth
[8:12] Where will the data catalog market segment be in 5 to 10 years?
[12:07] Unlimited context within the operating system
[16:08] Organizations adopting the right thinking behind metadata
[21:00] Is the data catalog a stand alone category of data infrastructure 10 years from now?
--
👉 Don't forget to join us at Data Council 2025 in Oakland, CA - where 1000+ data & AI geeks, founders & investors join each other in building the future of data. This year, we're featuring top speakers from companies like DataDog, Meta, OpenAI, DuckDB, Notion and more!
Purchase tickets here for the biggest data & AI event of the year!
If you like this episode you’ll love
Episode Comments
Generate a badge
Get a badge for your website that links back to this episode
<a href="https://goodpods.com/podcasts/zero-prime-podcast-221976/e01-data-orchestration-w-nick-schrock-25363744"> <img src="https://storage.googleapis.com/goodpods-images-bucket/badges/generic-badge-1.svg" alt="listen to e01: data orchestration w/ nick schrock on goodpods" style="width: 225px" /> </a>
Copy