Klaviyo Data Science Podcast

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into...

Thinking big-picture with A/B testing

We’ve discussed A/B testing multiple times on this podcast, for good reason. But there’s an important angle we have yet to cover: in the life of a researcher or marketer, there’s no such thing as an A/B test. There’s an entire system of A/B tests run for specific purposes over time. What is the best way to construct a system of A/B tests to help you learn, improve, and grow over time? How does that translate into tenets to hold while building software to help people run A/B tests? We’ve brought on three members of the data science team at Klaviyo, and you’ll hear about A/B tests in a variety of ways, including:

• Real data-driven trends observed by successful A/B testers on Klaviyo
• Why up-front thinking and vision translate into long-term success
• Why dad jokes might be far more powerful than you think

Check out the full show notes on Medium for more information!

This month, we return to a classic Klaviyo Data Science Podcast series: books every data scientist (and software engineer) should read. This episode focuses on the Clean * duology by Robert C. Martin, which teaches the principles of both clean code and clean architecture. We’ve brought on two senior engineers at Klaviyo who’ve learned, practiced, and developed their own opinions on the lessons in these books. Listen in to learn:

• How to use these books to level up your own skills and the skills of your team
• Why the book’s spiciest opinions make sense, and where you might disagree with them in practice
• What our panel’s deepest, most intimate thoughts on docstrings are

For more details, including links to these books, check out the ⁠full writeup on Medium⁠!

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into...

Anomaly Detection

It’s our third November on the Klaviyo Data Science Podcast, and if you work in ecommerce, you know that November means one thing: Black Friday and (usually) Cyber Monday, i.e. the month of the year where everything changes. Traditionally, we’ve talked about things that help prepare builders of software for when the world is about to change, such as infrastructure, readiness, scale-out testing, and other things along those lines. This year, we’re approaching it from another angle: ecommerce stores go through the exact same struggle every year. How can a platform like Klaviyo help prepare them for the unexpected? One answer: by automatically figuring out when unexpected things are happening, i.e., by detecting anomalous behavior. You’ll hear all about anomaly detection on this episode, including:

• How to pivot your research when your idea is valuable but your results aren’t providing that value
• How to label large swaths of data efficiently
• How to design algorithms for an extraordinarily diverse base of end users

Read the full show notes on Medium!

Welcome to the November episode of the Klaviyo Data Science Podcast for 2024! In years past, November episodes reflected the chaotic Black Friday/Cyber Monday season by examining unique challenges of readiness, scale, and fundamental changes happening with little to no warning, as well as how those challenges were handled; this November is no different.

What happens when two of the largest email platforms make sweeping changes to their spam filters, providing a few short months of notice? Stress, uncertainty, and an opportunity for individuals and organization to rise to the challenge. In this month’s episode, we talk with analysts, engineers, and product managers to discuss Klaviyo’s journey to meet Yahoo and Google’s new Email Delivery Requirements — aka Yahoogle, the colloqial name for a new set of rules that must be followed by email senders to have their emails make it to inboxes and not go straight to the junk bin.

Listen in to hear more about:

• Yahoogle. It’s more than just a funny word!
• The challenges in operating a large-scale email sending system
• Advice and retrospectives about large, cross-functional projects and tight deadlines

For the full show notes, including who's who, see the ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Medium writeup⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠.

This month, the Klaviyo Data Science Podcast welcomes Evan Miller to deliver a seminar on his recently published paper, Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations! This episode is a mix of a live seminar Evan gave to the team at Klaviyo and an interview we conducted with him afterward.

Suppose you’re trying to understand the performance of an AI model — maybe one you built or fine-tuned and are comparing to state-of-the-art models, maybe one you’re considering loading up and using for a project you’re about to start. If you look at the literature today, you can get a sense of what the average performance for the model is on an evaluation or set of tasks. But often, that’s unfortunately the extent of what it’s possible to learn —there is much less emphasis placed on the variability or uncertainty inherent to those estimates. And as anyone who’s worked with a statistical model in the past can affirm, variability is a huge part of why you might choose to use or discard a model.

This seminar explores how to best compute, summarize, and display estimates of variability for AI models. Listen along to hear about topics like:

• Why the Central Limit Theorem you learned about in Stats 101 is still relevant with the most advanced AI models developed today
• How to think about complications of classic assumptions, such as measurement error or clustering, in the AI landscape
• When to do a sample size calculation for your AI model, and how to do it

About Evan Miller

You may already know our guest Evan Miller from his fantastic blog, which includes his celebrated A/B testing posts, such as “How not to run an A/B test.” You may also have used his A/B testing tools, such as the sample size calculator. Evan currently works as a research scientist at Anthropic.

About Anthropic

Per Anthropic’s website:

You can find more information about Anthropic, including links to their social media accounts, on the company website.

Anthropic is an AI safety and research company based in San Francisco. Our interdisciplinary team has experience across ML, physics, policy, and product. Together, we generate research and create reliable, beneficial AI systems.

Special thanks to Chris Murphy at Klaviyo for organizing this seminar and making this episode possible!

For the full show notes, including who's who, see the ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Medium writeup⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠.