#13: The Netflix Recommender System and Beyond with Justin Basilico
02/15/23 • 80 min
This episode of Recsperts features Justin Basilico who is director of research and engineering at Netflix. Justin leads the team that is in charge of creating a personalized homepage. We learn more about the evolution of the Netflix recommender system from rating prediction to using deep learning, contextual multi-armed bandits and reinforcement learning to perform personalized page construction. Deep content understanding drives the creation of useful groupings of videos to be shown in a personalized homepage.
Justin and I discuss the misalignment of metrics as just one out of many elements that is making personalization still “super hard”. We hear more about the journey of deep learning for recommender systems where real usefulness comes from taking advantage of the variety of data besides pure user-item interactions, i.e. histories, content, and context. We also briefly touch on RecSysOps for detecting, predicting, diagnosing and resolving issues in a large-scale recommender systems and how it helps to alleviate item cold-start.
In the end of this episode, we talk about the company culture at Netflix. Key elements are freedom and responsibility as well as providing context instead of exerting control. We hear that being really comfortable with feedback is important for high-performance people and teams.
Enjoy this enriching episode of RECSPERTS - Recommender Systems Experts.
Chapters:
- (03:13) - Introduction Justin Basilico
- (07:37) - Evolution of the Netflix Recommender System
- (22:28) - Page Construction of the Personalized Netflix Homepage
- (32:12) - Misalignment of Metrics
- (37:36) - Experience with Deep Learning for Recommender Systens
- (48:10) - RecSysOps for Issue Detection, Diagnosis and Response
- (55:38) - Bandits Recommender Systems
- (01:03:22) - The Netflix Culture
- (01:13:33) - Further Challenges
- (01:15:48) - RecSys 2023 Industry Track
- (01:17:25) - Closing Remarks
Links from the Episode:
- Justin Basilico on Linkedin
- Justin Basilico on Twitter
- Netflix Research Publications
- The Netflix Tech Blog
- CONSEQUENCES+REVEAL Workshop at RecSys 2022
- Learning a Personalized Homepage (Alvino et al., 2015)
- Recent Trends in Personalization at Netflix (Basilico, 2021)
- RecSysOps: Best Practices for Operating a Large-Scale Recommender System (Saberian et al., 2022)
- Netflix Fourth Quarter 2022 Earnings Interview
- No Rules Rules - Netflix and the Culture of Reinvention (Hastings et al., 2020)
- Job Posting for Netflix' Recommendation Team
Papers:
- Steck et al. (2021): Deep Learning for Recommender Systems: A Netflix Case Study
- Steck et al. (2021): Negative Interactions for Improved Collaborative Filtering: Don't go Deeper, go Higher
- More et al. (2019): Recap: Designing a more Efficient Estimator for Off-policy Evaluation in Bandits with Large Action Spaces
- Bhattacharya et al. (2022): Augmenting Netflix Search with In-Session Adapted Recommendations
General Links:
- Follow me on Twitter: https://twitter.com/MarcelKurovski
- Send me your comments, questions and suggestions to [email protected]
- Podcast Website: https://www.recsperts.com/
This episode of Recsperts features Justin Basilico who is director of research and engineering at Netflix. Justin leads the team that is in charge of creating a personalized homepage. We learn more about the evolution of the Netflix recommender system from rating prediction to using deep learning, contextual multi-armed bandits and reinforcement learning to perform personalized page construction. Deep content understanding drives the creation of useful groupings of videos to be shown in a personalized homepage.
Justin and I discuss the misalignment of metrics as just one out of many elements that is making personalization still “super hard”. We hear more about the journey of deep learning for recommender systems where real usefulness comes from taking advantage of the variety of data besides pure user-item interactions, i.e. histories, content, and context. We also briefly touch on RecSysOps for detecting, predicting, diagnosing and resolving issues in a large-scale recommender systems and how it helps to alleviate item cold-start.
In the end of this episode, we talk about the company culture at Netflix. Key elements are freedom and responsibility as well as providing context instead of exerting control. We hear that being really comfortable with feedback is important for high-performance people and teams.
Enjoy this enriching episode of RECSPERTS - Recommender Systems Experts.
Chapters:
- (03:13) - Introduction Justin Basilico
- (07:37) - Evolution of the Netflix Recommender System
- (22:28) - Page Construction of the Personalized Netflix Homepage
- (32:12) - Misalignment of Metrics
- (37:36) - Experience with Deep Learning for Recommender Systens
- (48:10) - RecSysOps for Issue Detection, Diagnosis and Response
- (55:38) - Bandits Recommender Systems
- (01:03:22) - The Netflix Culture
- (01:13:33) - Further Challenges
- (01:15:48) - RecSys 2023 Industry Track
- (01:17:25) - Closing Remarks
Links from the Episode:
- Justin Basilico on Linkedin
- Justin Basilico on Twitter
- Netflix Research Publications
- The Netflix Tech Blog
- CONSEQUENCES+REVEAL Workshop at RecSys 2022
- Learning a Personalized Homepage (Alvino et al., 2015)
- Recent Trends in Personalization at Netflix (Basilico, 2021)
- RecSysOps: Best Practices for Operating a Large-Scale Recommender System (Saberian et al., 2022)
- Netflix Fourth Quarter 2022 Earnings Interview
- No Rules Rules - Netflix and the Culture of Reinvention (Hastings et al., 2020)
- Job Posting for Netflix' Recommendation Team
Papers:
- Steck et al. (2021): Deep Learning for Recommender Systems: A Netflix Case Study
- Steck et al. (2021): Negative Interactions for Improved Collaborative Filtering: Don't go Deeper, go Higher
- More et al. (2019): Recap: Designing a more Efficient Estimator for Off-policy Evaluation in Bandits with Large Action Spaces
- Bhattacharya et al. (2022): Augmenting Netflix Search with In-Session Adapted Recommendations
General Links:
- Follow me on Twitter: https://twitter.com/MarcelKurovski
- Send me your comments, questions and suggestions to [email protected]
- Podcast Website: https://www.recsperts.com/
Previous Episode
#12: From User Intent to Multi-Stakeholder Recommenders and Creator Economy with Rishabh Mehrotra
In this episode of Recsperts we talk to Rishabh Mehrotra, the Director of Machine Learning at ShareChat, about users and creators in multi-stakeholder recommender systems. We learn more about users intents and needs, which brings us to the important matter of user satisfaction (and dissatisfaction). To draw conclusions about user satisfaction we have to perceive real-time user interaction data conditioned on user intents. We learn that relevance does not imply satisfaction as well as that diversity and discovery are two very different concepts.
Rishabh takes us even further on his industry research journey where we also touch on relevance, fairness and satisfaction and how to balance them towards a fair marketplace. He introduces us into the creator economy of ShareChat. We discuss the post lifecycle of items as well as the right mixture of content and behavioral signals for generating recommendations that strike a balance between revenue and retention.
In the end, we also conclude our interview with the benefits of end-to-end ownership and accountability in industrial RecSys work and how it makes people independent and effective. We receive some advice for how to grow and strive in tough job market times.
Enjoy this enriching episode of RECSPERTS - Recommender Systems Experts.
Chapters:
- (03:44) - Introduction Rishabh Mehrotra
- (19:09) - Ubiquity of Recommender Systems
- (23:32) - Moving from UCL to Spotify Research
- (33:17) - Moving from Research to Engineering
- (36:33) - Recommendations in a Marketplace
- (46:24) - Discovery vs. Diversity and Specialists vs. Generalists
- (55:24) - User Intent, Satisfaction and Relevant Recommendations
- (01:09:48) - Estimation of Satisfaction vs. Dissatisfaction
- (01:19:10) - RecSys Challenges at ShareChat
- (01:27:58) - Post Lifecycle and Mixing Content with Behavioral Signals
- (01:39:28) - Detect Fatigue and Contextual MABs for Ad Placement
- (01:47:24) - Unblock Yourself and Upskill
- (02:00:59) - RecSys Challenge 2023 by ShareChat
- (02:02:36) - Farewell Remarks
Links from the Episode:
Papers:
- Mehrotra et al. (2017): Auditing Search Engines for Differential Satisfaction Across Demographics
- Mehrotra et al. (2018): Towards a Fair Marketplace: Counterfactual Evaluation of the trade-off between Relevance, Fairness & Satisfaction in Recommender Systems
- Mehrotra et al. (2019): Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
- Anderson et al. (2020): Algorithmic Effects on the Diversity of Consumption on Spotify
- Mehrotra et al. (2020): Bandit based Optimization of Multiple Objectives on a Music Streaming Platform
- Hansen et al. (2021): Shifting Consumption towards Diverse Content on Music Streaming Platforms
- Mehrotra (2021): Algorithmic Balancing of Familiarity, Similarity & Discovery in Music Recommendations
- Jeunen et al. (2022): Disentangling Causal Effects from Sets of Interventions in the Presence of Unobserved Confounders
General Links:
- Follow me on Twitter: https://twitter.com/LivesInAnalogia
- Send me your comments, questions and suggestions to [email protected]
- Podcast Website: https://www.recsperts.com/
Next Episode
#14: User Modeling and Superlinked with Daniel Svonava
In episode number 14 of Recsperts we talk to Daniel Svonava, CEO and Co-Founder of Superlinked, delivering user modeling infrastructure. In his former role he was a senior software engineer and tech lead at YouTube working on ad performance prediction and pricing.
We discuss the crucial role of user modeling for recommendations and discovery. Daniel presents two examples from YouTube’s ad performance forecasting to demonstrate the bandwidth of use cases for user modeling. We also discuss sources of information that fuel user models and additional personlization tasks that benefit from it like user onboarding. We learn that the tight combination of user modeling with (near) real-time updates is key to a sound personalized user experience.
Daniel also shares with us how Superlinked provides personalization as a service beyond ecommerce-centricity. Offering personalized recommendations of items and people across various industries and use cases is what sets Superlinked apart. In the end, we also touch on the major general challenge of the RecSys community which is rebranding in order to establish a more positive image of the field.
Enjoy this enriching episode of RECSPERTS - Recommender Systems Experts.
Chapters:
- (03:35) - Introduction Daniel Svonava
- (10:18) - Introduction to User Modeling
- (17:52) - User Modeling for YouTube Ads
- (35:43) - Real-Time Personalization
- (57:29) - ML Tooling for User Modeling and Real-Time Personalization
- (01:07:41) - Superlinked as a User Modeling Infrastructure
- (01:31:22) - Rebranding RecSys as Major Challenge
- (01:37:40) - Final Remarks
Links from the Episode:
- Daniel Svonava on LinkedIn
- Daniel Svonava on Twitter
- Superlinked - User Modeling Infrastructure
- The 2023 MAD (Machine Learning, Artificial Intelligence, Data Science) Landscape
- Eric Ries: The Lean Startup
- Rob Fitzpatrick: The Mom Test
Papers:
- Liu et al. (2022): Monolith: Real Time Recommendation System With Collisionless Embedding Table
- RSPapers Collection
General Links:
- Follow me on Twitter: https://twitter.com/MarcelKurovski
- Send me your comments, questions and suggestions to [email protected]
- Podcast Website: https://www.recsperts.com/
If you like this episode you’ll love
Episode Comments
Generate a badge
Get a badge for your website that links back to this episode
<a href="https://goodpods.com/podcasts/recsperts-recommender-systems-experts-537441/13-the-netflix-recommender-system-and-beyond-with-justin-basilico-69383368"> <img src="https://storage.googleapis.com/goodpods-images-bucket/badges/generic-badge-1.svg" alt="listen to #13: the netflix recommender system and beyond with justin basilico on goodpods" style="width: 225px" /> </a>
Copy