Augmenting Incident Response with LLMs
07/26/24 • 40 min
In this episode of the AI + a16z podcast, Command Zero cofounder and CTO Dean de Beer joins a16z's Joel de la Garza and Derrick Harris to discuss the benefits of training large language models on security data, as well as the myriad factors product teams need to consider when building on LLMs.
Here's an excerpt of Dean discussing the challenges and concerns around scaling up LLMs:
"Scaling out infrastructure has a lot of limitations: the APIs you're using, tokens, inbound and outbound, the cost associated with that — the nuances of the models, if you will. And not all models are created equal, and they oftentimes are very good for specific use cases and they might not be appropriate for your use case, which is why we tend to use a lot of different models for our use cases . . .
"So your use cases will heavily determine the models that you're going to use. Very quickly, you'll find that you'll be spending more time on the adjacent technologies or infrastructure. So, memory management for models. How do you go beyond the context window for a model? How do you maintain the context of the data, when given back to the model? How do you do entity extraction so that the model understands that there are certain entities that it needs to prioritize when looking at new data? How do you leverage semantic search as something to augment the capabilities of the model and the data that you're ingesting?
"That's where we have found that we spend a lot more of our time today than on the models themselves. We have found a good combination of models that run our use cases; we augment them with those adjacent technologies."
Learn more:
Follow everyone on social media:
Check out everything a16z is doing with artificial intelligence here, including articles, projects, and more podcasts.
In this episode of the AI + a16z podcast, Command Zero cofounder and CTO Dean de Beer joins a16z's Joel de la Garza and Derrick Harris to discuss the benefits of training large language models on security data, as well as the myriad factors product teams need to consider when building on LLMs.
Here's an excerpt of Dean discussing the challenges and concerns around scaling up LLMs:
"Scaling out infrastructure has a lot of limitations: the APIs you're using, tokens, inbound and outbound, the cost associated with that — the nuances of the models, if you will. And not all models are created equal, and they oftentimes are very good for specific use cases and they might not be appropriate for your use case, which is why we tend to use a lot of different models for our use cases . . .
"So your use cases will heavily determine the models that you're going to use. Very quickly, you'll find that you'll be spending more time on the adjacent technologies or infrastructure. So, memory management for models. How do you go beyond the context window for a model? How do you maintain the context of the data, when given back to the model? How do you do entity extraction so that the model understands that there are certain entities that it needs to prioritize when looking at new data? How do you leverage semantic search as something to augment the capabilities of the model and the data that you're ingesting?
"That's where we have found that we spend a lot more of our time today than on the models themselves. We have found a good combination of models that run our use cases; we augment them with those adjacent technologies."
Learn more:
Follow everyone on social media:
Check out everything a16z is doing with artificial intelligence here, including articles, projects, and more podcasts.
Previous Episode
Scaling AI for the Coming Data Deluge
In this episode of the AI + a16z podcast, Anyscale cofounder and CEO Robert Nishihara joins a16z's Jennifer Li and Derrick Harris to discuss the challenges of training and running AI models at scale; how a focus on video models — and the huge amount of data involved — will change generative AI models and infrastructure; and the unique experience of launching a company out of the UC-Berkeley Sky Computing Lab (the successor to RISElab and AMPLab).
Here's a sample of the discussion, where Robert explains how generative AI has turbocharged the appetite for AI capabilities within enterprise customers:
"Two years ago, we would talk to companies, prospective customers, and AI just wasn't a priority. It certainly wasn't a company-level priority in the way that it is today. And generative AI is the reason a lot of companies now reach out to us . . . because they know that succeeding with AI is essential for their businesses, it's essential for their competitive advantage.
"And time to market matters for them. They don't want to spend a year hiring an AI infrastructure team, building up a 20-person team to build all of the internal infrastructure, just to be able to start to use generative AI. That's something they want to do today."
At another point in the discussion, he notes on this same topic:
"One dimension where we try to go really deep is on the developer experience and just enabling developers to be more productive. This is a complaint we hear all the time with machine learning teams or infrastructure teams: They'll say that they hired all these machine learning people, but then the machine learning people are spending all of their time managing clusters or working on the infrastructure. Or they'll say that it takes 6 weeks or 12 weeks to get a model to transition from development to production . . . Or moving from a laptop to the cloud, and to go from single machine to scaling — these are expensive handoffs often involve rewriting a bunch of code."
Learn more:
Follow everyone on X:
Check out everything a16z is doing with artificial intelligence here, including articles, projects, and more podcasts.
Next Episode
Democratizing Generative AI Red Teams
In this episode of the AI + a16z podcast, a16z General Partner Anjney Midha speaks with PromptFoo founder and CEO Ian Webster about the importance of red-teaming for AI safety and security, and how bringing those capabilities to more organizations will lead to safer, more predictable generative AI applications. They also delve into lessons they learned about this during their time together as early large language model adopters at Discord, and why attempts to regulate AI should focus on applications and use cases rather than models themselves.
Here's an excerpt of Ian laying out his take on AI governance:
"The reason why I think that the future of AI safety is open source is that I think there's been a lot of high-level discussion about what AI safety is, and some of the existential threats, and all of these scenarios. But what I'm really hoping to do is focus the conversation on the here and now. Like, what are the harms and the safety and security issues that we see in the wild right now with AI? And the reality is that there's a very large set of practical security considerations that we should be thinking about.
"And the reason why I think that open source is really important here is because you have the large AI labs, which have the resources to employ specialized red teams and start to find these problems, but there are only, let's say, five big AI labs that are doing this. And the rest of us are left in the dark. So I think that it's not acceptable to just have safety in the domain of the foundation model labs, because I don't think that's an effective way to solve the real problems that we see today.
"So my stance here is that we really need open source solutions that are available to all developers and all companies and enterprises to identify and eliminate a lot of these real safety issues."
Learn more:
Securing the Black Box: OpenAI, Anthropic, and GDM Discuss
Security Founders Talk Shop About Generative AI
California's Senate Bill 1047: What You Need to Know
Follow everybody on X:
Check out everything a16z is doing with artificial intelligence here, including articles, projects, and more podcasts.
If you like this episode you’ll love
Episode Comments
Generate a badge
Get a badge for your website that links back to this episode
<a href="https://goodpods.com/podcasts/ai-a16z-379277/augmenting-incident-response-with-llms-63874683"> <img src="https://storage.googleapis.com/goodpods-images-bucket/badges/generic-badge-1.svg" alt="listen to augmenting incident response with llms on goodpods" style="width: 225px" /> </a>
Copy