Promptfoo raises $18.4M Series A from Insight Partners and a16z to secure AI applications at the enterprise layer

just like AGM had on his wrist, you can go to getbzzel. com. Your bezel concierge is available now to source you any watch on the planet. Seriously, any watch. And we will bring in our next guest, Ian Webster. Welcome to the stream. How are you doing, Ian? Good to meet you. Good to meet you as well. I'm doing well.

Thanks so much. Thanks for joining. Would you mind kicking us off with an introduction on yourself, your company? What are you doing? What's the news? Yeah. So, um the the company that we're building is called Prompu. Um we are building security tools for Genai applications.

So, if you have a Gen AI application, you probably want to secure it, make sure it behaves well um in in in production. And that's that's what we're doing. Um the big news today I guess is that we have raised a series A. Congratulations. Give us the numbers. Yeah, we we raised 18. 4. Um congratulations. Yeah.

Uh who did you raise it from? Insight Partners. Uh and A6 also participated. Yeah. Two kind of smaller smaller firms, you know, they really had to scrap the money together to get this deal done. Yeah. I mean, um, I'm I'm very lucky. Yeah, it's fantastic. Well, congratulations on the round.

Uh, talk to me more about security in generative AI. That could mean so many things. We've talked about LLM psychosis, someone, prompts deep. We've talked about leaking user data from one instance to another. Uh, somebody like hallucinations and just kind of like not not performing.

I'm trying to, you know, use an LLM to convert raw text to JSON and it's just kind of messing up. Like what are the key areas where you would define a security that you're going after solving or are you trying to do everything all at once? So there are kind of two sides to the coin.

Um the first is uh the foundation model security or safety. Um basically stuff like is the LM going to say something racist that reflects badly on on my company? Like is it going to drive people into a psychosis? like all that type of stuff, brand risk basically. Yeah.

But I I think the this the part that's actually more interesting is at the application layer.

So um you know the like like once you put a model into the hands of a developer, they put an application on top of it, they connect it with a database and PII, they connect it with an with an API and now you've got a rag, you've got an agent, etc. Um that's when there are many ways to shoot yourself in the foot.

Um so the the biggest concerns that we see um I mean there there's definitely an aspect on on that uh foundation model side but the the big problem that scares a lot of large companies that are working with LMS are things like PII leaks tool misuse through agents um and also just thing softer issues like like LM recommending competitors or um sure you know like giving you medical advice when they're just supposed to be like an e-commerce chatbot, stuff like that.

Financial advice, too, would be a risk. Yeah. I mean, we were talking about Amazon's uh Amazon has like a chatbot that you can ask about the product that you're searching for or whatever, but then people were getting it to write like React modules and getting it to write JavaScript for them.

And it's like that's not a big deal, but it's just like clearly not the intended purpose of that particular chatbot.

What are what are the key what are the key reasons why you think over the long term companies will want to outsource a lot of this function versus build their own kind of systems to protect against some of these edge cases.

I'm sure there's a bunch of good reasons otherwise you wouldn't be building this and raising all this money. Yeah. So I mean it's it's it's pretty hard to do. Um so like I I encountered this problem firsthand because I was leading um Genai engineering and and and products at Discord.

we were launching these these apps to hundreds of millions of people and like to to do this right you really need to be able to train models that can behave adversarily because your GPT or your claude is not really going to be a good redte teamer or attacker off the shelf um and uh just kind of test uh extremely extensively like over tens or even hundreds of thousands of test cases to try to find all those rough edges.

Like the the thing about AI is that the attack surface is all of human language and and then some. So it's just it's a really difficult problem and the way that many like large enterprises are are dealing with this right now is um they are doing it manually and just covering a very small fraction of of the risk areas.

Um so you know that's why we think that there's big opportunity in the space. Yeah.

And if and if somebody engages with prompt fu, I'm assuming you start running like a do you start automatically like trying to jailbreak the LLMs getting like are you kind of running these sort of like repetitive uh attempts in order to like find problems so that you can stop them or or is the product itself kind of uh you you have it things figured out enough that you can sort of that you let them kind of like and their communities try to jailbreak it and then react to it.

Yeah. So the the way to think about it is that we've trained models that behave as adversarial users like misbehaving users in your application.

Um and we we have we have agents built on top of these uncensored models that um target specific risk areas like PI leaks or toxicity or like talk about competitors or you know what whatever 70 plus different areas.

Um, so the the way that we work is we have these attackers generate use cases um or like attack objectives and then um we feed those objectives into a bunch of different uh what I would describe as like searcher optimization techniques to kind of poke and prod um the entire attack surface of of the application.

So what what this amounts to is we wind up having um you know thousands and thousands of conversations where like the attacker is just trying to wiggle its way around some of the guard rails or some of the safeguards that are put in place.

Is there a big uh difference right now in the perception of security or ability to deliver on a secure application whether you're using a closed source or open- source model.

I can imagine like if I'm like oh I'm using an open source model I have the ability to fine-tune it maybe I feel more confident or maybe hey I'm using a closed source model I'm paying a fortune to open athropic like they're going to do more work to secure this thing than I am like what's the perception around like which which paradigm leads to better security I would say right now the perception at the at the top of market at least like we we work with some of the world's largest companies Um, there's definitely a strong preference for closed source right now, but I don't really think it's the security that's driving it necessarily.

I think it's more just performance per dollar and or just overall intelligence, even if it's more expensive. It's just like I want the best. Yeah. Um, I would say so.

Um I think like it the the the the thing is for for security even on the closed source side um the the incentives really differ based on whether you're open AI or a company building on top of open AI.

Like OpenAI has has plenty of geniuses who are who are working on um you know making sure that it doesn't say racist things or whatever. But like no one's going to stop you from shooting yourself in the foot by by hooking it up incorrectly to to like a database of PII. Yep.

So that's yeah or open AI open AAI is like you know they're not releas giving you this API and telling you we're making sure that it will never the model will never recommend a competitor like you have to kind of build a layer on top of that.

Has there been a big case study of like LLM security going gone wrong that people like to point to in the industry yet? Like we were just talking about the T app getting hacked and that did not have any generative AI features that I'm aware of.

It was potentially just made with a lot of maybe it was vibe coded, but it seemed like it was just a misconfigured Google Firebase bucket. They just changed the the access rights on an old database that they were using and it seems like something that could have happened 10 years ago. It happened today.

It's not a uniquely new phenomenon, but I'm wondering if there's any case studies that you point to is to to your clients like, hey, the value of working with us is that we can avoid that happening. Yeah, I mean there there have been a handful of things that that have been pretty public.

I think it it really depends on like the industry and also the geography. Um there was there was a mail carrier in in the UK that that like had had a lot of issues with with a chatbot that basically had no guardrails.

I think there was a case where um a a Canadian airline, I think Air Canada, had a chatbot that that like committed to a refund that was out of policy. That's right. I remember that one. Yep.

And then of course you have Sydney and Bing and Microsoft Tay and there's there's the fact that the like biggest companies and labs are struggling with this at that scale with that amount of resources. Can you imagine some random company with that rolling this out to to users?

They just don't have the the ability to stress test these things the same degree. Yeah. So my my not so hot take on this is that um like this this is really the blocker to for for big companies to use Gen AI.

Um like if if AI doesn't live up to the hype, it's because companies are are too scared to go public with their with their AI because of incidents like this.

um like we we encounter like Fortune50 companies that have hundreds of internal AI use cases but just haven't pulled the trigger on on like pushing these prototypes out there really because of this issue. So that's why I think this is like a a big bottleneck in this in this wave of AI right now. Yeah. Yeah.

It's way better to just take all of your employees, give them chat GPT Pro and say, "Hey, you're still responsible for everything that you all your work product, but you can use these LLMs as a tool to speed up your workflow, but ultimately, you know, even if Cloud Code is writing a lot of the software, you got to review the death and you got to review the email that gets sent to the important client.

So that, you know, we go from 99% accuracy, 1% hallucination to zero. That's your job now. " and you're going to do you're going to hopefully be faster, hopefully be better, but there's still a human in the loop in these things. Fascinating. Uh well, thank you so much for joining. Uh congratulations on the new round.

Uh great to see it. Thank you so much for stopping by. We'll talk to you soon. Appreciate it. Take care.