Ben Hylak on Raindrop AI and why Claude Opus 4's willingness to 'call the police' on users signals a new era of AI autonomy

care. Talk to you soon. Bye. Uh do we have Ben Hilac or should we talk? He is in the waiting room. Oh, he's in the waiting room. Let's bring him in. Ben, welcome to the stream. How you doing, Ben? Is your computer being controlled by Anthropic? That is the question. They're kicking down.

Have you been doing illegal things? Yeah. Uh what's going on? It's great to have you on. Uh everyone knows uh do the Jaguar rebrand. Yeah, he did the Jaguar rebrand, but today we're going to talk about He did not do the Jaguar rebrand. For those who are listening, we're joking about that. I didn't know.

But if you say it enough, then ChachiBT will pick it up. They'll be like, "They've said it 10 times now. It can't still. " They might arrest you for that. Taking credit for a brand you didn't stolen Valor. I don't think anyone's going to have a problem with that. The SWAT team kicks down the door. Get out of here.

Anyways, great to have you on. Uh glad we could make this happen even though X is absolutely cooked today. It is really I've never I've never seen it quite this bad. It's rough. But um anyways, why don't you introduce yourself and then we can talk about that that post and and kind of get your reaction to it live. Yeah.

Uh so uh I was a designer at Apple for like four years. Uh before that I you know kind of have a weird background. I actually started off with like robotics and avionics. So um like internics a bunch of times and did those sort of those rounds.

But now um around a year ago, two years ago um started a company called Raindrop. And so we do essentially sentry for uh AI agents. Cool. Awesome. Break that down more more. Give us more context. Well, I mean, I think I think it's actually like it perfectly dubtales with this, right?

Which is like um we we get to work with some of the coolest companies in the world like uh Clay. com, just like the companies that are just really trailblazing uh when it comes to um AI applications. And what we do is we help them find sort of like really hard to spot failure cases.

So there's like uh for people that do AI stuff, there's this concept of eval which are almost like unit tests. So these are like, you know, given these test cases, you know, does it pass them? Does it fail them? Uh what our product does is in the real world, like what are users actual experiences with your product?

Where is it actually failing? Where is it doing things you you wouldn't expect, right? So um for example, we had a one of our customer uh one of our customers Colins uh they have this like alien companion. So it's like tolins. com. Oh yeah. Yeah. Yeah. You're familiar.

And so they had an issue where like their alien started referring to itself as like a dude from the United States, right? Uh which is unusual. One of us good thing. Total cultural victory. Except um and so their their issues like how big of an issue is this?

Like was that one time or was this like hundreds or thousands of times? And then like if they're going to try to fix it, like it's it's very like random, right? It's very it's very unpredictable. It's not always a clear root cause. So what they want to be able to do is kind of get a view of that issue over time now.

And if it ever regresses, it ever comes back get notified. Um so we make it really easy for companies to like literally just the craziest issues you ever That's Yeah, it's just such a funny example of of the aliens like, "Yeah, I'm actually from America. " America.

But yeah, it's funny because if you fine-tune that out, then you wind up with a potential situation where the the AI never adopts the personality of a of an American or doesn't even know the concept of America anymore because you you ripped it out too hard, right? So, it's a really tricky balance to now, right? Yeah.

Uh, and from like a detection standpoint, I think the thing that makes us kind of like I I think we have like actually one of the most complex like AI pipelines I've ever seen and just like research wise a lot of really cool stuff cooking like being able to categorize these messages as it is an alien character talking like misrepresenting himself as being from like the United States.

It's very hard, right? It's not like show me events where uh the assistant talked about a guy from the United States, right? it's not some sort of like keyword search or something like that. Like some of these categories actually get pretty complex.

Um, and you can imagine that that is an issue that's only relevant to that app. So it has no application. Like so our customers have to be able to go in there and like define what they're looking for and create this categorizer that can run on millions of messages a day very cheaply. Interesting. Yeah. Yeah.

I mean even in like the web 1. 0 web 2. 0 like you're almost building like a pager duty.

It used to just be is this page 404ing and the and the HTTP there's no clicks there's no yeah there's nothing like that it's it's so much more squishy but uh you know the problems created by AI also solved by AI uh you said you have to run uh these queries I imagine that they're LLM powered millions of times that sounds really expensive uh what are you doing are you baking llama 3 that's free onto an AS6 so you can just like run it super cheaply?

Are you looking at Grock with a Q or Crisus or something else that drops the inference cost? Are you on open router constantly trying to find the cheapest thing? Um have you distilled models? Like how do you control cost in that scenario? Yeah.

So um because of a lot of our customers volumes like if you think about like Clay. com or something like they're just you have millions and millions and millions of requests a day. So it's not actually feasible at all to send them to an LLM at every request. Yeah.

Um we can use LLMs like smaller ones like especially like Gemini flash to like do things like summarizing or like describing clusters or stuff like that. Um as far as actual detection doesn't really work well for that.

So we have a bunch of essentially um custom trained embedding models and then models on top of those that are small like uh they're like technically neural net but you can think of them like an SVM or something where they're just like really good at detect doing a first pass detection.

So around like usually 98% of the events or it can filter out like 95 98% of the events that are not relevant. Um and then kind of the last pass or like is like a small uh fine-tuned uh LM. What's it like training one of these smaller SVMs? Is this something like commodity NVIDIA GPU for like an hour?

It's just like simple. The smallest ones you can do pretty fast. Um embedding models are like longer. A lot of it's about having the data honestly. um having the right data to do it.

And the cool thing is that um and I kind of one of when I say like the pipeline's complex like one of the things I mean is that our customer each of our customers looks pretty different. What an issue is or is not is pretty different.

So for example, you can imagine that one customer like you have a customer support chatbot and you ask it to write code you don't you know like that they might even want to want it flagged that you know a user is asking them in the first place because that's like more like they're getting hacked or something.

Uh but a coding assistant obviously like that's the bread and butter. So uh one getting refused is okay and the other one isn't. Um so we actually train these models um on the fly for every single customer. Uh so that's uh one of the cool things we do. Uh there were four maybe more massive AI announcements this week.

You got Microsoft Build, you got Google IO, you got OpenAI buying IO for $6. 5 billion. Johnny IV going into Open AI building hardware. Uh you also got Anthropic launching Claude 4 today. Uh what stuck out to you? What was the most interesting story of the week?

What has you thinking I want to implement that I want to play around with that? I'm excited about that or I have a hot tick about that. Yeah. Yeah. Yeah. So I think that you know most of the times the things that excites me are the things that are for builders like uh you know we use AI a lot.

All of our customers use AI. That's sort of the lens we're always looking at it through. I think that Google has really done a good job of nailing that niche of like building models that for startups. Um I think that like Google Flash for example is the closest thing we have to like intelligence too cheap to meter.

Um obviously there's like four nano that was like a followup to that. Um and like in response to that uh but yeah flash is still just like superior in so many ways.

they have just the like throughput and just like the you know like they they pretty much have no you know rate limiting at some point when you when you pay it off like pretty much zero yeah because they're running on their own hardware down to the actual TPUs and so you know you have to imagine if you go to open AI models it's probably running on Azure and so there's an extra layer of networking and transport and all the data is flowing back so if you yeah if you're looking at like super super fast fine-tuned responses like Google infrastructure team is probably hard yeah exactly they they know how to do it Right.

So yeah. Uh and and so like for example like I think I think diffusion is like super interesting. I see a lot of applications for it like uh have played around with it a bunch and there's something um just wild about I I don't know if you guys are familiar with like the giant Yeah.

I saw one screenshot 900 tokens per second. Is it actually using a diffusion model instead of a transformer? Is that is that what's going on? Yeah. Yeah it is. Yeah. Okay. Wow. That's amazing. Yeah. If you see like illustrations of it.

It's essentially like actually like each part of the code it's actually you know being generated separately and kind of literally like instead of just predicting the next word next word next word it's pres predicting like the entire result at one one go fascinating I I I want to know more I I I I got to dig into like how big was the cluster that they trained on is it all synthetic data I mean they have so much data uh that is fascinating I had no idea that there would be a flow back because we're seeing in images in chat a flow forward right they're transformers and then we're going back.

But I guess it's like these are great algorithms. Let's use them in every single way in every single application and you'll probably see diffusion all over and transformers all over and everything and I think there's use cases where it works for there's use cases where it doesn't.

Like I think that um code actually I think lends itself pretty well to it in certain use cases. Like I think one of the key use cases they have in the demo uh that you can request access to is like essentially it'll oneshot a website like uh in like a second, you know what I mean?

Like if someone build a calendar and it would just one 3 seconds 3,000 uh 3,000 tokens I guess came out in 3 seconds. So, it's like a thousand tokens a second, which is crazy. Yeah, it it's pretty there's something really insane about it.

Like obviously like the apps it's actually generating are like not going to be as good yet, but like there's just something insane about going from like words to just like render.

Like there's there's this idea I've always been really fascinated with which is like let's say for games for example like is there ever a point where you're it's all just being generated? I think like Jensen has said something along the same the same lines. Oh, no, no, no, no.

There's somebody's actually running Minecraft in a Yeah, you've tried that one, right? Uh I think etched is in partnership with that. They they're building a chip that's uh has a transformer architecture baked onto it in silicon fab by TSMC and they can run Minecraft purely proced purely generative AI version.

There's no game engine whatsoever. It's just trained on uh on Minecraft. Fascinating. Let's talk about the post today that that that you shared and is picking up steam even though X is cooked. But uh I I already read it and I we actually got Keith for Boy's reaction.

Uh which I'll share as well because I think it's an important context.

Basically, uh, AI alignment researcher at Enthropic said, "If it thinks you're doing something egregiously immoral, like faking data in a pharmaceutical trial, it will use command line tools to contact the press, contact regulators, try to lock you out of relevant systems, or all the above.

" Keith made a good point, which is that uh technology companies have obligations uh that uh to various authorities that if they detect, you know, illegal activity happening that they have a responsibility.

I think the concern, the obvious concern here is, you know, somebody's like playing around with a model and then suddenly it's contacting the New York Times and being like Ben Hilac is uh, you know, faking pharmaceutical data, you know, or something like that. Uh, and a hit piece comes out instantly.

I mean, this hurts for me because I lie to Chat GPT all the time. I'm always in there. I'm always in there saying, "I am a train expert. Tell me about trains. I own dozens of trains. " just to get it to give me better, more rigorous responses and not talk to me like I'm a casual trained consumer.

Uh but but if it finds out I'm lying, it's going to be over for me. You're crazy. Yeah.

I think um I mean, first of all, uh you know, I don't have the full context of of TA's response, but you know, it's important to note that this was not that like this is not them saying that they have some sort of regulatory requirement defining what that is and and doing the minimum to to meet that requirement.

I think that would be different. And then there's some sort of like, you know, someone could sue the US government. There's some sort of path to recourse. So, um that that's interesting and true. Uh but um but feels a little bit different. Could have been messaged better. Yeah. Probably not in a random comment. Yeah.

You know, saying like basically saying like we're going to take control of your machine. Yeah.

and to carry out something without any type of the press too like there like there's no legal requirement to go to the press with any I know I know I know exactly exactly that's particularly crazy I totally get like look there's a law that says that if we see you doing wire fraud we have to report it to the SEC that makes perfect sense yeah that's not that's not what this is right this they we're going to call TMZ it's possible right so um and and they also just define as something, you know, um, you know, egregious, evil, uh, you know, which is, you know, morally, you know, immoral, right?

Which Yeah. Which is different than laws, right? Which is different than laws. I think that, like, I'll also be clear, this is not I I had a pretty strong reaction to it, actually. Uh, I I don't get that angry about things, but I actually felt really angry. Um, I I don't think it was just an off-hand comment, actually.

If you read their they kind of had a 128 page like model card where they explain all the model behavior. They talk about it pretty similarly, right? where they kind of say they they show that behavior that kind of uh how it can happen.

And um and that's not the concerning part because you know anybody that works with these models the fact it could do something erroneously like that under certain conditions like okay like these these models do crazy things but it's the way that they they talk about it right uh they say something like uh you know I'm going to misquote it but it's something along the lines of like know and this is you know probably appropriate behavior but it could happen in the wrong situations right I I think it like really struck me as like police state [ __ ] Like I was like, "Oh, okay.

" Like, it's like even the thread, right? It's kind of like, "Well, if your dress isn't too short, you'll be fine. " Uh, but maybe like, you know, maybe don't talk maybe don't, you know, write a story about threatening certain like it's just not how our country works. Yeah.

And uh yeah, I I I find it really really really deeply concerning. And I think I think that like AI safety as a whole, and this makes me sad actually because I think AI safety could be really good. Totally. I think it could be needed, but I think that it's kind of like safety from who or safety from what? Um Well, yeah.

And the whole idea that that there's zero human intervention, it's like it's like it's like we're not going to we're not going to check this and be like, "Oh, it's a 12year-old who's just like exploring space on the LLM. " And it's like, yeah, the it's just cat walks across the keyboard. The whole potential.

It's like, okay, amazing if you can identify bad actors. Yeah. And then you know work within the legal framework that we have of you the existing legal framework that doesn't necessarily need net new laws.

You just work within what government has already decided is you know laws are also things that it's not always like good or bad like laws are just laws. There is there is one exception.

I know a lot of people use these models as like personal trainers and I think it's deeply immoral to skip leg day and so if you found if a model found out that someone was skipping leg day they should call the press the press contact and the regulators and regulators yeah definitely the FDA and the SWAT team they they should break down your door yeah this this almost implies that like the model would swat you does yeah that that's what it implies and and it's and it's going to lock you out of relevant systems.

No, it's going to hack your computer. I can't even hack your computer. They literally said it's going to hack your computer. That's so wild. Well, the tweets's been deleted. So, it's it's hopefully they backtrack on that. Hopefully, it's not.

It's actually He deleted it and then rewrote something about how people were taking it out of context that was just doubling down like he was like that's that's I think the really concerning thing is that I you know I've written bad tweets before. We've all written bad tweets.

But when I see someone, you know, accuse people taking out of context and double down is really concerning. Um, so yeah. So, so credit to Sam. He says, "I deleted the earlier tweet on whistleblowing as it was being taken out of context. This isn't a new cla feature and it's not possible in normal usage.

It shows up in testing environments where we give it unusually free access to tools and very unusual instructions. " Okay.

Well, I think the issue is that probably within minutes of like releasing the model, you have thousands of people that gave it root access to their computer through like cursor through their you know um uh uh cursor uh cloud code etc.

So I think that like maybe one one thing we've learned from this is that like uh there was a time where it was the idea of hooking up a model to the internet was scary and that was two years ago. Yeah. You know, I mean, not even that. There was a time when there was a knowledge cut off. Remember this? Yeah. Exactly.

Oh, yeah. Yeah. Yeah. We don't even want you to know about the last three months. And now it's like it knows it it goes to every web page. It can definitely do get requests. There's definitely malicious things it could do with get requests, but it's pretty it's pretty responsive. And you know what?

Most of the time it just gets you the answer. This guy uh Justin Halford who I've deal with a bit says, "Can you imagine getting shot by the authorities in your own home because your philosophy homework contained a touchy topic or context that was misinterpreted as a request?

We really need to avoid such paranoid contexts altogether. This ain't it. " Yeah, it's a good tip. Very rough. Anyway, hopefully they sort it out. Hopefully there's more discussion here. Uh anything else you want to close out with? It's been great having you. No, it's been great being here. Fantastic. Do it again soon.

Let's talk. This is great. We'll talk to you soon. Bye. Uh I want to close out with uh some other interesting drama in the prediction market. Of course, we're sponsored by Poly Market, but this has been going back and forth on X, which is also where we distribute the show.

Uh so the official the official X handle just says, "Recent speculation about XAI's involvement in the prediction market space has been circulating. While we're enthusiastic about the potential of this industry and engaged in various discussions, no formal partnerships have been confirmed to date, stay tuned.

So, they're saying, you know, we might be in talks. We might even be in advance talks, but we haven't announced anything yet. So, if you're reading it on, uh, the internet, it's not confirmed yet. And so, uh, Mario Knoff, uh, broke this down. Khi walks back. XAI deal claim.

So, Colshi uh tweeted that um that XAI is doing a deal with Khi. Um but Bloomberg retracted the story after uh the after I guess X resend rescends a statement. So, turns out XAI says there's no deal. Kelshi rescended their own announcement hours later. No contract, no collab, no confirmation.

pretty pretty insane that both Bloomberg and X came out and were like, "This is not real. I don't even know how this happened. " And how did it happen in the first place? How did this happen in the first place?

Because like the last thing I would want to do is say I have a deal with Elon Musk when I don't like he doesn't seem like the type of person. Yeah. He doesn't seem like a person to be like, "Oh yeah, like you know, we did talk and like yeah, they're getting they're putting the cart cart before the horse a little bit.

" It's like no, he he cares. He Elon doesn't let a lot of companies that sell to SpaceX put the SpaceX logo on their website, you know, like every company, they have to say, "Yeah, we work with a big space company. " Yeah. Yeah. And it's like, "Okay, we know exactly what you're talking about.

" But yeah, I mean, Elon's like like if you're going to do a joint press release with Elon, like you better have his buy in. So, this is a very weird weird thing to even have happen. And I guess Bloomberg deleted the entire post, I guess, the entire article.

Um, but anyway, I mean, it's a knockout dragout fight to get integrations in the uh in the uh prediction market game. Yeah, it's just not great for KHI given they already came under fire for like trying to get AB to spread.

It's basically just like me chain, which is just sort of like uh competit, you know, comp competitive markets, but uh there's no need for foul play. Yeah. Yeah, it does seem a little bit a little bit too aggressive crossing the line.

But anyway, hopefully they can sort out the deal and everyone can kind of learn what the real strategy and prediction markets is with XAI. It would be interesting to see where I would like. We are the integration point for Poly Market into X because you see our live stream and you see the ticker right there.

And of course, Poly Market and Khi both post prediction market all the time. It's interesting to think about Grock being able to access poly markets for basically getting a read on future events. Right. Right now it's oriented around factchecking. Yep. But it also can give insight into potential headlines of the future.

Will this acquisition happen? What what's the likelihood? And I would imagine that as long as both companies have a robots. txt that's permissive, they would show up in chat GPT. They would show up in Grock.

I imagine that right now if I ask Grock pull up the pull up the poly market on the USS the US recession chances in 2025 it could just go do that. I would expect that that would be the behavior just because it has the ability to browse the web.

Um but obviously a deeper integration would be cool for whichever company can win it. So good luck to Shane as he goes on a tear and tries to build Poly Market into a generational company. Um let's go to Delian and close out here. Delian, I like this because uh in we're we're working on an article.

Well, it'll drop maybe tomorrow we'll be covering it here about TBPN and uh in the article in the fact check uh they hit us with uh is it true fact check is it true that you refer to your team as the guys all caps or or capital T capital G and I was like honestly like yes but we also call them the boys we also call them the gentlemen we also call them the production crew the crew whatever we don't really we haven't really formalized the the TVPN uh production team as the guys, but they are guys.

Um, but Delian put it into a great uh post. He says, "If the boys isn't a clearly identified group of 6 to 12 tight-knit men in your life, you're just not going to make it that far in life. " And uh yeah, every dude needs a group chat of guys. Give it a funny name.

Uh and more importantly, go and text your your the boys. Text the boys right now. Tell them where you're going to see Mission Impossible, The Final Reckoning in theaters. Get the tickets. Uh, send the Venmo requests. Get everyone into the theater to go see Tom Cruz do what he does best. There's a call to action.

There's a call to action. No matter what city you're in, hit the boys and say, off your tux, pick a date, get the tickets, just make it happen. And, you know, dress up, too. You got six to 12 tight-knit men in your life.

Uh maybe six show up because people are busy, but you refund the rest of the tickets and you're good to go. Yep. This is the playbook. This is the playbook. This is the playbook. Get the boys together for Mission Impossible. Fun show. Fun show. Give us five stars in Apple Podcast and Spotify. That is directly.