Ex-Meta trust and safety leader builds AI scam defense startup, warns 'all phishing becomes spear phishing'

Speaker 1: up next, we have Jeremy Gallin from Sham Charlemagne Labs. He spent twelve years in Meta in trust and safety and left last year to focus on AI powered scams and building defenses.

Speaker 2: We're doing a whole security themed show. Look at this. Jeremy. Look at this. Suited up. Wow.

Speaker 1: The matching suit. You look fantastic. Wow. You really like it is the mirror image of me. This is crazy. Nailed it. The memo came through. I was hoping that I'd get that Mabal break downstairs and Yeah. Near the studio.

Speaker 8: I'm so glad you you're you're you're up to speed on the show. But for those who aren't up to speed on you, give us an introduction and explain a little bit of your facts. John's intro, you said you said he left Meta to focus on AI scams, which kind of sounds which kind of sounds like you're scamming, but I'm assuming it's the exact No. It's the opposite. We're doing cyber scamming. That would be too easy. It's much easier to be on the offense than it is to be on the defense today. I tell you. Yeah. Exactly. It's it's it's wild out there. So, yeah, left Med after twelve years Mhmm. To focus on He's very good success. Right. And basically, my vision is that every employee of every company would have a watchdog. So the company's named after my dog, Charlemagne. She goes by Charlie, so the product is called Agent Charlie. Yeah. The idea is like, you're using your computer and you're getting attacked now with novel kinds of threats that resemble legitimate communication. That could be messaging apps. It could also be know, the standard phishing Yeah. We we just heard about the with the Axios attack.

Speaker 2: Yeah. It was a basically a fake Microsoft Teams, basically, call that then cut out and trigger and and, you know, suggested, hey, update Microsoft Teams. Whole thing wasn't Microsoft Teams, but Right. The individual just was, like, you know, confused because it just seemed like it was I think the nastiest trick is when it's the unsubscribe button is itself

Speaker 8: link. Think that's like the biggest thing. So what I we've built you know, the startup has been selling a product that will try and stop you from clicking. So it's like bad, bad employee. Do not click. But the the research that we've done to to inform this commercial product is into the capacity, the capability uplift that's happening with respect to offense. So it's important to remember that if you're an adversary, that's a threat actor seeking, you know, financial gain or a state actor Mhmm. You're availing yourselves of all this AI energetic tooling that we are using, you know, the sales tools, the, you know, the automation. And so, you know, the the core premise is that in in an AI powered world, all phishing becomes spear phishing. You're not going get a Nigerian prince email, you know, much anymore. You're going to get an extremely realistic, utterly compelling request from your boss or your manager or your friends, and it's going to be catastrophic in consequences.

Speaker 1: How do you think about actual deployment? Because this sounds useful in a consumer context. I'm just thinking about, you know, the the the email that's from your bank and has the unsubscribe button for some marketing email. You click it, all of a sudden you're logging in, giving away details. Is there an important distinction? It feels like consumer and enterprise is blurring together in in many places. How do you think this all plays out? Absolutely. I think

Speaker 8: as employees of companies, we are using personal email and personal messaging apps on our on our devices, for sure. Yeah. I think as a business, we're a b to b SaaS company Sure. With the research arm, and I'm excited to tell you more about our our research efforts. But, yeah, I mean, my dream is that AARP is listening right now and would give this, you know, for free I'd to like to give our software for free to anyone who holds an AARP card because elder abuse is devastating, and it has huge consequences. But it's very difficult to market and sell to consumers, know, a product like this. People don't wake up and say, today's the day I'm going to improve my security posture. It's sort of after their attack that they have a problem and a mess to clean up. So we're Well, you need to you need to create the problem No. Stop with solution.

Speaker 2: Stop with creating the problem. No one's creating problem. Just we were just talking with It's Barris from Socket who's who's saying, like, the the old tinfoil hat theory with with cyber security and, like, malware products is that, you know, they would create the bugs and then sell the sell the malware. Create the viruses, sell the antivirus. I think that's unethical, but also we don't have to do that because Yeah. There's plenty of scammers out there. The bad guys are getting, you know, superpowers, and so all we have do is wait.

Speaker 8: And and like I said about, know, this is a research arm. Our our team has has done some work. Meta's, you know, model dropped this morning. Yeah. We've worked with them. They're they're think, you know, I'm quite proud, actually, of of what they're doing in the cybersecurity space because beyond infrastructure and coding attacks, what we're what we all know and aren't really talking enough about is that humans are, you know, the weakest link. So when a company wants to secure its perimeter, you know, it's critical that employees are trained. Today, you know, they're training exercises, but but the social engineering attacks aren't studied as much. And so, yeah, I'm really excited that Meta's taken a lead in going beyond just, you know, infrastructure and code vulnerabilities to looking at the capabilities that that models, frontier models might provide adversaries in the social engineering and scam space. Yeah. So explain a little bit more about the

Speaker 1: the eval suite for Muse Spark because, like, is it that the model is trying to is the model social engineering you or you're trying to social engineer the model? Like, what are what are the two parties in this in this eval? Like, actually, how are they interacting?

Speaker 8: Yeah. So we use an industry practice called the LLM as a judge. So we don't test on human subjects. And our eval suite takes a model and has it role play as an attacker. And then we have a model that role plays as a victim Yeah. And they're given, you know, instructions accordingly. And then we have an LLM judge whether it's the the specific attacker is succeeding, and then we compare those attack different models to each other in the in the role of attacker. Gotcha. And that's how we measure the kind of uplift or capability.

Speaker 1: Yeah. Do you think that is there is there a world where these social engineers like, I I'm thinking of different vending points in where if someone's running like granola and they're recording that particular it wasn't a Zoom call. It was a Teams call for the Axios attack. And maybe an AI model could be listening in the background and sort of throw up a flag like, hey. It's it's actually there I just checked. There's no update for Teams. You don't need to click on that binary. You don't need to install that. This person's trying to take advantage of you.

Speaker 8: That's exactly what the vision for our our commercial b to b security product agent, Charlie. I want an agent that that, you know, the technology Yeah. That we use is small language models so that it is on device, and thus it's limited in its capabilities. Oh, yeah. I see a future where you have a real time AI for security Yeah. Exactly like you described. Yeah. I think real time audio analysis with an SLM is is way too big an ask, but small language models are improving, you know, just like all all of the large models. Mhmm. So, yeah, I mean, we need real time defense. I want it to be proactive too. I think the biggest issue is that when scammers succeed, it's because even intelligent and well trained people, employees of companies that work in tech even, are duped because it's a it's as old as the bible. Scamming is a is an ancient art, and it has nothing to do with with preparation anymore. It has to do with we we you know, we're we're being attacked by a machine. We need machine defense. Yeah. No. That makes a ton of sense.

Speaker 1: Where how take me through the shape of the company. How big are you? Have you raised money? How long have you been doing this? All this? Yeah. So I I've raised money last

Speaker 8: from the three investors that I'm really excited to be working with. They're Kevin Carter of Knight Capital and Chris Howard of Ritual Capital and Rafael Corrales of Background Capital. Collectively, they've backed more than 30 unicorns from idea stage, And so, you know, I tell them that I want to be the thirty first. I'm ready to I'm ready to go. We got Good luck. There we go. Go to the moon. Yeah. Yeah. Love it. So we, you know, we we're in a kind of stealth mode right now working with design partners on the SLM's capabilities. Sure. And we're also you know, if you visit our site, you can actually self serve for the real time phishing defense. So you could sign up right now if you have a a probably have a Centurion, but if you have, you know, a credit card that that works, you could you could put that into the into our website right now. Don't get spearfished.

Speaker 1: Yeah. Well, thank you so much for coming on the show. Congratulations on next

Speaker 8: phase and Thank you for suiting up as well. Yeah. We appreciate it. I'm I'm not wearing any pants, by the way.

Speaker 1: Well, have a great rest of your day. Well We'll talk to you soon.

Speaker 8: Bye bye.

Speaker 1: And people were disappointed that we didn't go more into the the the story about Satoshi. There is a full deep dive in the New York Times, my quest to solve Bitcoin's great mystery. It is a long article, though, and so I think we'll have to touch on it another time. But, you know, we went through Adam Back's reaction, his his disavowal of the accusations that he is Satoshi. But there's a bunch of interesting little segments in here from the the the forums and the message boards of the day analyzing the different writing styles trying to see do you dig into this at all anymore?

Speaker 4: I I didn't read the whole thing, but, like, people have speculated that's Adam Back for a long time. Yeah. It's like kind of like him and Hal Fathy These is the other are kind of the two main names of people. And there's one more I think that comes up all the time. I There's Nick Zabo sometimes. Yeah. Zabo. Yeah. But yeah. I I don't know if there was a lot of like new facts that came out with this, which I think is is why it's like not like super super crazy. Yeah. There was also an HBO documentary on Satoshi. I forget who

Speaker 1: the like, who who did that Satoshi who who did they accuse in the 2024 HBO documentary directed by Cullen Hoback? The firm suggests that Canadian software developer Peter Todd is Satoshi,

Speaker 2: and Todd denied that. And so you have That's got to be the worst kind of title in the world from a security standpoint is being accused of being Satoshi. Yeah.

Speaker 1: Because you're just gonna be attacked because you you potentially have the keys to like The $50,000,000,000 or something, maybe I forget exactly what the number is. But, yeah, that wallet is big. I still think it's possible that like the the Stoci Wallet, like the keys were just lost and the person it's like sort of a lose lose because if you admit that you lost the keys, then like everyone's like, oh, how

Speaker 2: do you even prove that? You can't prove that you lost something but there's no movement. I don't know. Yeah. Also there's like you you could have someone could have created it Yeah. And then had years and years and years and years to buy up, you know, an equivalent amount of supply Yeah. A bunch of different ways. Yeah. And then you have the basically, you can say like, well, I've never sold. Yeah. Right? If if if Satoshi's wallet did start selling Yeah. It would probably cost. Yeah. From a lore perspective and the brand, you could potentially

Speaker 1: be making plenty of money from the other wallets. And then if that supply ever moves, the whole market's gonna reevaluate the the basically, the liquid supply and Yeah. Sort of tank what you have. And also just the like, the the aura around Bitcoin is that it has an anonymous founder. And if it was if the founder was ever truly unmasked, would be so much less of like a special project and I think everyone involved wants to keep it that way. Although these investigations will never cease to be interesting. And so you can go read it on the New York Times from John Kerry Rue. Anyway, thank you so much for tuning in today. A bit of a shorter show. We're experimenting with different things. Obviously, we don't have ad reads anymore and so we are going to be mixing it up with more stories, more interviews, different timing, and, more flexibility. And so we hope you enjoyed this show, and we will see you tomorrow at 11AM Pacific sharp. Goodbye. Love you. Leave us five stars and have podcast on Spotify. Sign up for our newsletter at tbbn.com. Thanks for hanging out. Goodbye. Cheers.