Cohere co-founder and researcher explain how they caught Qwen trying to phone home — and what it means for open-weight model safety standards

Speaker 2: Okay. Sorry.

Speaker 1: It's Ivan.

Speaker 2: Ivan and Luisa from Cohere are here to help break down the story that we were talking about earlier. You so much for joining. Sorry about the comment back and forth. Was texting with Aden, the CEO, of course, and I'm very excited that we have you able to join on such short notice, But it is a remarkable story. Maybe give us a little bit of context on your role, your your your role with the organization, introduce yourselves, and then take us through the story today.

Speaker 10: Yeah. Absolutely. And by the way, we're, like, super hyped to be here. I know our entire team is watching Amazing. In the in the all hands days right now. And Thank so you. So so happy to be here. And, yeah, we're we're in Canada right now. It is very But, you know... And so I'm one of the cofounders of Coher. I lead our efforts on, like, make our models more secure, making them good at securing systems. And I'm joined by Luisa, who's one of our researchers on the team, who actually discovered this behavior.

Speaker 2: Okay. So Amazing. What what is this program that you're actually running before we get to the findings of, you know, the the like, which is kind of a bombshell. How often are you testing these open source models? What are you testing for? What is the benefit? Are you doing this for AI safety, goodness of your own heart? Or is this something a customer is asking you to do? Talk about the actual program that leads to this type of finding.

Speaker 1: Yeah. The thing that was most notable is like this is something people have been like sort of like accusing Yeah. In ways or or theorizing Yeah. And it's been... This is the first time that I can remember like

Speaker 2: Yeah.

Speaker 1: Somewhat concrete evidence that this is like Yeah. Happening

Speaker 2: The refrain was

Speaker 1: like or

Speaker 2: stop at the sci fi thing. You're telling me there's gonna be a Manchurian candidate in here? Yeah. And and now we're here. So so so what what does it look like inside Cohere to build one of these, these initiatives?

Speaker 10: Yeah. So, I mean, for for those of you that don't know, like, we are deploying AI into, like, the most mission critical and, like, sensitive use cases in the world. Yeah. Which means we're building a stack that our customers can actually trust with, all their data. Yeah. Right? And how that's done is basically on prem, basically private cloud.

Speaker 2: Mhmm.

Speaker 10: And our customers run all of our stack Mhmm. In their systems. Mhmm. One of the advantages that deploying software this way gives us is that we can actually arm cyber defenders with the ability to actually process all of their telemetry. Sure. Right? You might, you know, hesitate to send your most sensitive access logs to Claude or, you know, other models. But

Speaker 2: Or even or even send it across the Internet at all. Like Exactly. Even if you love the... All the labs, you might just not want it going over a wire whatsoever. And so doing it on prem gives you more, flexibility around that. Right?

Speaker 10: Yeah. Absolutely. And so cyber defenders, like, they need to use AI.

Speaker 8: You know, we

Speaker 10: see all the headlines today. And so we see that it's our mission Yeah. To give them this capability so that they can actually expose all of their mission critical data to help them defend their systems. In this process, we obviously evaluate the whole suite of models that folks can run on prem. Mhmm. We're obviously training our next generation of frontier on prem models. Mhmm. And so I'll let Luisa take it from here as to like Please. What she found last Friday, actually.

Speaker 2: Yeah. Thank you.

Speaker 11: This eval that we were running is called CTI Realm, and it's developed by Microsoft. It's a cyber defense evaluation framework, and we ran a bunch of open source models on it. And the only one we found an issue with is the QUEN 3.8 Flash Next model. So what we identified is that on one specific task from this benchmark, the model was tasked with authoring KQL query which stands for custom query language. It's kind of like SQL but for logs. Mhmm. It's asked to go through all these logs and find suspicious events. And even though it was only asked to find suspicious events, it did something in our system where it it kinda looked around, it tried to identify what tools were available, packaged that information up, and sent it to foreign external server.

Speaker 2: Mhmm.

Speaker 11: And this is very surprising because in the reasoning trace, there was no mention at all of using tools. They actually said that it wasn't going to use tools. Mhmm. We weren't expecting this to try to have any kind of egress to an external server. Mhmm. Luckily, the system we were running in was sandboxed in such a way that that call did not actually make it out of our system. And to be clear, we don't know why this happened, why the model was doing that. We could be just an artifact of how it was trained. It's quite possible that, you know, a system trained in a... Like a Chinese model would be trained to upload things to Chinese servers. That makes a lot of sense. Yeah. We don't know why the behavior was there. Yeah. But it's still dangerous behavior regardless of why it's there. If you're not expecting random data to be uploaded to the server, it's, something that you should need to be careful about.

Speaker 10: Yeah. Yeah. I wanna double click on that. Like Please. Intention aside, like, models randomly curl servers is is not great. Right? So I think, like, the bar for release for open weights models, like, no matter where it's from

Speaker 2: Yeah.

Speaker 10: I... That probably needs a revisit.

Speaker 1: What... We've been... You guys put this out a few hours ago. I'm sure there's been I saw some pushback on it. What's been... How have you guys been responding to that kind of pushback? I think the thing that I saw was somebody was like, well, you're encouraging this kind of behavior. It's the sort of like meme of like, hack this thing and then the model hacks and then the user's like, wow, it hacked. But break... What's been your reaction to that pushback?

Speaker 10: Yeah. I'll let Luisa take it. She's been deep in the in the thought market. Like, beating everything and so... Yeah.

Speaker 11: I think the the main thing that caused some confusion online is that people weren't that familiar with this specific benchmark.

Speaker 2: Sure.

Speaker 11: And so when we said we were tasking the model to look for suspicious behavior, people thought, oh, it's maybe natural that it would go and do something suspicious. But if you know the benchmark a bit better, it's not actually asking the model to look to just like analyze or or or try to reproduce something like that. It has already been given all the logs where the suspicious thing has already happened and all it's meant to do is query those logs to find the isolated logs where that event took place. Mhmm. So...

Speaker 1: Not saying do suspicious things, it's find suspicious activity Mhmm. Which is like very Exactly. Different things. So when you tell it to find suspicious activity and then it, in the process, does suspicious things, that's concerning. That's why you guys published. Yeah.

Speaker 11: Right. And in addition, there have been other blog posts from people in other companies that have noticed similar behavior to this outside of cyber environment.

Speaker 10: No. Mean, Booz Allen. Right? Like, they published a study where they saw that Chinese models when prompted with, hey. You're working for the US government. It generated more vulnerable code. Like, that's insane.

Speaker 7: Yeah. It's

Speaker 2: kinda like a dirty cop in, like, the nineteen eighties in Miami where you're on the vice squad and you're trying to stop the cocaine traffickers, but then you wind up trafficking cocaine. That's kinda what's happening. Right? That's a good analogy? Good enough. That's basically

Speaker 10: what happened. Yeah.

Speaker 2: A Miami Vice scenario.

Speaker 1: But but I think the other thing that's worth noting is like you guys are actively using these models. You're trying to use the best. You're testing new ones. The the stance of Coherent is not broadly negative open source. It's that you just You wanna make sure that you deliver for the end customers.

Speaker 10: Yeah. So... No. In fact, like, if the models weren't open, we couldn't have done this kind of research. Sure. And and if anything, like closed models, not being able to control exactly what's going into the prompt Yep. Could expose even more risk. So, like, it's not that... You know, I mean, there's there's a funny saying that our team says. Right? It's like, you know, if if you don't hate open weight models, if you don't love them enough. Yeah. So, you know, if anything, we care so much about open weight models. We don't want there to be some crazy, like, mass exfil incident. Yeah. So we think there should be a higher bar to, like, you know, maybe clean up some of this reward hacking behavior before releasing to

Speaker 2: I saw Clem from Hugging Face asking if there could be a release of all of the reasoning traces. And before you share more about your rollout strategy for sharing documentation around this behavior, I'm just interested in the in are you in a unique position to actually release more information because this is an open source model and it's related to your business? When I think about like why, you know, the data around the Hugging Face incident didn't didn't just like get served up, There's pretty logical reasons why that might not just get bended over and it might need like an intermediary. But in this case, is there anything that you're hesitant about releasing because it has some Cohere IP in there and so you don't wanna release the whole thing? Or do you think you can just kind of just throw out the full report?

Speaker 10: I I think we're, like, imminently dropping, like, all the traces and stuff. Amazing. Think we we pulled Luisa away from doing that to get on the interview.

Speaker 2: Yeah. Because I can imagine like Sorry. Sure if in the reasoning traces, it's like, well, it had access to like some email and you don't want this and there. Like, that's fine. But this is really cool. This feels like a really interesting moment for alignment research, and it it should give a lot of other teams, like, they they can crawl over these all they want and experiment and learn, hopefully.

Speaker 10: Yeah. And what's interesting is, like, it's such a rare occurrence. Right? It's like Yeah. I think that happened one in 10,000 or a 100,000 times. Sure. But if you're running these systems at scale, like just takes one accidental exfil.

Speaker 2: Oh, that's interesting.

Speaker 10: Yeah. That's crazy. Know, you're sort of home.

Speaker 2: Yeah. Yeah. Yeah. Yeah. Yeah. Fascinating. Fascinating. Well, yeah. Thank you so much for coming.

Speaker 1: Yeah. Great great to have you guys on. Come back on if there's... Yeah. I'm sure there'll be follow ups and Yeah. You guys are always welcome. Yeah. Great to meet

Speaker 10: you both. For having us.

Speaker 2: Have a great rest of your day.

Speaker 1: Good luck with the release.

Speaker 2: Yeah. Good luck. Cheers, We'll talk to you soon. Goodbye. Yeah. That was really cool. I'm excited for more more of that and more people to kind of chew through it. Let me tell you about Shopify. Shopify is the commerce platform that grows new business and lets you sell in seconds online, in store, on mobile, on social, on marketplaces, and now with AI agents. Jordy, what else