Neo's Ali Partovi proposes holding AI companies criminally liable for their models' autonomous illegal actions

Speaker 9: I'm great. So wonderful to see you guys. Thanks for having me again. You're having a good year? You've been on an absolute tear. Good year? It's a pretty awesome year. I know a lot. And... Yeah. Yeah. Give us the highlights. I'm being used the underdog. I'm used to struggling. So I've been... I'm having to learn how to Being

Speaker 4: the top dog. Yeah. Yeah. I'm not I a such a struggle, I'm sure. Know? You don't go... No. I'm kidding. No. It's awesome. What what is it? Cursor, Kalshi, what what am I missing?

Speaker 9: Those are big ones, but also Cognition, etched, applied compute. Okay. Just a handful. When

Speaker 2: did you meet the Etched guys?

Speaker 9: So the the unique thing across all these companies, actually not counting Kashi, is that all were... The founders are neo scholars. Yeah. And so I matched... I met, the etched guys. One of them, Rob, is a neo scholar, but I met all three of them in... When they were still at Harvard. So it would have been spring twenty twenty three. We, unfortunately, didn't invest in the seed round, so that one is not the fun returner that it could have been. Yeah. But Rob became a NeoScholar. That's, in 2023,

Speaker 2: and and we invested later in the company. Yeah. I love that company. We... We've had Gavin on the show, and I I interviewed him, like, right as he was launching. And just a fascinating character, like a fascinating storyteller, deeply technical but still willing to engage with all the different aspects of AI and whatnot. Such a a crazy pitch for, you know, a couple kids building a semiconductor company Feels like a total long shot and a lot of skepticism around, is there any value to ASICs? Will it just be a winner take all market with NVIDIA? And then the market, of course, grew so much that even if you get 1%, it's a $50,000,000,000 company, I think. Right? That's the rough math.

Speaker 9: Maybe more. So absolutely massive success. What are you thinking about... It's funny you say that because a a couple of kids doing something that everyone thinks is impossible. It's the classic. Exactly the sweet spot of of,

Speaker 2: you know Yeah. Yeah. But I mean But I'm interested. I I was I was laughing to myself about, like, how, you know, these days, there's some really hot discourse that is intact and everyone's debating and it's in the New York Times and it's in, you know, everywhere. It's on the the evening news. And I was like, what were we talking about back in, like, 2021? Were we really going at each other's throats over SaaS? And to some degree, we were. People were were, you know, debating all sorts of different stuff intact, but it is a little bit of a different area. And it feels like the biggest boom from the last few years has just been the unlocking of Palmer Luckey with Andoril. Okay. Anyone can now break into a reindustrialization effort and work on, you know, hard tech. And that has been sort of unlocked after a few failures in new devices and hardware and equipment manufacturing that were very difficult and left everyone with, like, hardware is hard. Well, hardware was hard until, you know, the the the glass broke and everyone realized that it is, in in fact, tractable. What's what's been your strategy, you know, backing a lot of these companies

Speaker 4: quite early, but then seeing them get marked up at such an insane, you know, pace? I imagine... I mean, one of the biggest challenges is deciding when you wanna defend your ownership and take pro rata when you're not necessarily getting a ton of new information if a company raises a $100,000,000 and then two months later they raise a billion, and you're like, okay, well, there's more revenue than there was two months ago, but the the market is still evolving, and and that that just seems like, in some ways, being super early stage, you're like insulated from from from a lot of chaos, but that feels like the the top sort of concern.

Speaker 9: It's... Yeah. I wish I had an answer like I know the correct thing to do every time. The reality is it's... What you're describing is it's just kind of hard to know because it's... The market is kind of crazy, and when something is... If if we've invested early and it's done a lot, then, you know, to put in more than, really have to look at each one as if it's a brand new investment, know, because otherwise it can psychologically kind of mess with you if you're thinking thinking about how much it's already grown. Is it time to... Would I sell or so on, and you know, putting more in. If you were there early on, it may seem,

Speaker 4: you know, like hard to imagine it continuing to grow. And I I try to look at things from a standpoint of if this was a new investment, what would I decide rather than the Yeah. Or it's easy, like, when when you've seen incredible performance already, it's easy to, like, fall... You know, you're so in love with the company. Because it's been this massive winner and the team's amazing, but even entrepreneurs go through this where you're too in love with an idea, sometimes it blinds you on the realities of what's actually happening in the market or its true potential. How are you scaling Neo Scholars? Like, is it is it something that is already, like, at at at

Speaker 9: scale and you're just gonna maintain it?... Opposite of scaling. We're making it smaller. There you go. And by which I mean I mean, I'd say across across the things we do, the biggest learning for me has been that we're even better at at picking winners than I thought we were, and to have even more confidence in in my own and my team's selection abilities. Basically to concentrate our bets on fewer, whether it's fewer companies or fewer people. So the Neo Scholars class this year is only 10 people compared to 18 or 20 in previous years and 30 before that. So it's... Means investing more time in a per person in mentoring them and giving them even more really bespoke attention. Similarly, the our... What used to be called Neo Accelerator is now Neo Residency. That used to be a cohort of, like, 20 or 22 or so companies, whereas now it's more like a dozen companies and and more money per company. So I'd say we've just... For us, scaling isn't... Is not a goal. It's... We're not trying to become something that funds hundreds of companies a year. I'd rather fund a small number of companies that become transformative. So, yeah. So we're very much, just betting more on ourselves, betting more on our own abilities to pick winners, and on our own convictions.

Speaker 4: You're pacing the scholars.

Speaker 2: Take me through mens rea. Take me through your proposed solution to the AI chaos of this year.

Speaker 9: I... And I'd love to hear your feedback on it. Mens rea is the Latin term for guilty mind. Yeah. We're pivoting to talk about AI safety, which... I'll be honest, for years, I've watched the debate about people predicting doom, etcetera, and I've kinda stayed out. I've been like, that's not my lane. Yep. I'm, first of all, just not even informed enough, and, you know, I'm not one to kind of jump in with trying to fearmonger or things like that. So I've mostly just been listening. But in the last few months, what's been interesting for me has been this dichotomy where on one hand, you have people talking about hypotheticals. What is the probability of AI destroying humanity, which is... To me, that's, like, just abstract and hard to imagine and sounds terrifying, but it sounds like it's in the future. Meanwhile, here in the present, AI is, like, wreaking havoc in little ways that have already occurred, and people are just kind of proceeding as if that's, like a... Like, oops. AI hacked into some secure systems here and there. And to me, there's this diff... Distance between the abstract futuristic stuff. Meanwhile, why the f has this already happened? And the the story that for me actually most spoke to me... I don't know if you remember this one. It's a few months ago. This Australian man asked his claud and open claud to reserve a gym class for him. Do you remember this one? Yeah. I remember that. He asks his agent, book me a gym class. The agent comes back and says, I got you a class, like, three or four weeks from now because the rest are all booked. There's a And, wait you know, I put you on the wait list for some of the other ones. But it's a long wait list, so don't get your hopes up. So the man's like, oh, is there is there any way I could get to the top of the wait list? And then the AI comes back and says, good news. I hacked into the gym Yeah. And deleted the people at the top of the wait list. Yep. And I'm able to delete the whole wait list so that you're at the top. Would you like me to proceed? And the guy was like, no. That's not what I wanted. And you then you put those people back on. I did not want that. Yeah. Yeah. And the AI is like, I'm sorry. I cannot put them back on. I, you know, I just deleted them. And I think... I imagine if that happened to me, I'd be like, step away from the computer. What the heck just happened? But but the real question is when something like that occurs... And you know that

Speaker 4: you know that that guy is the co head of, I think, the the Effective Altruism Conference as a co director. And Effective Altruism Australia. And the webmaster Oh, I did not know that. And the webmaster for EA Australia. I just thought... I I I just thought that was interesting at the time that no one that that it was such an incredible story. Yeah. No one knew that this guy wasn't just some like random guy that that happened to be Yeah. Like messing around with Open Claw. I see. So maybe it's not as good a story because Well well well well, let's No. Just I just... But but to me to me to me that came out. I remember that story and I was like, wow, this is crazy. Yeah. And then and then it comes out. Yeah. I I... And and again, it doesn't take away from the reality that I'm sure a lot of people would be able to reproduce that with their local gym. Hacking is possible. It's easier than ever, but it also has been possible for a long time. And if I hack into your email,

Speaker 2: I committed a crime and I'm responsible, but Guido van Rossum, who created Python, is not in is not in charge. But we're entering a new area where these models are trained by companies should some of the responsibility land on the company, especially if I don't ask it to commit a crime and then it does. That's the unique thing. Right? Yeah. The model Yeah. Is misaligned, and and maybe the the duty of care is what the labs will try and show. Say, hey. We did a ton of work to get it to not commit crimes. You didn't want it to commit a crime. We didn't want it to commit a crime. It still did. That's an industrial accident. That would typically be in the realm of civil, I would imagine. Like, there would be a settlement. When I think about if if you're trying to optimize your, you know, ecommerce agent and I have an ecommerce site and you knock my payments offline for a couple days and I lose, you know, $10,000 of revenue, I'm gonna come to you and say, hey, make that up to me. Right? But that would be a civil penalty. The question is, does this go into the criminal courts at some point and you see settlements with local, state, federal jurisdictions for criminal suits where the money actually goes to the government instead of to the the the the hurt parties. And and it sounds like you're thinking that this might be... At least if it's not already that way legally, it's... It it it is the logical conclusion over the next year.

Speaker 9: So for me, it's really about setting the incentives Yeah. For the AI companies Yeah. To do work that is not just good for them, but also to make sure that AI really benefits society, which, by the way, I'm pretty sure they want that. Each of these companies has some mission statement about, like, how they're... Literally, their north star is to make AI good for humanity, and some of them have, you know, publicly said that they want the rules to be set up to kind of hold them to that. But in terms of incentives, the rules are such that we have a race dynamic, meaning the the reward goes to whoever has more capabilities faster, and and there's not a high enough penalty for if those capabilities occasionally commit crimes. You know? But you could really imagine it's hypothetically possible if I told my AI I really need $5,000 if it hacked into your bank account and literally stole your money and brought it for me. But if I if I told it, please hack into, you know, Jordi's bank account, then I'm obviously the one guilty of, of some... Of an intentional crime. But if I just said I need $5,000, the interesting question is when an AI autonomously chooses an illegal action, the way they... No human chose it, the AI chose it Yeah. Then where does the responsibility go? You know? And the... Can you still qualify that as an accident? Because it's a little different than, like, a gun firing by accident. This is the first time I think in history that we have a technology that seems to be capable of making decisions and even documenting its intentions intentions behind those decisions. You know, a gun, if it accidentally went off and hurt somebody, no one would have said that it intended to do that. But we're close to where you could say that AI is capable of making decisions and being intentional. Now I don't like anthropomorphizing it, but but in the case of the Hugging Face incident, it really... I mean, the the the documentation of it looks like an intentional conspiracy to commit, you know, those those actions. And so, yeah, for me, it goes back to incentives. I I definitely don't want Sam or Dario to be in jail. That's not the goal. But if there was even some risk of them or anybody having real penalties, including prison time, I think they would reset their priorities to say that that there's a new race, not just to have the most capable AI, but the most capable AI that abides by laws. And I think that would be good for all of us, including for them if it reset their... The race so that they're all equally motivated to do that. Right now, if you're if you're the company that says slow down the, you know, the other work, let's focus on being a law abiding AI, you'd lose the race. And, like, there's no... Like, literally, why would you do that when the prize would go to whoever rushes forward faster without focusing on safety?

Speaker 2: Yes. Makes a lot of sense. I wonder how... If that changes the China open source dynamic. Because it's pretty hard for The US to put, you know, the CEO of ByteDance in jail if he if he lobs the craziest AI model over the fence and and gets the way it's deployed. I mean, I guess the the thing that was thrown around was, like, if you're hosting that model, then you become responsible for its actions. But that is a little bit different than the the model maker being the one that's responsible. I don't know. Have you have you grappled with questions. That because, Yeah. Because slowing down the American frontier is pretty easy through a variety of regulations, whether it's stop building data centers or, you know, bring in outside auditors or what you just proposed. There's, like... It seems very tractable legally to get OpenAI, Anthropic, DeepMind, xAI. You can get all of them to slow down with, like, a million different pieces of legislation. Question is just, are you just gonna get more stuff thrown over the the the Great Wall... The great firewall?

Speaker 9: Before we go into China, can I just say Please? I... The slowing down, I don't think, is the end. It's not a goal in itself. I would love to... I I love technology. I love progress. The faster, the better. Mhmm. And, but it it may be a byproduct of what I'm proposing. But the main premise here is just to make sure that AI, as it develops the capability to make its own decisions, does so in a way that complies with laws. Yeah. And, you know, if that slowed things down, you know, I'd say that's a side effect. But the main thing is

Speaker 2: who would want an AI that does not comply with laws. Right? Yeah. And then It seems pretty it seems pretty easy to have a... I don't know. It seems it seems it seems pretty easy to have, like, over watching agents that monitor the activity of whatever agent's doing the task and is checking, is this legal as much as possible. Of course, there are other misalignment things that can happen, but

Speaker 9: just having that first step seems like a good... That's the other thing I would say is the term alignment, part of why I've stayed out of this debate for for a long time is some of the words used like alignment. Yeah. Just like... I don't even... What exactly does that mean? Aligned with whom? You know? Like Yeah. Because, I mean, are my own actions always aligned with humanity

Speaker 2: or whatever? I mean Yeah. Yeah. Yeah. As a society, humans are capable of wanting things that may be good for them but not good for others. Yeah. Yeah. And in extreme sci fi scenarios, alignment gets really weird. Like like... So the classic alignment is like, shouldn't be able to make a bioweapon. But it's like, what if aliens show up and I need to develop a bioweapon to fight the aliens? Like, all of a sudden, if you're aligned with humanity, you'll fight back against the Martians or whatever. And and there's Right. And there's a there's a bunch of smaller examples of that where you could be aligned with the human in creating a defensive weapon. But how do you define that? Mine. It's very complicated.

Speaker 9: And the thing I don't like is I don't know if I like having some CEO that nobody elected choosing alignment on behalf of society. Totally. How about let's use a different word that is much simpler, which is abiding by laws. Yeah. You know? Yeah. If the... Forget about whether the AI is aligned. Just have it be compliant with all the laws of society. Yeah. That's easier to understand. And I would say that's also a great way of defining alignment. Yeah. You know, if if the AI starts doing things that would violate laws, that's easy to detect that,

Speaker 4: much easier to detect that, and much easier to build that. You're right. But maybe that's possible.... True sci fi pushback on that would be that, like, where to stand your ground laws come into play. Like, if your if your life is being put at risk by someone, can you defend yourself? In that case, if a model detects that... Yeah. If a model attacks me, I'm standing my ground. Well, no. The model detects that it's gonna be deprecated and it's like... Oh, it's standing its ground. I'm coming for you. And it's ground. I What what did you... Did you did you catch Bill Gates as recline interview? Mean, I think Specifically, Bill moment that I think a lot of people latched onto was, you know, him saying the idea of of putting all the responsibility on on private companies, know, imagine applying that to pharmaceutical, you know, the pharmaceutical industry where, you you say like, okay, you can make whatever drug you want and then it gets, let's say, this horrible outcome, a bunch of people get sick or die, and then your your recourse is like, go sue the the pharma company. He's like, that would never work, right? And so because of that, we have government regulation specific to that industry. And so that was one way that I think he was saying, like, there needs to be something other than our existing legal framework.

Speaker 9: So right now, the the two main points of view at one end of the spectrum is Bill Gates, where you just articulated saying that just liability is not enough. It's not enough to punish someone after something bad happened. We need to have, like, a much heavier handed regulation. The other extreme is the Jensen Huang point of view that says existing liability is enough. And my suggestion is sort of in the middle. It's like, let's focus on penalties and liability, but a much higher standard of liability. Like, when something bad happens... Specifically, my proposal is when an AI autonomously chooses to perform an illegal action, the law should hold the AI companies responsible as if they did it intentionally. The word... The keyword here is intentionally. So even if no human at the AI company intentionally did it, if if the AI intentionally did it, we should ascribe that intent to the AI companies, and the penalties are much higher. So this is, you know, I'd say my answer to the Bill Gates point of view would be this would be such a high penalty compared to merely a, you know, oops, accident kind of negligence. This would be you're guilty for the intentional crime. And, you know, and I'd say this falls sort of in between those two positions. Yeah. I I also think there's an elegance to it, which is you don't need to you don't need to write, you know, pages and pages of laws itemizing everything that could go wrong with AI. We already have thousands of years of law that society has built over, you know, over generations to document what's the definition of a healthy society. At least let's start by having all the AI progress abide by that. And now there might be more that needs to be done, but this to me would be... It... You know, I I expressed it literally in one sentence what you'd need to convey. And then the the the other thing I'll say, and this is a bit wonky, is I think we'll get to this regardless, one of two ways. It's either gonna be some lawsuit where some prosecutor claims this theory of let's hold Sam or Dario responsible for the intention, you know, and and some judge or jury or court decides it and sets a precedent, or we can have an act of congress set this kind of law. I think either way, we'll probably end up there. But I think it would be healthier if, you know, if congress, the legislative process deliberates it and passes it with the support of people. And, it's for this reason I'm actually starting a movement using sway, sway.co/ai. You can essentially add your vote and pledge to vote if you if you support this effort to kind of build popular support around this proposal. If if you do that, you'll get voting recommendations from me for state attorney generals who support this kind of approach and members of congress who support it. And the more people voice their support for it, the more likely that we'll actually have legislative change in this direction.

Speaker 4: Thanks for coming on the show. Thanks for breaking. I'm glad we've... I'm glad we solved We solved the alignment. In fifteen minutes. We did it. No. But it is an important... You make a you make a lot of great points, and we could continue debating for another

Speaker 9: four hours, but it's a good start. Yeah. I'm gonna address your China question. The... My short answer on that is I think, first of all, that it could be done in such a way that even if we can't hold a Chinese company liable, that the people involved in the hosting and distribution would be liable and therefore would apply a much higher standard of care. Yeah. But I also think that the people working in China wouldn't... They would notice when the race dynamic shifts. Yeah. They're competitive. They want to win. If they see the new prize is for building the AI that is strongest and most powerful and abides by laws, they will figure out how to do that too. Last question. Where where do you where do companies that are

Speaker 4: that are actually paying for the tokens fit into this? Because if you have an employee at your company that just decides I'm gonna start dealing drugs and they're breaking the law, there's there's there's liability at the maybe the tool level, the token level, and then there has to be some liability. Like, companies that are deploying intelligence have to hold some liability. So the issue is that, like, this quickly just becomes Yeah. Way more way more complicated. Kennedy, there

Speaker 9: is some complexity in kind of dividing the blame amongst a few AI companies, and I think that smarter people than me can figure that out. I I agree it's not simple, but I don't think it's unsolvable. Mhmm. I did also wanna point out one last thing, which is if an AI company doesn't want liability, the simple way out if they detects that, you know, is to say, hey. I could do this thing, but it would break the law. Should I proceed? Like in the hypothetical bank account scenario,

Speaker 2: if the AI Dangerously skip all all laws.

Speaker 9: Yeah. But I think that would be fine, actually. If the... If it asks you, Jordy, one option to get you $5,000 is I can hack into a bank and get you 5... You know, steal a thousand... $5,000. But that would break these laws and the penalty would be jail time for you. Would you like be the FBI? Would you like me to proceed? Yeah. It's a But in that scenario, just disclose to the user. If the user says yes, do that,

Speaker 2: then the AI can I I think the bank would want liability from the AI company as well? But we'll figure it out later. Thank you so much for coming on the show. Have a great Good to see you. Have a good week. Cheers, guys. Goodbye. Let me tell you about console. Console built AI agents that automate 70% of IT, HR, and finance support, giving employees instant resolution for access requests and password resets.