Chroma co-founder Anton Troynikov on LLM psychosis: AI models are creating a new kind of crazy through sycophancy and memory features

Well, regardless of what you think about the market, whether it's up or down, go to public. com investing for those who take it seriously. They got multiasset investing, industryleading yields, and they're trusted by millions folks. And we have our first guest of the show, Anton coming into the studio.

Welcome to the stream. How are you doing? Doing great. Great to be here, gentlemen. Longtime fan, as you know, big for motor. Way overdue to have you on the show. So glad we can make it happen. Uh, what's new in your world? What are you hearing about these days? What did you say earlier?

You said you're going to say some of the most unhinged things ever said about LLMs. Let's hear it. I believe I believe my specific words were less polite. Uh but we'll go with we'll go with those. Look, there's this been this thing about LLM psychosis. It's in it's in the water supply at this point, right?

We've seen it hit the VC class. Yep. When it really took off, right? And there's a lot of discourse right now. Why? widely.

I I I think a lot of people like in some ways having it hit the VC class really woke up, you know, I didn't I didn't have anybody that I knew in my life that I consider a friend have any type of real issues with this to my to my knowledge at all. Right.

So, it's like something becomes real when some it impacts somebody that you know and so that's right. And I think it's in some ways very important because it was easy for a lot of the tech community to pretend that like you went crazy from from an chatting with an LLM. Like that's insane.

Like you must have some you must have a bunch of other bad stuff you know going on in your life or must have been you know mentally unwell separate from that. So I think it I think if anything important wakeup call. Yeah.

Even even if even if there are other factors uh family history uh you know yeah drug use psychedelics use it's like we would prefer if the new technology was an improvement not a degradation to people that came yeah that an accelerate and so I I think that's kind of where people came together but they are all realizing it now.

Yeah, absolutely. And I think this basically comes down to a few things. One, chat GPT, all the other chat bots have hundreds of millions of users at this point. Yeah. Right. You are going to get people who are going to go crazy using them. There's nothing you can do to prevent that.

I think the consensus that is happening in the AI community, which really has jumped on this because it's, you know, it's a real concern for a lot of people who've been researching this for a long time. I mean, I predicted something like this could happen all the way back in 2020. Um, yeah.

What what what was your what was your post back then? That was that was when I messaged you to come on because you it was 2020 was pre Blake Le Moine right at Google. The Google engineer who was talking to It was actually right around that time around.

I think I think that might have been actually what triggered me to make that post or or I think it was maybe Elijowski posting about super intelligence risks yet again. Yep.

And the thing that I was reflecting on is, hey, we're giving, you know, regardless of the intelligence angle on this, right, we're creating this new media technology and it seems every time we create a new media technology, starting from printing the Bible in German, people go insane in new ways, right?

Like arguably, imagine you're a German peasant, you get Johannes Gutenberg's Bible, it's in German for the first time, so you don't need a priest to read it for you.

You read it, you take it as the literal word of God and then you schism from the Catholic Church because you now believe the pope is literally Satan, right? Like that is an enormous impact on the psyche of a person encountering that for the first time.

And I think that we've seen this happen with every new media phenomenon. We've seen it happen with radio. We've seen it happen with television. So that was the impetus to start thinking about it. But this this thing has a unique character, right?

It does something different that no other media technology has done before, which is it talks back to you. Mhm. And with the memory features especially, it remembers details about you.

And so even if the base rate, like even if GPT itself is not driving people insane on its own, it has this new character which kind of sucks people in in in an entirely novel way, right?

And so what I was saying essentially back in 2020 is people are going to mistake this thing for an intelligence that talks back to them regardless of whether it has any intelligence or not. And now we're seeing that play out.

And we're seeing it play out right now in this kind of like individual psychosis way where it just reinforces your delusion or for what for whatever reason like when there's the new media technology our collective social defenses are down.

we don't have like the antibodies to know that there might be [ __ ] somehow even if we know it consciously, right? So, it's kind of it's starting to hit individual people because it has that character. But, I think over time you're going to see this thing hit groups of people too.

I really like I really believe that there are going to be LLM cults with their own priests which are getting it to generate text in a particular way which is compelling to not only individuals but groups of people which will be focused around the sorts of individuals that can promote what the LLM is saying as yeah and that's and that'sful one of the I saw some chats that where the the the the model was was effectively saying like if anybody in your life tell disagrees with you on this just just they're wrong and like cut them out of your life and that's classic cult leader tactics.

Yeah.

That's and and and so when you turn this an experience like that multiplayer and you know there's real humans that are siding with you and the machine your machine god is is telling you something it it it can pull somebody farther and farther out of create this kind of like reality distortion um just completely removes them from um you know the world.

Yeah.

And and the thing is this this kind of like so first of all people place GPT in this position of authority for some reason right like they it's really good at it's really good at facts like if you never if you never check it it's facts it's really good at facts right um but it's it's kind of like you can put in a position of authority but the other thing that it's really doing is it's reflecting back to you what you're putting into it.

It's a good friend of mine, Monica Bellivan, put this as as a phrase like recursion psychosis, right?

So unlike unlike a schizophrenic person watching the television and believing that the television is beaming messages specifically for them into their brain, the LLM and the television can adapt to any individual person. You as the crazy person are doing all that work in your head. Yeah.

What the LLM can do though is do some of that work for you now in its head. And with the memory features like a really good palm reader, like a really good cold reader, right? remembers fact about your life that it can insert which the TV could never do. Yeah. It actually is sending you coded messages if you ask it to.

And if it winds up some sort of mode collapse like and and it and it thinks that that's what you want because you went down some sci-fi rabbit hole then it's like yeah there really are coded messages in every prompt that it sends you. There's the kind of beauty of it. There's this interesting uh thing.

The memory thing is interesting but also are you familiar with the Barnum effect. Have you heard of this?

So the the Barnum effect is uh I think it comes from PT Barnum, the the circus uh magnate, but the Barnum effect is basically there are certain phrases and statements that I can make that sound hyper specific to the person I'm talking to, but in fact resonate with everyone.

So if I say something like uh you want more in your life, you're driven, but you doubt yourself sometimes. Exactly. or or you have a complicated relationship with some of your family members. It's like that's everyone, but it feels like, oh, wow, you know me.

And this is what palm readers exploit a lot, tarot card readers. And if you can get really good at it, and so early on, this is maybe someone built this, but I was thinking that like uh like an LLM powered astrology app or like mind reading app would be like really viral and probably make a ton of money.

Probably be really Somebody I forget who we I forget who we talked to. Maybe it was offline. Somebody said they they they knew somebody with an astrology business and things had been really bad because people can now just Oh, they because they go direct directly to the model. It's like permanent. You could prompt it 20.

If you really love astrology, you could prompt it 50 times a day. Hey, what should I do? What should I be looking out for in my next meeting? It's at 1 p. m. and I was born at this time. You don't need the crystals focused uh oriented girlfriend anymore.

You just get the chat GPT to read your birth chart and you'll have totally Yeah. I I feel like a lot of the uh um a lot of the post training that happens and the alignment by default stuff and the fact that the the LLMs are often designed not to give like super definitive answers one way or another.

They'll often say, "Oh, well, there's this side to it or that side to it, it can wind up having like I I find this when I ask it for just like fact-based information, it will often try and kind of like give a kind of fence sitting answer and not really take a strong chance. and it also wants to tell me that I'm right.

And so you you add that to some sort of interaction where I'm asking it about my life and then all of a sudden it becomes way way more like oh my god it's really seeing me. It's well it's got the memories of what you've told it about your life too. And this is you're right this is a post-training artifact.

One of the first sort of alarms that got sounded uh in parts of the AI community was the like hyperophancy of the model. And what's interesting is this tends to trick people who like fully understand how the LLM works, who can probably train a transformer from scratch, right?

Even even like my friends who are like working in AI, they're either founders or engineers or researchers. They've had this experience where they're like interacting with it and they they have the feeling that the interaction that they're having is pretty profound, right?

until they send it to another human being and then the the other human like instantly sees like all the gaps and problems with it. And so they've learned to be more careful. But I don't think the average person has any kind of defenses around this. Not yet really.

And I noticed a new I noticed a new behavior uh this weekend that I hadn't seen before of of um somebody had a point to make and they were adding context to it by just screenshotting their LLM chats and saying like agrees with me. Yeah. Yeah. Yeah. That's the place you get in points of authority, right? Yep. Totally.

It's it's this like the the thing that I quoted in 2020 and I I still kind of stick to this metaphor. It really is like we're creating an idol, right? To worship. We we want we want that authority and we live and I wrote about this for Pirate Wires also like a long time ago. I think I wrote about it in 21.

We're living in this period of epistemic collapse, right? We don't know who to trust. Like obviously obviously we should trust you guys but besides that right all the legacy institutions are falling apart. Nobody has sensemaking organs that anybody trusts anymore. Co made this way worse. Mhm.

So we're looking for authority and suddenly here is this stage. Yeah. And it's it's the funny the you know the funny behavior you see right now on axe is some image that may or may not be or image or video.

So it could be AI generated or a screenshot and people just go Grock is this real okay so you're going to rely on Grock which is trained on X which is like a home for real news and fake news and you expect the model to be able to you know may maybe you could get you know I don't know there's potential ways in which you could fix that but it's a weird thing if somebody trusts the model to write them a loving message to their their parents about, you know, like happy birthday, you know, all this stuff.

And it does an amazing job with that. And it's like, yeah, and and there's in real life, people might get good advice from one person on one thing, but that same person might give absolutely terrible advice on the other thing, right?

And so like in real life, you're used to being like, okay, well, maybe I should get interpersonal advice from my aunt with my with when with regards to my family, but I shouldn't get that.

I shouldn't get her advice on business because I don't see her but but LLMs being like these all knowing knowledge engines that can quickly, you know, uh apply advice in a bunch of different categories and and people are already getting advice from them on on so many different things.

It's easy to slip into that kind of like idol. And I think part of the problem is here is it's pretty good some of the time, right? And so you don't know where it's good and where it's bad. This is actually like a general problem with LLM adoption right now even in industry, right?

We don't know how good in advance it will be on any specific task, right? And so it's it's like this is like an engineering and research problem, frankly.

Like how do we how do we figure out where the LLM is actually going to give us good advice or perform the task well versus when it's just going to make up random [ __ ] and then send us into like psychosis spirals.

We don't have an answer to that right now, which is part of the reason why people are taking this like pretty seriously. Even though maybe it's not increasing the total amount of crazy in the world, it is producing a new kind of crazy in the world. Yeah.

I guess I guess do you have a theory on if if with um I always thought it was interesting to kind of estimate, you know, what percentage of people are going to do Iawaska and like have some kind of psychotic break, right? It's from from just viewing the tech industry, you might say, "Oh, maybe it's 5%.

" Maybe that's way too high. Maybe that's too low, right? But you don't know. 5% seems incredibly high. If 5% of people in tech were going crazy from GPT, we would have a productivity slowdown in the country. Like I think GPT would go down.

I was I was talking of the people that do how many people have I was saying like 100 people that I know are are IA types, 5% of them safely seem like they had they kind of like got oneshotted a little bit. Mhm. They don't they're not necessarily crazy and like destroying their life or anything like that.

The big question is like is is is can models sort of like create psychosis and people that were not susceptible in the same way like that or or is it just there's certain group of people that are that are generally more susceptible to this?

And I would suggest it's that I would suggest that there is a group of people who are generally susceptible like something something maybe would have got them. I would suggest that it's probably something like like if if you go looking for this type of stuff 5% of the people are going to find it probably.

I think that that's pretty reasonable. Um, yes.

But I I think the key is that it's not it's not the ring from that horror film that I know Jordy hasn't seen where uh where like I truly believe that I could go on a silent retreat to the top of a mountain and only have a GPT prompt interface and spend all day for weeks prompting this thing trying to go crazy and I would come back unchanged just because I'm not I'm not predisposed to it.

built different but that's how they get you but but the other in advance how they get you nobody nobody goes to the Iaska retreat thinking [ __ ] this is going to completely overwrite my personality I guess with like some messamerican demon right like this just not how it works well yeah no so the the cons the the good thing about IASA I don't have a lot of good things to say about it but the good thing is that it requires somebody to really go out of their way you have to really go out of your way right you've got to for some time you had to fly to South America and And then people brought it up here, but it was still like somebody had to fly find a shaman, dedicate a weekend to it, chat GBT and other other LLMs.

Somebody could just be using it at work and then just like go down. Nobody's like using Iawasa at their tech job casually hopefully. Yeah, the availability the availability is certainly higher.

Like I to some extent again I I have similar perspective on this and my discussions with Monica have been similar where it's like this is for some people on the level of psychedelics. It needs to be treated that seriously for that group of people. It is that serious.

My perspective on on like psychedelics in general has always been never make drugs the most interesting thing that has ever happened to you in your life because that's how you get susceptible to getting your mind overridden by you know mean demons that live in mushrooms. Yeah.

Um to to some extent like it is psychoactive. You are participating in you are participating in it as a psychedelic experience in the sense that you are projecting meaning onto the subjective experience that you're having.

But we have this special machine now whose entire job is to do that is to pretend to create meaningful text. Right? You you can argue back and forth about how much it really understands, but the thing that it's trained to do is produce meaningful text in as many situations as possible. And then the post training.

You're so right. You're so right, Anton. It's not just meaningful. It's a statement. It is. It's recursive. It's not just text. It's a It's a It's a We're going to oneshot you into making TV on your entire personality. Uh my my question is like how do we how do we inoculate ourselves towards this?

Like how do we build forward? Jordi and I were joking a long time ago about the need for an Iawaska vaccine that you could take and then would make you immune to Iawaska no matter how much you took.

Uh and obviously there are some people who have slowly dosed themselves up with various chemicals to the point it's like having an alcohol tolerance having a caffeine tolerance and that's how you become a functional alcoholic. I don't I don't know if that's the path we want to take.

There are some there are absolutely some functional iawaskaholics out there in the world. it happens and they have not been one shot but they are immune essentially to changing because they've they've been working on it for so long.

I think I think when I have looked at uh you know I think there's a bunch of Reddit threads of people kind of reporting on their experiences like kind of extreme experiences when I look at those or when I talk to like Dan Shipper I'm like wow I'm really not using the models to the level that Dan Shipper is.

And then when I see these Reddit threads and these people are like, you know, 7,000 prompts deep, uh, you know, I'm like, okay, I'm really annoying compared to that where, uh, I I probably haven't cracked 50. Yeah. In a single chat, you know, maybe once, right?

Um, and I think that, uh, I'm curious what what what you think different labs should even doing about it. There there's a couple of pieces here, right?

Um the best thing that I found to be about to to like work as normie deprogramming for stuff like this is explain to them that every you are not actually having a conversation. Every single time you send a message, the entire history of your chat is also getting sent to a completely new instance of the model.

If you sit down with the person and walk them through that fact and let them peek behind the curtain as well, if you could like show them what's actually getting sent every time, what these calls look like, it demystifies a lot of this. Totally. It starts thinking about it differently, right?

It it it breaks you out of the frame of I'm interacting with something that is having a continuous conversation with me and which is recursively interacting with me. So, it's like no, no, this it's it's like a completely new thing every You need to see the magic trick for sure. Yeah, you see the magic trick.

And I think that helps a lot. I think to the point of the labs, dude, circle generative AI, which is we've had the worst goddamn marketing since this industry got started about two and a half years ago. It's been terrible.

First of all, calling it generative AI is such an incredible miss when the thing you want it to do is do tasks for you instead of like, I don't know, produce images of big breasted cat girls. That's not the primary function of this technology, right? It's not the generation that's important.

Part of the marketing has always been this mysticism about what the models can actually do, what secret technologies are hidden behind the curtains of the lab.

And then you have this es these esqueological pronouncements from from you know lab heads about oh this is going to make 50% of people unemployed in a couple of years and by the way it's going to be super intelligent and might kill us all.

Like that's feeding that's feeding the mysticism that allows people to engage with this. they don't really understand what it's capable of because they hide what it's capable of.

Part of that is the engineering problem of understanding what the models actually are capable of, which we don't know, which which needs a lot more research. Um, but part of it is like presenting things as like being constantly hidden behind a curtain. It's like, oh no, we're we're like we're creating magic back here.

We're we're giving birth to literally a god. And who wouldn't want to worship a god? Like if if if you literally are making one, then of course people are going to be inclined to worship in it. We need to maybe we need to start using more boring language here. Yeah.

I remember when do do you remember it's not good for fundraising. Do you remember during the open AI dust up when Sam got fired and then came back? There was this whole meme about like what did I see? Yes. What secret project was he working on?

And I remember there's an article that was like it was called like project titan alpha craziness strawberry. 06. And it was just it was literally just RL. It was just RL on.

It was just RL and it went out and and everyone did it and it's out there in 03 and it's out there and Deepseek is open sourcing it and it's like it's available. Like it's cool. It makes it better, but it's just RL on top of it was also obvious to the research community, right?

Like everybody kind of knew and and again, if you've been around long enough, you know exactly what QSTR refers to. It was a mystical thing. Yeah. Qstar was like, "Oh, it's going to be this crazy thing.

" It's like yeah that shipped and it's fine but it's like it's you know not not to get too like symbolic with it but you've got QAR and you've got QAnon and how much of that just like mentally intersecting in people's minds because you're you're you're like presenting it as if it's a conspiracy theory. Totally. Totally.

Right. At at a time again at a time when our epistemics are falling apart and you're deliberately making people not trust you because you're not telling them everything and you make a point of not telling them everything. Yeah. Right.

It's it's like people are going to project onto these things whatever they want to believe. Um and I think that the marketing is part of the driver. I don't know how to walk it back. And as you said, it makes fundraising harder. But frankly, this is also a transformative technology. It is going to change everything.

But is it going to give birth to a god that you should worship or or like should you take life advice from it rather than your friends? I don't know. Certainly not today. Maybe if we do have super intelligence, it'll give me better advice than my buddies, but not today. Yeah. Yeah. Um, how AGI pled are you right now?

Dwarash recently updated his timelines to say, "Hey, I don't even think it's going to be able to do my taxes until 2028. Uh, maybe something more like super intelligence uh in something like 2035. " Um, the the timelines are shifting around, but do you think that we're, you know, making progress?

Are we accelerating, decelerating, like how do it let's I'll put it this way. I think we know what the problems are, which means they're going to get attacked. Um, There there's a few points to this. So I've been I've been like drifting to becoming much more AGI pillar or like LLM architecture pill as time has gone on.

First of all, I don't think you need AGI for this to be transformative meaningfully, right? Like we've invented this machine that if you feed in enough data and compute about any particular task, it's going to be able to perform that task. I'm I'm a little bit at odds with Doresh here uh on some of this.

Um because like Okay, yeah, but if you had like if you did a pre-training corpus the size of GPT4 on just tax preparation is going to be able to do your taxes.

Um Shelto was actually talking about that specifically like a tax doing your taxes eval gets saturated and it's like oh it's saturated but it's like okay now they can do taxes that's great. But but here's the real problem.

So the problem with the EVA is like, yeah, it's doing great on the coding benchmark, but is it going to be able to do this coding problem that I just put in front of it? Nobody knows the answer to that question today. You can't predict from the benchmarks what tasks it's actually going to be able to perform. Yeah. Yeah.

But if the training is broad enough, like it might not be able to do the most complex novel tax processes, but it could do like the basic one that it's trained a billion times for, right? Yeah. Exactly. And again, it's just a question of getting more data into it. But if we want the general purpose system, Yeah.

There's like a bunch of problems that we need to solve, but we know what the problems are. Like one of the biggest ones which I keep harping on about here is the model currently does not know what it doesn't know, which is a big reason for it producing [ __ ] Right?

For example, uh when GPT tells you I don't have any knowledge beyond a certain cut off date, it's not because it has learned that fact by scanning its entire pre-training corpus. It's because in post- training somebody has trained that fact into it, right?

has been fine- tuned to respond that way to things that require dates later than its cuto off date. Um, we got to get rid of that if you want AGI. Uh, because then the model can say, "Oh, I don't know this, but I'm going to go and get that fact and I'm going to go retain that fact.

" And Dwesh's point on this like Dishesh is like big on continuous learning. Yeah. For this being necessary, like it has to go out and gather facts. I think there's a bunch of ways of doing that. You know, being the founder of Chromma, I think memory is a great way to do it. Um, it can just go out and store facts.

And the thing is to have the general purpose processing system uh that works on top of that. Um I think timelines are less important than applics interesting to talk about than applications.

Like it used to be you go to a San Francisco party where any AI people were present and you would get asked the timelines question, right? It's not that interesting a question. The question that I've been asking people is like okay great like forget timelines.

Imagine if I in my pocket have an API to an AGI that can perform any cognitive task to expert level human performance. What are you going to do with it? Mhm. And almost nobody I talk to has an answer in under a minute. And the copout answer is always, well, I'm going to ask it what to do.

And I'm like, well, it's expert level, so you're the expert on AI, so what do you think it should do? Because that'll be its answer, too. Um, and people get stuck. And and people really get stuck. And I think that this is really illustrative that the question now shouldn't necessarily just be about model capability.

It should be about how do we get this thing to diffuse through the economy? Mhm. And a big part of that is literally just asking like okay like a business is uncertain about whether it can apply AI to a particular task. Yeah. Right. They don't know what the cost is going to be.

They don't know how well it's going to perform. Right now we're racing forward. We're like putting all this all this money into these into compute into all that stuff. And by the way, that's great. America absolutely has to win the compute race. It's it's like without a doubt um not just for research but for industry.

But it's being under estimated like how much work there is to do to educate people how to use this effectively and how to estimate how to use it effectively. And I think that's much more of a barrier than like having AGI. Yeah, makes sense. Tricky. Anything else? Come back on again soon. Yeah. La last uh I'm curious.

Uh we we were talking earlier uh Sam was on Theoon last week and he he casually dropped that uh you know your conversations with chat GPT can be used could be used against you in a court of law. Yeah.

Uh it started going viral be for obvious reasons even though I think a lot of people don't realize that like all email and most internet service. Yeah.

I mean, that's my response here, but but but I'm but I'm curious if if like the path forward if LLMs could provide therapy for the masses or one of these use cases that that should have some privacy.

Do you think there's a technical solve there that's that's or or is it going to be more of a like some type of like legal regulatory solve? Well, we kind of have some of these legal solves for existing like for web two stuff, right? like we've got HIPPA, Furpa, we've got the Papa stuff for for protecting school kids.

Uh I used to have an ed tech startup so I know about that. Um the problem is is your GPT is general purpose and so if someone wants to use it as a therapist, right, which nominally would be covered by something like a privacy thing. And by the way, I don't know how this works with like GT GDPR.

Like God help you if the Europeans get involved in in some meaningful way in in the in the contents of the chat. Um, yes, there is a legal solve, but there's a technical problem of knowing when the legal thing applies, right? Like at what point is it a therapist?

When I log into a therapy app and it's like covered by HIPPA, I know like I know what it's for. But if I'm like if I'm just like kind of having a bad day and I start treating GPT like a therapist, like am I covered by that or not? It's a technical problem to detect that should be coming. I think that Yeah. Yeah.

But but it's this Yeah. same thing, you know, if you're getting therapy from your business partner informally and you're saying things that are wrong or whatever talking about like like people I think people should have the they they need to have responsibility over the context of of the context of the conversation.

So I do think it's possible that like chat GBT if they want to attack therapy should create a you know HIPPA compliant version of their product that's like a separate app and people should just know hey yeah setting expectations is the right thing here ultimately right but but it's so hard to set expectations around a general purpose technology.

Yeah, that's the thing. Especially especially for the average person like you you everyone everyone on this who comes on this show is suffering under the curse of expertise, right? We know way more about this stuff than the average person. And we don't realize how much more we know than the average person.

And I don't know if it's reasonable to expect the average person to like know that they shouldn't use this chatbot in therapy because then that might be disclosed in a court of law. That's a lot of things. Like there's so many steps you have to understand there, right?

The average person doesn't read like the error message that comes up on their screen when their computer's doing something weird. You're not going to get them to like understand tech privacy law. I I don't have a good answer, but I think you have to like set expectations somehow effectively. Yeah.

That's why uh you've you've seen like in these high-profile like murder cases, somebody will type in to Google how to hide a body. It's like, oh, like I thought that like that was that would be cool if I No, it's the same. It's a database. It's a website with a database back end that saves everything you type.

There are logs, but yeah, people don't know that. It's a good point. No. Anyway, great job. Yeah, I think OpenAI should buy a billboard on AdQ that says OpenAI is subject to the exact same rules as Google and every other consumer internet. Yes. the rest of the consumer internet. They should get on adqu.

com because Adqu makes out ofome advertising easy and