Simon Eskildsen on Turbopuffer: making vector search 10x cheaper and why AI is starving for context

faster. Figma designs helps design and development teams build great products together. You've seen the Turbo Puffer, the puffer fish live on the show for the past is always with us in spirit. He sits right next to you. I'll let you hold your Oh, yeah. She's beautiful. She's beautiful. Yeah. Give us some background.

You were at uh Shopify, not Spotify, for for years. Trenches. What is that? No kidding.

It was a common uh mixup course and uh back in the day when you actually flew candidates into town sometime would some people would show up and think that they were interviewing really Spotify course because they're just like some like I don't know like Sweden Canada I think to some Americans it's the same thing.

It's pretty similar. It's it's not America and therefore you know. Yeah. I mean yeah Shopify Shopify is a like powers the world's commerce. How long were you there? 10 years. Yeah. Eight years. Eight years. What in what what was your role the whole time? I worked on the infrastructure the whole time.

It would mostly be basically I I I don't think you would uh make this claim, but when you think about how important infrastructure is at Shopify, where when your service goes down, you cost your customers real dollars. Every minute that it's down is dollars out of the retailer's the brand's pocket. That's right.

I mean it's you probably had some crazy on call I was uh there was a there's about six or seven of us that were on the last resort pager in the duration that I was there um and yeah sometimes you just get woken up in the middle of the night and some percentage of of everything was down um and you just had to figure it out and I think on once you once you've been on a pager like that for so many years for better or worse it changes everything about how you write software those lessons I've now put into the today.

Justin, you mean you mean you're not gonna rush something because, you know, if you do, it'll create, you know, a panicked 2 a. m. , you know, pager call and you're going to have to be is just more consequential. I think there's I think there's two things to it.

So, when when you know that this line of code can wake you up, you are just very conservative in what you write. You're very conservative, but it wakes you up. Yeah. It wakes you up and uh or even worse, it wakes someone else up on the team. And so you really just try to go for simplicity.

And I think that's also one of the other things that I really benefited from being at one company for such a long period of time was that you see how software ages. You know, here in in we're not in the valley today, but in the valley you sort of like play your LinkedIn like a portfolio, right?

Were you like two years here, three years here, one year here? And you don't get to see the both the stuff that went through some crazy design process and aged really poorly and some of the stuff that was just someone doing a quick hack to just like you know duct tape on the on the ship.

Um and it lasted 10 years and no one would have changed anything about it. And so once you see that cycle go through many many times and in parallel across the entire company once again changes everything about how you design software. What do you think the impact of agentic commerce will be on Shopify's infrastructure?

So in one way it's like well you don't necessarily need a front end anymore I guess and so the load on the system comes down but on the same time maybe AI chat apps in general are like scraping and hoovering up data like every Jordy was making this point about uh Google Trends data maybe being uh sort of everybody's been posting Google Trends screenshots and it they all look like these crazy spikes because and and anything you search but people have been sharing them in the context text of economic indicators.

So, it'll be people searching if you if you look on Google Trends for help with my mortgage. It's just like vertical. And so, it's tough to read into that because is that just more humans that need help or is that an AI agent running the same type of search over and over and over?

If every query becomes 100 queries, then all the like the the stress on the infrastructure should go up. I don't know. How do you how do you puzzle it through? Yeah, I think um I think I just immediately my thoughts and prayers go to the search team at Shopify.

Um and I mean there's always going to be a lot of fronts where commerce is going to be done, right? And that's what Shopify was so good at, right? There's the store that gives you a particular experience. Your storefront that you can create which gives you a particular experience.

You can blast it out on all these different channels. And I think on inside of the chat consoles, it's it's it's I think it's a great front and I would love that because now when I go on some of the big marketplaces, um I feel like I'm getting the LLM slob version of physical SKUs. Yeah.

Um and I think Shopify has done a great job through both the shop app but also just in general of you know arming the rebels that they say um to ensure that their actual entrepreneurs are standing behind their products to combat the LLM slop equivalents of all these SKs right that we see and we all accidentally buy.

How do you think about the business uh kind of like at at Shopify? Like the stock's done so well, the company's done so well.

Uh at the same time it feels it feels like there's been uh just a a continuous motion to just try and bring the take rate up essentially like uh it started with you know you bring your own payment provider then it was Shopify payments that's a little bit more of a take rate then Shopify plus obviously I'm a customer love Shopify plus I was one of the first customers uh back in 2013.

You're one of the best customers. It was the year I started they launched Shopify Plus. I was one of the first customers. Yeah. I remember talking to my rep on the phone and asking for a bunch of features. Um, but uh, but then there was the actual shop app.

There was the idea that there would be like maybe an ad platform in there like that would be a direct competitor to Amazon that you start your your commerce journey. Uh, the agentic commerce if your people are starting their commerce journey in OpenAI products that feels like that take rate won't live with Shopify.

Do you think that there's any uh, like hard feelings there? Is this just like like the the the the chips have fallen and you got to play the game on the field? How do you think it's being processed internally?

Well, I think I think you you I I look at the relationship between chat GBT and Shopify not that differently than Meta and Shopify and that like where is the discovery happening? But Meta was taking a shot at Shopify, too. They were trying to do in inapp purchases.

So, you would buy with a card saved in your Instagram account. They backed off of that. Um there's just to me there there's so many other steps behind after you hit buy, right? What happens when you need to do returns? Who's handling shipping? How does it the order get tracked? Right. Oh, I had the size wrong.

I got to send it. You know, there's there's still like this c I feel like Shopify's role totally. Is this centralized? It's almost is the CRM basically, right? How do you I think that's right.

I think that obviously Shopify the business is wants to own as many of the canvases as possible but where where they can't or someone else is going to do a phenomenal job at it they'll integrate. We don't know the details of these partnerships, right?

The take rate might be much better than we assume, but you you both sell products on Shopify, right? So, you know, you know how much there goes into the back office, right? Building a team around it. Um, and and all of that.

And I think Shopify will always try to do a great job at helping the merchant and putting them into as many places as possible to make them successful. That's always what it's been been about for them. Yeah, it's never really hit hit as like the tax.

Like every e-commerce entrepreneur will talk about like the Google tax and then maybe they'll say the meta tax. They're usually pretty happy. Yeah. Stripe tax, but people are usually saying like, "Yeah, I pay a couple thousand bucks for Shopify because of Shopify Plus, but it's like a great piece of software.

I'm getting more the value, so it doesn't feel like uh uh like a direct value exchange. " Sorry. Uh Jordy, what were you saying?

Um uh I I wanted to ask because as we were talking about the meta AI uh with display glasses and one thing that uh I was interested to get your point of view on is when we demoed them they had this live AI functionality that could do translation. It can give you directions.

It can it's basically you know constantly uh taking in data uh from what you're experiencing what what you're talking about. Uh, and one thing that was notable is right now they can run that for an hour before your battery completely dies.

And so I wanted to kind of uh since you are are uh quite a exponentially more technical than us kind of get a sense of like just why that is such a hard problem.

I mean I would I would I would assume that they just they only found that the product worked with a certain amount of parameters that they needed to put in the glasses. Yeah, they presumably run it inside the glasses because they needed that to get the latency to be good enough.

And so, you know, let's say this is a billion parameter model, right? And then now you have a couple things fighting, right? They're going to try to decrease the amount of um of power that you need to power this.

And if they found sort of an equilibrium right now at that uh at the amount of power that they're using, that would be the first iteration. And the batteries will also get better. But I think the models will get uh more intelligence per watt sooner than they'll get the batteries there. Interesting.

I I would take the other side of that. I think all the inference is being done on the phone and then piped back into the glasses. It seems crazy how how constrained these glasses are in terms of form factor, but I agree with the general point.

Yeah, I I have no idea about batteries being the context here is that we're the ones locked in on the news and and Simon will tell us how how how it all um what are these glasses like this? So, they don't have a chip.

They're No, I mean they they they they have chips, but the chips are mostly uh just reconstituting graphics that are essentially rendered on the phone. So imagine it like the like the Apple Watch is getting most of the data processed in an app and then sent to it.

So it's doing some ondevice processing and uh and the Apple Watch over the last what 10 generations has moved to uh it it can you know listen to music without your phone nearby. But for these glasses it's such a tiny form factor.

I have to imagine that almost everything is being done on the phone and it's acting more like a Bluetooth headset peripheral just kind of porting information back and forth. You take a picture, it goes onto your phone, then it's filtered on your phone, saved on your phone. It's doing everything on your phone.

We talked about Shopify. We we ranted a bit about Meta Meta's glasses. We should you should probably I I'd love uh to hear the actual genesis of Turuffer as well since we're here for a few minutes and then I want to do some timeline. Sure. Sure. Yeah. So yeah, Turbopuffer is uh is um the company that I co-founded.

What we do is we we build a search engine that helps you index enormous amounts of data and so it's used by um by cursor, by notion, linear, superhuman um and these are companies that have enormous amounts of data that they need to connect to AI and generally when you do that you need to somehow search over it, right?

So when you're using a cursor agent, you need to draw in context from potentially enormous code bases. Um, and that's what we helped them with. And we found a way to store the data in a way that's u tens even a hundred times cheaper than some of the solutions that came before.

Um, all of this came out of um I mean if Shopify had touched search and we talked about systems to get woken up by before, the worst system to get woken up by was the search system because the recovery time was so long. Um, and I I didn't think I would ever work on search again.

But then I discovered um after Shaba I was helping some some friends companies um I was discovering that search was only becoming more important because these AIs were so hungry for more context to sift through what's so hard about search like why why is Gmail search is so bad iMessage search is so bad like why is it such a even even for noni applications where I'm literally just searching for plain text like why it feels like it feels like no at least in Gmail search which I I'm running into all the time.

There's no context around like rolling up what the email is actually about. And so if something's in the footer or something's in some cookie that's even white text at the bottom or some URL, it'll just immediately show up and just say, well, this was a string match.

Uh, so yeah, where does this go and why does it did like was it just I remember search used to be good. Is it just like we got more data and then we need new methods? I think I think there's a couple things at play. One is that we have we we're experiencing in more products better search.

So then when we get bad search, we're really allergic to it. I think also that uh email search is is is a weird wart because you expect it to be somewhat exhaustive, but you also expect it to show in date order.

Like what your email client probably wants you to do is show the best match, but that might be a match from 10 years ago. Exactly. And so there's email search in particular is weird.

But I think underestimated because what you're trying to do is you're trying to take these these strings of text and try to turn them into some semantic thing. Y it's it's been very difficult. Even the chat GPT app is bad at search. It's crazy.

You think it would be the best because it's the most cutting edge puff and I Yeah. Yeah. And I'll go in and I'll search and it will take a it takes a long time just to search because there's so much text, but it still doesn't it'll be like, oh, there's a string match here.

And I'm like, well, that's not the that wasn't the gist of my that wasn't the actual topic of that particular interaction. I remember the interaction and I would I would roll it up in this particular way. Uh, but it's just doing string matching all over the place.

So, and and so I think what's happened recently was that um before it just used to be we would see these tail searches at Shopify where someone searches for a red dress and they only have a burgundy skirt and that used to be sort of a PhD level problem to solve in the tail fuzzy search very difficult and what what we found out you know a few years ago and it the big the big companies the fan type companies have done this for a long time was that you can kind of cut the head off of a model and then just take the numbers that come out plot them in coordinates system and then you plot all the things in a coordinate system and then when you search for something you search for you plot that in the coordinate system and then the things close to it are the results and that works really well if you're searching for music right oh this song what's close to that in the coordinate system and so on and using that for search um is part of what um what Turboper helped commoditize because it used to be very difficult you would have this kilobyte of text and it turns into 20 30 kilobytes of of of text um we have mutual friends in in Readwise Right.

And Read Wise is this app that helps you read articles. And the the sort of the founding idea of Turopuffer was that um I helped them build a little recommendation engine. And we ran the numbers on it and it was going to cost about 30 to 40 grand a month to run the recommendation engine.

And at the time I just optimized just helped them optimize into running at three grand a month on Postgress. So they would have spent 10 times as much for one feature which was search to do vector embeddings of all of it. So it worked great, right?

We had sort of these problems of like okay couldn't quite remember the word. It's way less intensive than if you're cursor and you have power users, right? That are just constantly Yeah.

But even for cursor and and readwise, you have you just have so much data and the the value for the products that they can ship on top of a search engine is not $20 per user. So they need to get the economics to the cents per user.

And that's that's where I found okay, we're not going to pay 30 grand a month for this at Readwise. So there must be other companies that are constrained in the product that they can ship because of the economics of this new wave of search. Timeline. Set some timeline. First, let me tell you about Vanta.

Automate compliance, manage risk, improve trust continuously. Vantis trust management takes the manual work out of