Databricks co-founder Patrick Wendell: AI coding costs went exponential, smart routing cuts them 30%

Speaker 1: Agents, meet the canvas. Your AI agents can now create and modify your Figma files with design system context. We have Patrick Wendell from Databricks. He's the co founder and VP of engineering coming on to talk about AI coding costs. Patrick, how are you doing? What's up guys? What's up?

Speaker 2: What's happening?

Speaker 1: Glad to have you on the show.

Speaker 5: Long time listener, first time caller.

Speaker 1: It's a pleasure to have you here. Maybe since it is the first time on the show, give us a little bit of background of what you're focused on day to day because I want to talk about AI coding costs, how that interfaces with your customers and your business internally. But having a little lay of the land might be helpful.

Speaker 5: Yeah. Absolutely. Have you guys had any Databricks folks?

Speaker 1: Oh, yeah.

Speaker 5: Any of founding team yet?

Speaker 1: Oh, yeah. Yeah. We only yeah.

Speaker 5: I think twice.

Speaker 1: Okay. But then we've also hung out with him a few times off

Speaker 4: the To

Speaker 2: be honest, my some of my favorite moments of podcasting have not actually been podcasting. Yep. It's just we we hung out with with Ollie recently and for like two hours. It was amazing. We were just all all three of us ranting.

Speaker 1: Yeah.

Speaker 2: It was incredible.

Speaker 5: Awesome. Well, Ali and I are co founders. So I'm one of the the founding team. We left we left UC Berkeley.

Speaker 1: Yeah.

Speaker 5: It was a research group. There was like some grad students and some faculty. Ali was a visiting faculty member, I was a graduate student

Speaker 1: Cool.

Speaker 5: And a few others, a few of the rest of us. And we left to start Databricks in 2013. We've always been interested in like the intersection of large scale data processing and what was then machine learning. Mean, the company actually started very focused on early machine learning stuff. Yeah. You know, now it's evolved into like AI, basically just deep learning techniques. Mhmm. But, you know, today we build data and AI infrastructure for a huge fraction of sort of the global 2,000. You know, we have we have 20,000 customers, think, as as of our latest announcement, and we just basically help yeah. Thank you. We we help businesses who wanna store and take advantage of data, increasingly, involves leveraging AI in the way that they take advantage of their data. So yeah. So that's kind of that's kind of what we do. And then my my personal role, I I'm responsible for our AI products. Mhmm. But I also am the one internally at Databricks who has been kind of the champion of of aggressively adopting AI tools at Databricks. Sure. And, you know, we have a we have more than 10,000 employees. So so we were among the earliest to kinda roll out at scale tons of different, you know, AI tools for developers and other employees.

Speaker 1: Yeah. So take me through that You're

Speaker 2: you're the guy the CFO comes to.

Speaker 1: You're token matching? Patrick.

Speaker 5: We Yeah. I'm the guy where he's like, what's this? Like, how do we project these costs?

Speaker 1: Yes. So before we got there, walk me through the history of AI tooling because there was a moment when I remember I think it was in the very original ChatGPT demo on three point five DaVinci where someone got it to spit out a to do list app in React just from the just from the context window. It didn't even have tool use yet and people were like, wow. Is a glimpse

Speaker 2: deep script.

Speaker 1: Of what's coming, something like that. And and so there was a moment where people would go to LLMs and sort of copy paste some code. Then we got the cursors and the wind surfs, then the clogged codes and the codexes. What's been the journey inside of Databricks in terms of actually getting value and how have you been measuring it? Just walk me through some of the journey.

Speaker 5: Yeah. So the first like product market fit in Gen AI was this more personal chat type use cases. And that did translate into the business. You know, a of the early AI companies, the foundation models built like an enterprise version of their initial chat product.

Speaker 1: Yep.

Speaker 5: But the and and it was it was somewhat useful. It could kind of, like, read your business data and stuff like that. But but I would say what the real breakthrough was when the coding and agentic models got a lot better

Speaker 2: Yep.

Speaker 5: And and could actually generate useful sort of enterprise workflows and and in particular, generate code. I mean, by far the biggest ROI we see internally and I think is true industry wide is developers are expensive. They take a lot of you know, the every company needs their engineering team to move faster. And if you can get them something that improves their productivity meaningfully, that's of immense value. So I would say that the real ROI curve significantly changed maybe eight months ago or twelve months ago as the first really good coding models got there.

Speaker 2: How do you talk about ROI with coding models to, maybe other engineering leaders, your customers? And and how do you talk about it with, like like, for example, like Databricks' CFO. Right? Because a lot of people will will look every engineer will tell you, like, yes, this thing makes me a lot more productive. But at the same time, people will try to dig down into the data and be like, okay. There's a lot more, You're shipping a lot more code, but I'm actually looking at how many new things that you've shipped and maybe it's not sort of rising at at at the same, at the same speed. So how do you where how do you kind of like wrestle with that and prove ROI month to month?

Speaker 5: Yeah. So ROI has, the benefit side and the cost side. And on the benefit side, we do track a lot of different engineering output metrics. Now none of no one metric is perfect. Right? Like, you can look at how many pull requests are coming out, how many features are coming out, how many lines of code are being written. None of those is independently perfect, but they can give you a sense in aggregate of, like, you know, r and d is a big machine. You put in resources, you get out features and code, and, you know, how much more is coming out of that machine. And the the results there are pretty good, like, much you know, at in aggregate, maybe almost doubling capacity from a fixed size team, and then in certain teams where they've highly optimized it, they're, you know, moving even way faster than that. That's where they've optimized their processes, basically, to take better advantage of AI. The cost side, just quickly, is where we actually encountered some problems. So, you know what, at the beginning, we were just trying to at the beginning, we had the opposite problem. No one wanted to try the new stuff. I was going and bugging everyone, tried it and we tried it and we tried it and we never got to the token maxim kind of thing but I do think that that arrived out of a actually well intentioned thing of just, like, trying to get people to try the new stuff. Mhmm. And and what happened, though, is that once we got people to use it, we just started seeing this exponential cost curve. Like, the these these tools all do consumption pricing now. So we're not paying a fixed seat per user. We're just a user can, in principle, spend an unbounded amount of money. They can run a little loop on the most expensive model. And so we started seeing basically this, like, exponential growth curve that, you know, although we were getting the the two x or more output from from our engineering teams, it's just you can't like, if if your costs are growing exponentially, you you're gonna hit a problem. I mean, at some point, it's gonna exceed it's gonna exceed your revenue if left unchecked. So so so we actually hit a point where the costs were threatening to kind of reverse the the purported efficiency benefits of having AI tool adoption. And that's when we that's when I actually started to get very, very involved in, okay, how do we think about managing the costs long term? Because we need to we need to get both the productivity benefits, but we also can't have it be outshined by just the amount of money we're spending. And and and, you know, at around that time, I also talked to a bunch of other you know, we're we're in touch with Coinbase, in touch with Uber, in touch with

Speaker 1: Yep.

Speaker 5: Other tech companies that are, I would say, the very early adoption edge of how many employees, you know, giving tens of thousands or more of employees broad coding tool access. And and, you know, collectively, we we kind of found some techniques that actually worked quite well in terms of of of curbing that that exponential cost curve in a way that keeps costs, you know, constant or on a per head basis roughly constant even as we have more and more consumption.

Speaker 1: Can you help me understand the various ways to save money? I'm I'm thinking of this because the Unity AI gateway, your you you the the the smart router here has cut average task cost by 30% while maintaining similar quality. We've all seen the trade offs on the curve of different levels of intelligence at different costs. But there's like an internal change management coaching that happens where a task that can actually be done faster as a human costs 100% less in token costs. And there are some times when you just use the wrong model for the particular task because you don't realize that a smaller, faster model can actually do that task better. And then there's also the flywheel of a developer who's sitting there using a big model and waiting twenty minutes per prompt. Sometimes if they're only waiting two minutes per prompt for using a smaller, faster model, that can save more time because they're being more productive. So the shape of productivity is more complicated than just price per token at a given intelligence rate. What is the full picture that you see companies having to balance out?

Speaker 5: Yeah. So that's a great question. In the end, we had to apply a few different techniques.

Speaker 1: Mhmm.

Speaker 5: The our favorite one is just when more efficient and better models are released Yeah. And those are sometimes open source Yeah. Increasingly. Sometimes there's also really good high efficiency models that are not open source. But if you just that's almost like a rising tie. Like like, it it just shifts the Pareto frontier

Speaker 2: Mhmm.

Speaker 5: So to speak. The frontier expands. Now Yeah. Even if no one changes their behavior Mhmm. You suddenly get, you know, you get the same amount of output for less cost. So so those are our favorite type of changes because they don't require any user behavior change. They don't require, you know, any fancy routing. It's just like the everything just got cheaper, basically. And and I and I I mean to emphasize that because it's happening quite often. Like like, if you look at every week now, there's probably five models released between proprietary and and open source vendors. And not every one of those will be a new sort of efficiency frontier, but maybe one a week or one every couple weeks is. And so it is a nice place to be in that you just have this deflationary pressure coming in and, like, making things cheaper, making things cheaper, making things cheaper. Mhmm. But what you need to do as a company is you need to quickly move traffic over to those cheaper models. You know, if a new model comes out, but no one's actually using it in your company, it's like a tree falls in the woods. So so among the the technique we most liked, because it requires no changes in anyone's behavior, is just quickly looking at new models as they come out, doing the right analysis and benchmarking. And then if they are cost competitive, we very quickly shift workloads over to those models. That that is actually by far the most impactful thing we we've been able to do.

Speaker 2: What are what are some AI use cases that are, like, non coding use cases that you're seeing across the Fortune 2,000 that aren't being talked about on x? Oh.

Speaker 1: Great question.

Speaker 5: That's a great question. I mean, I would say not to avoid your question, but but the dominant, at least as it comes to cost, remains software engineering workloads. Sure. Because because you just have this property where you know, when when a human is simply asking a question of an AI and getting an answer, it's bottlenecked on that human's brain, basically. Like, there's just only so much the meter can spin because I'm interpreting that answer and I'm sitting here and spending a minute or two before I ask my next question. When when you know, software is this sort of digital artifact. It's this thing that has value, but it's not a concrete, you know, physical good. And these AIs can just iterate on the software, make it more valuable, make it more valuable, make it more valuable, and they can kind of accumulate value over time. Mhmm. And they don't have to wait at sort of a human response speed. So so software remains dominant. Now you asked about nonsoftware stuff. Definitely, the next phase of use cases we see is people just trying to automate, like, everyday processes that they're dealing with. You know? They might be a knowledge worker that's you know, we we are we're a data company. So in a in a in a a typical enterprise, maybe you have a handful of software engineers, but you might have a thousand people that work with data every day. And, you know, they're they're sitting there doing really drudging through tables and running queries and trying to figure out if this metric is defined in the right way or using spreadsheets or whatever. And and we've actually seen a huge amount that we can automate their workloads, and we have, you know, various products around that at Databricks. So I would say it's like stepping down the ladder of sort of technical depth of the employee with software engineering being an early one, but but a lot of other types of knowledge work job families, I think, can can get a lot of productivity wins from that.

Speaker 1: I I I would think outside of coding, although some of these collapse into coding tasks once they're automated, but customer service, business intelligence, and probably design marketing ad creation is, like, coming up on the frontier of of capabilities. Even if it's not being used for the final deliverable, Every Fortune 2,000 marketing agency is at least using gen ImageGen in the process for like storyboarding or design exploration.

Speaker 5: Yeah. Totally. I don't know. Yeah. On the coding side like what

Speaker 1: we did is Yeah.

Speaker 5: We actually took we took a lot of these learnings like adopting the new models, doing routing, like you said, routing can get you another 30 ish percent.

Speaker 2: Yeah,

Speaker 5: yeah. And then there's other types of like pretty traditional engineering optimizations you can do to just you're just squeezing, squeezing, squeezing, can I get more out of these models? And we ended up productizing that because we realized every other company has the same problem that we have. So that's our you know, we have this Unity AI gateway which Yep. Which lets and, you we have thousands of customers using that now.

Speaker 2: Yeah. How do you how do you see that the routing market evolve over time? You have you guys, OpenRouter, there's a bunch of other company. Like, it sounds theoretically incredible to let there just be like this absolute dogfight of competition and then you're sitting in the middle, you know, helping helping your customers make sure they're they're getting the job done while spending as little as possible. But it it feels like routing could end up being like equally competitive as like the models themselves as every company decides like we're gonna do this.

Speaker 5: Yeah. That's certainly our view. I mean, like, we've been pulled into this by our customers actually who who just have this problem. The costs are getting really high. There you can exploit the fact that different models have different strengths and weaknesses to reduce your costs. And in a world where it looks like there's less and less margin on the the actual AI models themselves, like, this is an area I I think the the routing and optimization, I think, actually is a a quite interesting area to go into as a business. And another nice thing is, like, that area has no high fixed costs. You know? Like, just just to do the routing itself, you don't need to buy a gazillion GPUs, and you don't need to sort of have, like, a huge amount of capital expenditure. So it's a very asset light kind of business model when you're just doing this optimization on top.

Speaker 1: Unless you accidentally use the god model to route the queries. Yeah.

Speaker 5: You gotta be careful because some of the routing itself uses AI.

Speaker 1: Yeah. Exactly. But but the these

Speaker 5: routing models need to be extremely fast. So they're Yeah. Very small and efficient models. Not like these massive, you know Yeah. Huge AI models.

Speaker 2: Yeah. Being being so asset light Yeah. Means that you're gonna have competition. But I think that that in many ways ends up benefiting Databricks because you guys have this massive sales force, these deep integration, you know, deep relationships with many of the most important customers already. So

Speaker 5: Yeah. And also, it's just like hard to do it well. I mean, we have a large research team and, you know, our research team isn't as focused on making the models themselves. We're a lot on focused on all the practical issues of using the models, which which itself is like there there's quite a lot of open research problems there too. So I I think the there's significant IP in doing this well is my Very cool.

Speaker 1: Well, thank you so much for coming on the show.

Speaker 2: We gotta talk to the rest of the the founding team.

Speaker 5: By the

Speaker 1: way. Roundtable with everybody.

Speaker 5: I got a parting question.

Speaker 1: Yeah. Yeah. Yeah.

Speaker 5: How much diet coke do you guys go through every show?

Speaker 1: I drink three every show across two to three hours.

Speaker 2: Keep just in the chamber. I honestly rarely drink it. Yeah. I I'm comforted knowing that it's there.

Speaker 1: And then maybe I'll drink one of later.

Speaker 5: Jordy, you kind of nurse it over there and John What

Speaker 1: you don't see is that before the show I drink two to three Yerba Mate's from Mataina, Andrew Huberman's podcast in a can. Also recommend those. Okay. Funny caffeine.

Speaker 5: Diet Coke just keep things they kind of just keep things moving.

Speaker 1: Exactly. It's nice and stable just to, you know, we're in the tens of milligrams of caffeine. It's not a Celsius where I'm going to crash. It's it's the ultimate. It's the drink of kings.

Speaker 5: We know this. This is guys. Thanks for having me guys.

Speaker 2: Yeah. Great to meet you. Let's do it again soon.

Speaker 1: Yeah. We'll talk soon. Cheers. Goodbye. Let me tell you about the New York Stock Exchange. Wanna change the world? Raise capital at the New York Stock Exchange.