Radical Ventures partner Rob Toews on pre-training plateaus, robotics data gaps, and Waymo's civilizational implications

it was cool which is fantastic. So we'll bring in Rob and ask him a bunch of questions about artificial intelligence. How are you doing? What's going on? I'm doing great. Thanks for having me guys. Welcome to the show. Uh what is top of mind for you right now in AI?

What's the biggest thing that you've gotten right over the last few years? I know you have that scorecard 10 predictions and then you rank yourself. uh as you look back on all of your predictions, what's the big one you got right? What's the big one maybe you missed on? It's it's a great question.

Yeah, every uh every year I write a column of 10 predictions for the the year to come in the world of AI. And to keep myself intellectually honest, I go back at the end of the year and grade which ones were right, which ones were wrong. Let's see.

So far this year, um I uh one of the predictions I made at the end of last year was that um President Trump and Elon would have a messy falling out, which would have various implications for the world of AI, which looks like it may be playing out. We'll we'll see how that all comes together.

Um yeah, we were reading that today because Elon is no longer a special government employee, but at the same time, messy seems like the key word. There might be a split, but if it's clean, you're getting a yellow like a clean split. We need satellite imagery of whether the te the red Tesla is parked at the White House.

If there's a Tesla still at the White House by the end of the year, I think you got to you got to fail that one. Yeah. Yeah. Not messy yet, for sure. Yeah. It's interesting because it is kind of a narrative violation because half of the half of the world was like they're going to be together forever. They're in love.

They're going to be partnered for four years, maybe for 10 years. Um and then the other half was like this cannot last. It's going to blow up and it's going to be super messy. And it feels like right now it might just be, hey, back to business. Uh, sleeping in the factory. Who knows? Yeah.

Well, the the the book is not yet over, so we'll Yeah. Yeah. Yeah. Anything could happen. Anything could happen. Um, what what about um things that you feel like have played out as expected.

Have you been uh have you been uh kind of reassured or shocked by uh the data wall, the pre-training wall, the shift of the focus to reinforcement learning, uh the importance of tool use going forward, the importance of productization, uh versus this like this narrative around, oh yeah, just scale up the big transformer model, GPT7 will be ASI done.

Yeah. Yeah. No, I think I think those trends are all very much playing out and it's been interesting to watch.

So I think it was it became increasingly clear over the course of last year that the pre-training scaling laws really were plateauing and um this narrative emerged around towards the end of last year as opening I was increasingly teasing its reasoning models and it its O series of models that uh there was this kind of next frontier and next vista for scaling which was inference time compute and and uh you know um scaling reasoning models and so forth and the reasoning models 01, 03, 04, you know, deepseeeks, R1, etc.

They have been very powerful and have unlocked a lot of new capabilities and use cases and so forth. I think the jury is still out as to whether they really represent like this next massive uh runway for scaling that will take us many years into the future.

I think it's still maybe the case that the the fundamental underlying capabilities of the models are not growing as quickly as they did in the like GPT2 to GPD3 to GPD4 era. But importantly, that may not matter that much to your point, John, because I think the value and the activity is increasingly moving up the stack.

And even if all model capabilities basically were frozen in time today and there were no further advances, there's literally trillions of dollars of economic value to be created by just figuring out how to productize these models.

build particular solutions for particular end markets and even the big frontier labs openai anthropic etc are increasingly focused further up the stack so I think that's where more and more the action will be yeah what what is your take on how the shift from pre-training to uh to test time compute affects like data center buildout the need for these like hypers scale stargate like massive data centers that stuff made a ton of sense in the leopold ashen Brener, we're just going to scale it up and get this massive data center going for like the biggest pre-training run ever.

Uh, in in a test time inference regime, it feels like maybe it's more on demand. Maybe you don't need as many big superclusters, but am I just thinking about that incorrectly because maybe we still need the massive clusters because the rate of token production is just going uh going completely parabolic.

No, I think that I think that narrative conceptually holds true and and I think a key insight as you touched on is inference can be done in a much more decentralized way. You don't need to have like hundreds of thousands of GPUs that are all really tightly interconnected like as close together physically as possible.

And so in a world where more and more of the compute is inference either like productionization of models or even inference time compute that can be done on GPUs that are more spread out and you don't need to have these like matt like 5 gawatt clusters.

The counter narrative is the like and I'm sure you guys got sick of like hearing Jevans paradox get referenced like a million times over the past few months but we love Jeb's paradox. We think it should be taught in middle school. I we almost got it tattooed right here. Never forget. Never doubt. never gets.

Yeah, that guy that guy has really been immortalized. But uh but I like I think there is a narrative that even as inference becomes more important, you as computer and cheaper, there will still be bigger and bigger and bigger pre-training runs. And so that will justify the the massive capex buildout that we're seeing.

Yeah, I've always been interested to see if there's going to be like the the the the you know the big transformer training runs like the GPT45 training type training run in image diffusion or in robotics or even in I was thinking about like you know we we we've kind of solved chess engines but like what happens if you scale that up seven more orders of magnitude like just do you get some weird outlier uh outlier scenario the the question I always have is like when can I see it on semi analysis when can I see it from a satellite image, then I know that the big training runs happening.

I'm wondering, are there any other areas or or or regimes of of training that aren't just predict next word that we could see a huge data center run happen or or is it really just text is the only option for that scale?

or will we see like a V4 training run and we'll be hearing about like, oh wow, it's at it's at 5 gigawatt scale because V4 just needs that much pre-training. Yeah. Yeah.

So, this this was one of my predictions for 2025 is that we would start to see more and more scaling laws in data modalities other than text, other than language and and I do think that we're starting to see that play out in robotics, for instance, in biology.

I mean there's like the the the the whole premise of the whole justification for building such massive compute clusters is this notion of scaling and as you increase compute and increase data the model gets reliably better and we are starting to see like nent signs of that in some of these other data modalities.

Um I think one one important challenge or constraint though is there are very few other modalities that where there's as much raw data available as there is for text like there is not an internet for robotics training data or an internet for biology even and so the training data sizes can limit it like it it doesn't make sense to have a massively overparameterized model if there's just not enough training data and so for the foreseeable future at least like I think the amount of training data available will cap like how big the biggest robotic foundation model can be for instance relative to the biggest general purpose language model.

Do you buy the whole like uh robotics thesis of like but but transfer learning is really effective and like we can totally simulate this because we have Unreal Engine.

It feels like a little bit of potential cope being like well we know that you don't have the trove of the robotics data and so you're coming up with something that hasn't worked in other modalities but at the same time Unreal Engine's pretty real and you can imagine walking a robot around and learning a bunch of stuff if you train.

Um so what is your take on on robotics data and transfer learning simulated learning that type of stuff?

Yeah, I mean I think there's no question that robotics foundation models are getting increasingly generalized and like there's more and more signal that you can in fact build a general purpose robotics foundation model and and you can expose it to some previously unseen scenario and it knows how to navigate the real world uh in a way that you know generalizes the same way that that language models started doing a few years ago.

I think this question around training data is a really really important one and a really key one and and um there are very strong opinions on both sides.

There there are robotics experts who swear you can use simulation and synthetic data to massively scale up the training data set and use that to kind of erupt these scaling laws and there are other folks who believe that you know there's really no substitute for real world data. Yeah.

And you can you can supplement it a little bit with simulation, but like the sim toreal gap is still a very real like there's a meaningful gap there. And so you can't get that much juice out of just simulation. I think today I fall more on the side of the importance of real world data.

Like I think there's just so much nuance about the real world that uh that isn't adequately fully captured in Unreal Engine or whatever like sophisticated simulation engine you can build.

Um, but I do think as time goes on like that sim to real gap will probably close and I would imagine it ends up looking not dissimilar from the way the autonomous vehicle industry I was about to ask where um like real world data is essential right and everyone remembers seeing Google's cars driving around for you know over a decade collecting data um but no major autonomous vehicle program today is not deeply based on simulation as well and so I think some mix ends up being necessary I think In robotics today, it's still you still have to be heavily weighted toward real world data, but hopefully just for the sake of leverage and technology advancement.

I think simulation will end up getting better and better.

Is there an analogy between uh the pre-training to test time inference regimes in LLM training and chat models to what's happening in uh autonomous vehicles and the difference between pre-training on all the data and on policy training that you can do and then also add simulation on top of that.

is that I is there is there an evolution of the of the kind of like mix of training paradigms that you would use in in AV? And I guess like the bigger question is just like Whimo versus Tesla. Is there a meaningful data gap between those two companies?

Because Tesla's obviously been tracking tons and tons of data, but Whimo seems to be working pretty well when you get in the back of one.

And so it seems like both companies might just have enough data and it and we might wind up in a situation where you know like no one's talking about a data gap between Gemini, ChatgBT, Anthropic, Llama, they all just have all the data. Yep. Yeah. Totally. Yeah.

It's it's interesting the way that autonomous vehicle AI stacks work today is like they're because they were kind of crafted and developed in the pre-generative AI era. They're not like these massive pre-trained foundation models where they just fed all of the data in the world.

Like they are much more handcrafted, but it is an interesting thought experiment and there are like more younger next generation autonomous vehicle companies that are taking this approach like let's build a foundation model for driving um and you know and can we do something you know similar to what companies like physical intelligence are trying to do for general purpose robotics.

Can we just apply a model like that to autonomous vehicles? How would a startup like that get data? That that feels like the hardest thing uh because you can't just crawl the web, right? Exactly. Yeah. Yeah. Yeah. It's it's Yeah. It's much harder to get training data for it.

To your question on Google versus Tesla or Whimo versus Tesla, I think this is like a fascinating age-old question. And you're right that Tesla has way more training data in the sense that it has this fleet of personally owned vehicles driving around.

Uh the quality of the data is is lower than Whimos though for the for the sure specific concrete reason that they don't have LAR as a as a sensor modality.

And so this is like the big debate that before I got into BC I worked in autonomous vehicles and like even back then this was a debate like can you get to level four autonomy without lidar and Tesla needs to believe that the answer is yes because their business model selling cars to consumers and lidar is so expensive that like it would completely wreck the unit economics of selling a car to an individual.

Whimo can can can stomach the cost of a ladder because their model is a robo taxi model. Um I honestly I think the jury is still out like it's it's Tesla folks will tell you that like we're very close to that and LAR was never necessary but it's I think it's not it's not completely clear yet.

Um and so in that sense like the the data set that Whimo has is more robust just because it's much more multimodal. Do you have an do you have a sense for why LAR is so expensive? Like I mean Elon Musk was able to make rockets cheap. Like the iPhone is cheap.

like how can we not get lidar down like even just to like a couple thousand dollars like that wouldn't break the Tesla paradigm right but I assume that when we're talking about a LAR package on a Whimo we're talking like five or six figures and that breaks the model but we historically humans have been really good at like mass manufacturing expensive stuff and make it cheap so uh I've always wondered about you know Elon for a decade has been LAR is doomed we don't need it we don't need it but he also changes his mind and so I wouldn't be surprised if one day he's just like, "Yeah, we figured out how to do it.

We have LAR in the cars and like, you know, too bad. " Yeah. Yeah. Yeah. Yeah. And there there have been rumors of Tesla like experimenting with LAR here and obviously under wraps. So like you're right. I wouldn't it wouldn't shock me. I think it's a great question.

I think the short answer is just like it's a complicated sensor. There's a bunch of lasers. They're moving around and so forth.

But but to your point, uh the cost curve has been coming down on LAR and we'll continue to so like the original like kind of like bucket shaped LAR that Valadine made were like $64,000 a pop on like the really old school Google self-driving vehicles. Now they're probably like a few thousand each.

So you need you need several of them on a car. So it does add up. But I think you're totally right. And like there are like research prototypes of LAR that are solid state, meaning they don't have pieces that move and that makes them a lot cheaper.

Um, and people talk about like $250 per unit light, which again are not like production ready yet, but I think you're certainly right that like the way this debate could be resolved is like it may just all converge because in a few years lighter gets cheap enough that like Tesla can use it and everyone can use it.

I I had a buddy uh friend of the show recently. I'll give you some context. He said Whimo will change the world. Walk down the street $1 million in metal sitting unused 20 hours a day on every street in America. Garages will be useless, turning into storage or ADUs. Parking lots, the same thing.

Street parking will be more lanes. Uh, driving offense revenue, DMV revenue, parking tickets all go to zero. Gas stations will be worthless. Sounds insane, right? But Uber went from nothing to everywhere in a decade. Whimo might take 15 years, but the disruption will be nuts. Um, how do you think about the downstream?

you know, assuming that you believe some of that is is real and and I think Sean makes some great points. Um, how do you think about the investment opportunities that are downstream from ubiquitous autonomous driving and like is it even are there going to be v as many venture opportunities or is AI car washing startup?

Pull the autonomous vehicle in, it washes itself. It's great. That would be different than a regular drive through. No, no. Roll up. We're doing a roll up for humanoid. Humanoid. Yeah. Yeah.

So, I just to me this seems like a lot of opportunities on the real estate side is like, hey, can we turn this into more housing or, you know, whatever. But parking lots aren't cheap though.

But yeah, this is this is the context here is that Whimo monthly rides were sort of ticking up gradually and then just shot up to 708. And I guess they're now doing more rides and lift in uh the Bay Area as well. It's crazy. Yeah, it is.

It's been amazing to see how quickly Whimo has gone from like a novelty to just a a piece of the fabric of life for people in San Francisco. And like most people that I know in in the Bay Are in San Francisco use Whimo more often than they use Uber and Lyft.

And it's like quickly spreading to to beyond just being a Bay Area phenomenon. Like they're now live in LA, as you guys know, and in Phoenix and in Austin and so forth.

Uh and yeah, so I think it is um it's remarkable to see it after all these years finally become like a real business that's scaling quickly and and I really like the excerpt from your from your friend. I mean I think I I very much agree with that line of thinking.

And one of the things that initially attracted me to the world of autonomous vehicles back like a decade ago was this fact that it's really fascinating technology. You know, it's difficult technology.

Being able to get a car to drive itself, but it also once you solve it and you start scaling it, it has so many broader implications, second and third order impacts on so much of society and the economy. Like so much of modern life is built around roads and cars and the way cities are designed and so forth.

So, I do think it will have dramatic uh implications on civilization, bigger picture. it will play out over a longer period of time because we're talking about the built world and it takes time to adjust. But yeah, I think especially in cities, at least to start, especially in urban areas.

Um, it's some crazy stat like a third of real estate in the average American city is devoted to parking and so much of that can go away and it's like such an inefficient use of space. So, you can imagine cities being totally redesigned around humans rather than around cars, more pedestrian areas and so forth.

Um, you can also imagine there's a a lot of people talk about the rise of exerbs, like being able to live further and further outside of of an urban setting because when you're commuting, you don't have to be driving like you can imagine the entire form factor of a car changes and you know, maybe you have a desk.

Yeah, I want it to be a desk and a couch and I just want to Yeah. I mean, I've spent a bunch of time thinking about this because I kind of have a gnarly commute right now. And uh it will just become so much better, you know, within when I can get Jord's in Malibu. I'm in Pasadena. You've spent time in both.

We'll get you back here eventually. Yep. Yeah. No. Yeah. I'm excited for LA for Whimo to expand the geoence in in LA more and more. Yeah. Yeah. Yeah. Uh well, this is f this is fantastic. We'd love to have you back. This is a lot of fun.

We could talk about 25 other topics in AI since it is the most fascinating in industry right now. Uh we didn't even get a chance to talk about deals and stuff, but I'm sure we'll I'm sure we'll go into all that in the next time you're on. Yeah, that sounds great. Thanks for having me, guys. Thanks so much.

We'll talk soon. Bye. Next up, we got to sing. We got to hit gongs. We got lots to do. We're going to say, are we Is he going to sing with us? It's always hard to sing with a remote guest, but maybe we should sing beforehand. Find your happy place. Find your happy place.

Book a wander with inspiring views, hotel grade amenities, dreamy beds, top tier cleaning, 247 concier service. It's a vacation home but better folks. And we have the founder of Wander in the studio coming in to announce a massive round of funding. Welcome to the show. How are you doing? Get out that wide.

Hit that gong. Hit that gong. Breaking the gong. Hit that gong. Welcome to the studio, John. It's great to have you here. It's It's great to be here. I was hoping that I'd be able to sing with you guys. So, I'm a little I'm a little disappointed that I didn't It's really hard with the delay.

We got to figure this out somehow. Some latency mitigation or something. We also need to make it a full song because oftent times I get to place choruses and verses. We need to integrate the whole pitch into one big song. But congratulations. Break it down for us.

Ideally, ideally when somebody walks into a wander for the first time, they're sort of serenated by us and we can sort of, you know, two, three minutes on the whole home speaker system. So, we'll work on that. We can make that happen for sure. That'd be great. Pull that off. But yeah, give us the business update.

Break down the fundraising round. What are you actually spending it on? Because I know it's an asset light model, but uh explain how the round came together, the progress of the business. Totally. Uh, so it's a $50 million series B led by QED, Fifth Wall