Roblox VP of AI on 25M concurrent users, 3D generation models, and safety at scale

These are alternative assets, right? Yeah, these are great. Uh, next up, we have Anu Pom from Roblox in the studio. Uh, I've been fascinated by this company for years. Uh, it's a company that I don't have a ton of experience with personally, but it's obviously a behemoth with a generational founder at the helm.

Very interested to get to know you. Uh, how are you doing today? We're good. Are you happy Friday? Happy Friday. Uh, sad Friday for us. We love what we do and so the weekend is just a tough time where we're we're not on counting the hours until counting the hours. Yeah. But, uh, but it's great to be here. Yeah.

Would you mind uh kicking us off with a little introduction on yourself and what you do for the company and how you wound up there? Yeah. So, first introduction about myself. Uh, have done a couple of big data startups uh before.

So have been a founder and I've been involved in at least at least scaling a couple of companies uh and then uh uh I was retired for a year and uh Roblox came along. It was such a fascinating uh business.

It was such a fascinating technology stack that I came in and now my responsibilities are AI used to be called machine learning at that time uh um and uh uh infrastructure um so that's that's generally my remmit in the company. Yeah. Take me on a little tour of how artificial intelligence is being used right now.

Uh we're we're hearing stories of just accelerating software development, new products being unlocked, new actual features. um how would you characterize the various ways AI is transforming the company right now? Yeah, so sort of three arcs. The first one is it's been around for a long while for companies like ours.

So our homepage for many many years has been generated by machine learning. What you all call recommendations whether you like them or not is different but the recommendations are machine learning generated. So if I watch a TVPN uh YouTube podcast, maybe I'll get I'll get other TVPN podcast.

collaborative filtering, recommendation algorithms going back to like you know Amazon you might also enjoy classic machine learning that's it right and so classic machine learning would be everything that we do that we've done for many many years uh in that area people are starting to think about using large models in production interesting uh which is very difficult uh because it's super expensive to invoke a large model and you wouldn't like your YouTube homepage or Roblox homepage or Netflix homepage Amazon homeage any of these homepages loading slowly would be really really tough.

So the entire industry is trying to figure out how to get these large models working in in traditional uh uh machine learning. Then very unique to us is the amount of AI we use in safety. Mhm. So everything that you type in the chat for Roblox goes through a filter. Yep. Right.

uh and that we we we developed technology with machine learning but now we are developing technology with AI which means we are deploying a very large model which has a huge memory of your chats y so that we can detect bad behavior on that makes a ton of sense yeah that's huge uh talk to me about really quickly we'll we'll we'll continue on the I think there's probably a third option but um just inferencing these large models at cost is open source important are you looking forward to like getting some of these LLMs that are good enough or at the frontier like onto AS6 just dropping the cost.

We see these we see these news news items every week. Oh, this cost is 80% cheaper. This is 90% cheaper, but it still feels like it's probably expensive.

Have you just been bearing the cost and you're waiting for it to come down further or have you developed tricks to actually reduce the cost to uh something that looks more like a traditional, you know, database query cost which is basically negligible? Yeah. Yeah.

So the so the latter very very good question by the way this is this is something that is top of mind for us because if you're going to uh you know invoke that model 250,000 times a second if every one of them was 10 cents you would be in trouble you wouldn't have a business right so uh then what what we focus on is uh creating smaller models from large models so we still start with the large model but we then create smaller and smaller models through distilling and quantization.

Both of these techniques are well understood. Yep. Yeah. Yeah.

I mean, we talked to a company that's doing this for other companies coming up with it's just a profanity filtering LLM and it's and it's trained on a consumer graphics card and it's so small it's really baked down and and this makes a ton of sense for a large enterprise that has very high performance computing needs at scale.

talk about the how you see the potential of AI at the game experience level for players. How much time you guys are thinking about that. I can imagine you know players leveraging it to create worlds, characters, storylines, narratives. Yeah. All that kind of thing.

So you teed me up for the third point which which I was going to which is uh you know generative AI classic generative AI is text image video and kind of stops after that. So we decided around one and a half years ago to pick up from that so we don't try to create our own large model which is trained on text.

We've created our own large model for 3D generation.

M so uh today if you want to build a video game or if you want to build a game you are going to open uh some software you're going to blend all the all the stuff together you might have to you know work with your artist for many many hours but imagine if you could just type a prompt a stylized Japanese village pagora and you want to make it gothic for whatever reason um we have a large model right now uh open sourced uh people can go to hugging face and and and use it uh and and uh build an entire scene using 3D generation model uh all of it is possible because obviously all the advances that have happened in AI interesting yeah I imagine the traditional workflows like Cinema 4D or Blender or Unreal Engine and then you're you know bringing that in to do the shading and all of that stuff is like super teed up for uh for image generation makes a lot of sense.

I'm also interested in in in uh when do you think we will see uh the advancing the advancements in AI play into actual NPC creation.

So I I design a game in Roblox and I want to uh instantiate a character a boss and I'm don't just want to give them a star algorithm to or or oh yes the boss like you know attacks three times with the sword and then cast the fireball and it's the same you know when you play those games and you're just like this doesn't this feels too algorithmic like I understand the algorithm that's running there.

clearly uh you know um deterministic code that I'm interacting with and it feels like oh it's a bullet it's a bullet sponge boss instead of like oh wow like it feels like I'm playing against a human. They're being creative.

Maybe they're not maybe they're not just impossible to defeat because they're superhuman AI but they're just creative in a way that the default AI algorithms are have been in the past whether you're fighting like you know a boss in Doom or something.

uh is there any are there any uh resources that you're going to provide to the community on that front? Yeah. Yeah. So that that's something that we debated a lot. One is that we as a platform become very opinionated about what an NPC is and you know this is how the goss NPC is.

But that's not our philosophy as a company. As a company, we are a platform that deploys a series of APIs and then we just wait for people to come up with things like grow a garden or dress to impress. Yes. Uh trust me, you know, I don't know about y'all, but I couldn't have thought about dress to impress.

It is such a simple game and it appeals to such a very human thing. You dress to impress, right? And so, I mean, we we do we play our own version of that here on the show as you can see. tailoring suits. All of us are, you know, even my teenager who wears a toned shirt is actually trying to convey something.

So, going back to your NPC question, uh we release large models as a service. So, we don't want our uh creators to worry about what you all asked, right? The the cost of inference, which model should I use, all those details we take over as an infrastructure team. So, infrastructure is invisible to them.

And now many creators are using that to give their players capabilities to build their own NPCs. So it's it's two layered now rather than us building the NPC. It's like you can you know like that old thing you can do it and you can do it right. Can you can you give a an update on the scale of Roblox?

I think there's been headline you know every everybody I think should know by now that that it is a massively scaled platform but even just insights into average user minutes on the platform scale how many America you know what percentage you know I imagine it's doubledigit percentage of American children play can you give us some ground on the scale right now yeah so but I'll give you the scale the latest scale which was I can't believe it's Friday so it was just 5 days ago 25 million concurrent people players showed up.

25 million at the same time at the same time. That is so much. It's like it's more than the Super Bowl, right? It's like significantly I think it's significantly more than the Super Bowl. Like it's like Super Bowl every significantly more than the population of certain nations that I shall not name.

Not quite not quite the Super Bowl. It's 127 million. Okay. Okay. Well, you're getting there. But concurrent is the you know concurrent is specific what concurrent is is essentially on a random time.

But then our our founder Dave always reminds us that yes today of course there's a lot of news around these all-time highs. But for the last 16 years every other weekend we've hit a new high. Hit a new high.

It just happens to be that this weekend we just hit a Guinness, you know, the world record of a single game having 16 million people playing simultaneously. Uh that's never happened before. I didn't know that. I'm an infrastructure person. Uh uh so uh we've been working on this for a while now.

Um last December we thought we saw one of the biggest highs with dress to impress and then we saw fish then we saw dead rails. It's every two or 3 weeks uh we see a massive massive high. Yeah. Uh tell me the story of grow a garden specifically. Um do this come out of nowhere?

Does do these things surprise you at this point or are you able to predict and track these things? Yeah. How early can you tell if a new game is is actually a hit? Is there initial data that you can see from the first 10,000 players or anything like that that says, you know, this thing's going to go into the millions?

The third the third Google result for Grow a garden, by the way, is a Reddit post on r/Roblocks from a month ago. Why is Grow a Garden so popular? So, even even the Reddit community can't figure it out, but what do you know about it? Yeah. Yeah. So, so um I'll take you back a little bit.

Uh you know, two or three years ago, we used to do annual planning. Mhm.

which tries to answer your question which is can I predict how many players will be playing how much at what time and then we realize it's it's sort of as an as an engineer it's a little bit of a fool's errand because you're almost capping the popularity of your platform by saying I'm going to predict some 10% growth 20% growth u so what we do is we do long range planning but we also do short range planning which means as you as you talked about if if I see 10,000 people really engaging with it.

Uh uh the recommendation algorithm picks it up and starts giving it more traffic. Mhm. Okay. In the meanwhile, we are tracking that the game the game is going up.

But on the capacity side, uh uh we don't try to predict exactly how much a game is going to be popular because we're a platform with literally millions of experiences. So we wouldn't have been able to tell you that grow a garden is going to be at this.

But we could have told you that as a platform we continue to grow a at a very high rate. Mhm. So we plan for the platform and then if one of the games becomes popular it's okay. Every four weeks we have had a new popular game. Uh uh um so uh so we just we just track the total growth.

Uh we don't try to be good enough to track each game. Yeah.

Do you is do you find it uh I can imagine if you're a S sur you want to go this is like the the the NBA of site you know reliability engineering because there's almost there's few there's few places where you can get this this amount of pressure this scale this much unpredictability who's LeBron James of uh of site reliability engineering we want to ring the shout out they immediately get poached Ah, so here's the here's the interesting thing.

Thank you for bringing that up. Very few people would would would bring up the the the site reliability. The heroes the heroes of the unsung heroes. The unsung heroes of the internet. Well, we would be nothing without they keep the internet running. 100% agree. But here's what we we have done culturally.

Number one, every executive is on an on call rotation with SR. Oh, interesting. That's a crazy cultural move. I love it. I'm on call I think from today 6:00 p. m. or something. Uh uh and I report to the CEO and the SVP of engineering here. So, you know, yeah, for some of my industry friends, it's just odd.

I thought you kind of work up towards not being on call, but the way you institute a reliability culture, Yeah. the way you institute a reliability culture is uh uh uh everybody takes some on call. We're not always on call, but we always take a rotation. So, that's number one.

The other is our S sur team is really a software mindset. Uh so so every Tuesday they run this thing called taco Tuesdays. Now you would think that they eating tacos, right? Well, what they're really doing is testing the actual capacity.

Every Tuesday they try to bring Roblox down by taking capacity from a particular service. Interesting. Okay. Yeah. So that is a nibble nibbling a little bit of a little bit of the server little bit of the taco testing.

You take the voting you take the discovery service and you starve it of capacity as if we have too many players but we are all in the office so we can watch it. Right. The other one that they do is uh um uh is chaos testing. We have a very unique name but our coms team doesn't like to talk about that name.

So, I'll not say the name on this on this talk, but but let's say there's Roblox and there's Godzilla and if Godzilla was in our data centers, I'm still not naming the service, right? So, it creates so much chaos in our system, including like unplugging servers. Yeah. Now, most people don't know this about Roblox.

We have 24 data centers globally. Wow. Uh uh we look like gaming company from the outside, but really to your point about being the Super Bowl of site reliability. Yes, I agree because we also have servers to rack and stack.

Uh so we do these taco things and these chaos testing throughout the week trying to keep bringing down uh uh uh Roblox as if we want to see what happens and that's why we get ready for big Saturdays. It's amazing. Um do you have anything else? I want to let you get back. Every day is the Super Bowl.

Every Saturday Super Bowl on Roblox, but this was fantastic. Thank you so much for stopping by. You said it. Thank you very much. This was great. Yeah. Thanks for joining. Yeah, we'll talk to you soon. Uh, how'd you sleep last night? Did you put up good numbers?

I'm not going to I'm not going to I'm not going to lie to you, John. I got a little bit cocky about a month ago. Yeah, even just earlier this week. Completely falling off. I got a 75. Only put up 7 hours for 96. Let's hear it for me. I want a soundboard. Play the Ashton Hall. Yes. 96.

that I'm going to go home this weekend, reset, and get back because you're putting me putting me to shame. Yeah, I I'm I'm I'm beating you up out here. Um anyway, let's run through some news before our next guest gets here in 12 minutes. Um and also go over to eight. com, get a Pod Five.

They have a 5year warranty, a 30-day risk-free trial, and free returns, free shipping. Um the US prepares action targeting allies chip plants in China. This is from the Wall Street Journal. The move isn't meant to escalate trade tensions.

White House officials say it sounds like the number one thing you would do if you were trying to escalate trade tensions. But uh let's see. Beautiful image of TSMC here. Can you see this, Jordan? This factory looks this looks amazing. I don't know if this is the factory.

It might just be like the headquarters, but wow, what an amazing building. This like building. They were like, make us a Pinterest board of the most sci-fi building of the future and then let's just build that. This is giving Apple a run for its money with a donut. This is really, really cool.

I had never seen this picture before. Yeah. The cannon. The capital cannon. Now you you see why MASA saw this. It was like $1 trillion to TSMC. Let's get it. Yeah. Let's get it going. Yeah. I mean, right.

Uh Jensen earlier this year after all the trade war stuff, started setting up new facilities for Nvidia, believe in in Shanghai. Yeah. Yeah. So, I'm sure he's sitting there needs a heater after seeing this headline going back and forth.

Um, a US official wants to revoke waiverss that allow global chip makers to access American technology in China. People familiar with the matter said this has been something that people have been talking about with the chips act for a long time.

Just the idea that yes, ASML makes lithography machines that are used by TSMC. ASML is not is a European company, not an American company, but they use a lot of American intellectual property.

And so the the American government feels that they have the ability to restrict trade of those lithography machines, which is obviously a key point of leverage in the AI race. Now, there's differing opinions as to how real the AI race is or or how seriously should be taken. There's folks who are saying certainly a race.

I think the debate is is it a war? Yes. Yes. Is it Yes. Is it a or is it a dunk contest? Who knows? You know, Deep Seek shows up. Windmill. Oh, reasoning model for free, breaks the backboard. You love to see it.

Uh, the move could strain relations with South Korea and Taiwan whose companies would be most affected by the change. The revocation could disrupt the global industry as companies deal with trade war issues such as China's limit on rare earth magnets.

And so, you know, you you hit them, they're going to hit back at some point.

So um currently South Korea's Samsung and SKH Highix very important company that makes memory and as well as TSMC enjoy blanket waivers that allow them to ship American chipmaking equipment to their factories in China without applying for separate for a separate license each time.

So your TSMC or your SKH you want to do packaging in China because that's where the industrial hub is. That's where the special economic zone is. You got to ship in a whole bunch of different equipment. Maybe it's just, you know, a CNC machine or something or whatever from America. Yeah.

Well, that's going to throw off everything you're doing if you have to go to America and ask for permission every single time.

So, Jeffrey Kesler, the head of the Commerce Department unit in charge of export controls, told three top companies this week that he wanted to cancel those waiverss, according to people familiar with the matters.

They said Kesler described the action as part of the Trump administration's crackdown on critical US technology going to China. If carried out, the move could be both disruptive diplomatically and economically. Earlier this month, the US and China agreed to a fragile trade truce in London.

The part of the deal involved each country agreeing to hold off from introducing new export controls and other measures designed to hurt the other. And so there's it feels like we're we're leaning war here, trade war more than trade race or AI race. Um but we will see.

This action isn't a new trade escalation, but we would be designed to make the licensing system for chip equipment similar to what China has in place for rare earth materials.

The White House officials said the US and China continue to make progress on completing the agreement they reached in London and negotiating on trade. Chipmakers will still be able to operate in China.

the new enforcement mechanisms on chip mirror l chips mirror licensing requirements that apply to other semiconductor companies that export to China and ensure the United States has an equal and reciprocal process. So that's the story from the journal. Uh this is a fun story. We're getting AI pancakes.

Applebee's and IHOP plan to introduce artificial intelligence in restaurants. People have been begging for this. Yes, people have been begging for this. Uh it's a very funny story, but the silence from Applebees on on the AI war has been definite. Yes.

So, uh this is all about dying, a company called Dine Brands, and I saw this and I thought it was interesting because AI pancakes is funny. Um but, uh this is a $450 million company or $425 million company. They're public on the New York Stock Exchange, Dine Brands Global.

They're partnering with uh the they're the parent company of the two chains. They aim to streamline operations and encourage repeat diners. And guess where they're headquartered? Pasadena, California. We got to get the CEO in the studio. This is amazing.

Company behind Applebees and IHOP plans to use artificial intelligence in its restaurants and behind the scenes to streamline operations and encourage repeat customers. Dine Brands is adding AI infused tech support for all of its franchises. They're trading at about 0. 5x revenue.

And so if you slap some AI in there, they could trade at 300 times revenue. make it happen. I'm I'm excited. I don't care that they're a restaurant. I'm rooting for them. It's passing a homegrown company. Let's go.

Uh so, Dine Brands adding AI infused tech support for all of its franchises as well as an AI powered personalization engine that helps restaurants offer customized deals to diners. So this is this is more just like retention and uh typical marketing stuff.

But I mean you could imagine that a lot of these AI tools will would as silly as it sounds it would benefit a restaurant chain because there's a lot of logistics and operations and marketing that needs to be done and why not have AI play a role in that. The question is always like where does it sit?

Is should this be a uh a SAS product that you're buying from a startup that's leveraging AI? Should you go should your current marketing partner offer this as just an add-on or improvement to what they're already doing? Should this even just be your your marketing team now has a Chad GBT license?

Um, it sounds like they're going a little bit deeper, though. So, the Cal the Pasadena, Californiabased company, let's hear from Pasadena, which also owns FA Fuzzy's Taco Shop and and has over 3,500 restaurants across its brand, is taking a practical approach to AI by focusing on areas that can drive sales.

Um the chief information officer Justin Skelton is commenting to the journal in this streamlining tech support for dine brands more than 300 franchises is important because issues like a broken printer take valuable time away from actually managing restaurants.

So it's not for the customers it's for the franchises they call in. This is the this is the classic question of like why at McDonald's why is the McFlurry machine always broken?

Well, now if you're a franchisee owner and you have something that's similar to a to a McFlurry machine, a printer, receipt printer or PL as these chains roll out AI across different customer touch points and in the back office, you can imagine that that waiters like like worldass waiters will actually end up being like even more of a luxury and people actually seeking them out, right?

Yeah, it is a truly great waiter and service team at a restaurant can massively elevate the experience, right?

It's like it's and and so I think this is one of those things AI will take some jobs in the restaurant industry but also elevate roles, you know, like like uh you know, sitting down at your favorite restaurant, seeing a waiter that you see once or you know, twice, three, four times a month, right?

And and having that experience, that relationship. So, the IHOP in past is zoo on Sunday mornings. Yeah, it's just dad. I've been there. I've been there with my son once and it was it was fantastic. Uh yeah, it's it's all the dads taking the kids out, letting mom sleep in on the weekend. That's the meme.

And it's just like line out the door. I I had no idea it would be so popular, but it's extremely popular. Uh so they're building tools on top of Amazon, uh which has uh the Q generative AI assistant. It allows the company's field technology services staff to query its knowledge base for tech help using plain English.

So, you know, chatbot, but but trained on specific uh questions that the restaurant tours or the franchises have to answer. And then they're also rolling out a personalization engine. Um, pretty fun that they got a nice Wall Street Journal write up about this.

Uh, other tech initiatives Dine Brands, Dine Brands is testing include AI powered cameras that can detect when a table needs to be cleared. So you will not need to wait for your waiter to come around. They'll have a dashboard. See, hey that you clean up on aisle six.

Uh as well as an AI app for restaurant managers that helps them oversee day-to-day operations like staffing kind of just better timekeeping, better management, better just tooling for these companies. Uh very interesting.

Uh with these updates, D Brands joins McDonald's and Yum Brands Pizza Hut and Taco Bell which are similarly tapping AI in their restaurants and behind the scenes.

One promise of AI is that it can make fast food and casual dining restaurants more automated, lessening the need for human labor and speeding up the work of existing staff. The ability to offset labor costs, whether by reducing or more accurately forecasting it is a major cost savings for restaurants.

Unclear what that would mean for a price war. If both companies implement AI, they're both able to reduce labor costs. Would they both just drop prices or would they actually both reap higher margins? I actually don't know how. Or just spend more money on marketing. Maybe. Yeah. Yeah.

May maybe we just see more McDonald's ads in the Super Bowl and I have ads in the Super Bowl hopefully. Anyway, AI is moving quickly and I believe it's going to be embedded in everything we do, says Skeleton. Fantastic. Uh we at least give him a 1x revenue multiple. I've seen enough. I've seen enough.

Um we have our next guest in the studio. Great. Let's really quickly. Let's tell you about Wander. Find your happy place. Find your happy place. Find your happy place. Seriously, it's summer. Get on. com. Now is the time. If you've been waiting around, get on there. They got inspiring views.

They got hotel grade amenities. They got dreamy beds. They got top tier cleaning. They got 247 concier service. It's a vacation home, but better, folks. Um uh there's also the Tik Tok ban, which has been delayed for the third time.

Uh, the president gave the Chinese controlled video app another reprieve from a 2024 law requiring its sale or closure. Another 90 days. Just 90 days. Trump is kind of the king of these like, you know, stalling it out. This has been like his leverage.

He's known for like moving extremely quickly and just being like all of a sudden everything has a massive tariff and then also like really dragging his feet on things. Art of the deal. Art of the deal. I'm banning Tik Tok. Art of the deal. I'm delaying the ban. Art of the deal. Art of the deal. I'm banning 40 chest.

chest. Um, he issued another 90-day reprieve. White House spokeswoman said the administration will work over the next three months to close a deal ensuring American user data on Tik Tok is safe and secure.

Uh, prior Supreme Court ruling upheld the law, citing national security concerns regarding Tik Tok's data collection in foreign ties. What's interesting about this is that the the narrative of like of like why is Trump soft on Tik Tok during the election cycle was completely different reasons.

It was like there was there was someone who had Sesuana, what is his name? Jeff Jeff Hass or something, I forget. Uh yes. Uh yeah. So this investor at Saskuana had a huge position in Bite Dance which owns Tik Tok. And so a Tik Tok band might like personally impact him to the tune of like $5 billion or something.

So, so obviously it was worth it to him to invest like hundreds of millions in lobbying to prevent that from happening.

So, there was kind of this narrative emerging that like Jeff Yas was was like the driving force economically in America just a natural incentive not like a CCP agent just like just like he has a financial incentive to to not let his asset get burnt by by a ban. And so do the investors in DJI.

Yeah, there are few American venture investors American venture investors uh in G in DJI who would similarly be against a ban, right? Uh just if you were just tracing like basic economic logic.

Now there was also the narrative that Trump wanted to uh keep Tik Tok alive because it was very good for his campaign because there were a lot of viral Tik Toks promoting Trump but or or just very bad for the campaign because how many young people would be like you banned my favorite app. Exactly. They took my Tik Tok.

I'm voting against this guy. My precious. All of those I all of those reasons are out the door because he's elected. So I don't I still don't understand the logic here. Um and I hope that something gets done here. Anyway, we we will continue to track the story.

Uh at the end of this 90 days, start the start your engine, start putting a 90-day hold on the calendar. We'll be checking in and obviously check Poly Market to see the updates on. Right now, Tik Tok sale announced in 2025 is sitting at a 28% on Poly Market by the end of the year. By the end of the year.