OpenAI intern connects Astra to a $200 robot arm to paint, signaling a coming DIY robotics boom

Speaker 2: I've got some bank resets

Speaker 1: Yeah.

Speaker 2: That I'll put to work.

Speaker 1: Yeah. Anyway, we have our first guest. I don't wanna mispronounce. Tyler, how do we pronounce his name? Thijs. Thijs? Let's bring him in. How do you pronounce your first name?

Speaker 6: It's Thijs. Thijs.

Speaker 1: Thijs. Nice. That's right.

Speaker 6: Like nice.

Speaker 1: Yeah. Well, welcome to the show. Thank you so much for taking

Speaker 2: the

Speaker 1: time.

Speaker 2: Great to have you.

Speaker 1: Tell us about your your role in robotics and also some of your recent work, hooking these models up to robotic tools and and infrastructure. It feels like we're going to be entering a boom of like people they ordered the Mac Mini recently. People are gonna be ordering three d printers and robotic arms and doing hack projects. Super excited for like the DIY world to explode over the next couple months. But let's start with just like your most interesting projects recently.

Speaker 6: Yeah. I'm incredibly excited for that. And and that world's definitely, like, just around the corner. So I'm I'm super excited about that. Yeah. I work here on the robotics team at OpenAI. I'm an intern and have been doing lots of exploratory projects. And this is just one of the projects I I I basically noticed that our models are really good at painting and doing various, like, computer use tasks in software like you were just talking about and learned a lot that you could actually basically connect these into physical robots in the real world and wanted to see how they would do that sort of drawing and tasks like that as well.

Speaker 2: Yeah. If you can if you can paint in Google Calendar with calendar invites,

Speaker 1: it

Speaker 4: would

Speaker 2: probably translate that to to the real world.

Speaker 1: Yeah. It's really like everyone is doing everything. Like, I've seen people paint the Mona Lisa in Microsoft Excel or Google Sheets, then you take take it back and you can do Excel and MS Paint now if you want with computers. And it's like everything becomes everything. What what what is actually the the important, like, precursors to a good experience? We were talking to a a YC founder who built a humanoid robot with some basic claws for, like, under $2. And it feels like there's there's an importance of some on device, you know, API or some on device models in some case doing some slam on device. But then there's also just tools that give you a very primitive interface that might be kinda clunky, but it doesn't really matter because you can just vibe code your correct interface. But what what what have you liked? What are you excited to to put in the repertoire of tools?

Speaker 6: Like, within robotics?

Speaker 1: Yeah. Within robotics.

Speaker 6: Yeah. I think that, of course, like, lot of these demos and stuff is very early. I think, like, the whole space and exact modeling and methods are still definitely being figured out. But I did really like the fact that you're sort of able to hook in the intelligence of, like, what the models are able to do, as you can see with all that painting stuff, and plug it into something physical in the real world and get it to do things. There's probably gonna need to be, like, some combo right now, like, running basically, my experiment here is is plugged right into Codex, and that's, of course, probably quite expensive. Yep. Definitely pretty slow. I think these paintings took between one hour and, like, two hours depending on the methods exactly that that model, like, Toast to use here. But, yeah, there's definitely a lot of improvements to make, but it's quite cool to see what you're able to do already.

Speaker 1: Yeah. How are you balancing trade offs between speed and reasoning effort? I was I was I've been testing computer use on a bunch of video games that I just mentioned and there's like the, you know, like part of what the feel the AGI moment is is when the cursor's moving at like at least near human speed And but then also you don't want to be making a bunch of mistakes and just so that's this, like, balancing act is really key. Did you try multiple reasoning effort levels across different paintings and see, like, noticeable results? Can you see qualitative differences in speed and quality based on models that you pick?

Speaker 6: Yeah. There's, like, a lot of different ways to sort of go about doing this. Yeah. And I initially started out these experiments by just telling the model, like, here I I as you can see, I actually have the bot behind me. Okay. Over here, it's currently painting a TBPN logo.

Speaker 5: Oh, cool. Amazing. I

Speaker 7: love it.

Speaker 6: But it has it's it's doing its best.

Speaker 2: There you go.

Speaker 6: Good start. We got basically this camera up here

Speaker 1: Yeah.

Speaker 6: That the model, which is connected right into the codecs over here, is connected to. Yeah. And the biggest challenge is that images are quite large Yeah. To process, especially, like, for the model, there there are a lot of tokens. Yeah. And the loop of, like, taking an image, doing taking an action, taking another image, taking an action, that's the thing that has taken a lot of time. So what I ended up doing like, it's not directly doing, like, image, like, one small movement. Image, one small movement. And I think, like, that would probably result in the best performance for,

Speaker 2: like Yeah.

Speaker 6: General robotic tasks and stuff. But, like, at the moment, it's basically taking an image, writing a plan in code of where, like, what it should do and where it should go next after it has done, like, a lot of the calibrations, maybe a slower sort of methods at first. Sure. And then after it has that plan, it executes it and then monitors it in the background, like, taking images every second or few seconds and watching, like, to make sure things are going well and, like, making it small adjustments to the plan throughout. And I I found that has been, like, a a good balance of speed here.

Speaker 1: Have you been thinking about, like, compression on the input? You mentioned that the images are lots of tokens. I I I have this monitor that renders at like five k resolution and I was like Mhmm. This is probably gonna be really slow for computer use. I should like run this game in a window and give it like a seven twenty p input because that's probably enough information. But I'm wondering like how important the the the resolution to speed and quality is that you've seen.

Speaker 6: Yeah. I think it's very important. In this case, for, like, a lot of the early demos, I was using, like, five twelve p, just compressing

Speaker 1: Oh, it yeah.

Speaker 6: Just to make, yeah, make sure. Because I think the model is just pretty much it's pretty much solved, like, a lot of the perception challenges here. So it's, like, really good at understanding what's going on even with, like, a lower quality photo here. But in this case, like, with the longer planning and and sort of, like, a plan sort of a minute of action out, it's less important, the, like, specific resolution probably. But, yeah, as we go to more, like, faster, much more smaller loops of, like, control, it's definitely gonna be important. Jordan?

Speaker 2: I I'm super excited about about this project mainly because it feels like we're right on the precipice of of sort of, like what what I what I think is gonna be a big breakthrough is, like, a sort of a deep research moment for robotics, like simple robotics use cases where deep research for so many people was their first time using agents. The idea that you could type out a prompt and then get back what would have been maybe, you know, at least hours of human work. Right? Somebody reading all these different sources and combining that information into a document that's that that has a consistent narrative and and understands the the right information. That was just such a big moment because a lot of people were saying, wow, can't believe that that that the AI was able to do something that would have otherwise I would have hired had to hire somebody to do or just taken a bunch of time. But there's like very simple tasks that I feel like you could probably start working on sooner than later which is like an example, at least for me, would be like if I could just take all the mail I get and dump it in front of a robot like that and have the robot sort it.

Speaker 1: Interesting. Yeah.

Speaker 2: You know, take all the, you know, 50% of everything I get is probably some sort of advertisement. So like, figure out what's an ad and shred that. And then actually, you know, basically like photograph and respond to if I have like a utility bill or or or any any number of things that I actually need to respond to. Theoretically, you could close that sort of like IRL to digital loop where the agent would like actually get a task from the real world and then close that loop online with with just normal computer use. Like that's the kind of thing that you don't need to you don't need like a $50,000 humanoid robot. Sure. You theoretically could have a actual desktop robot that was able to do this thing that otherwise takes me I dread going and like, okay, I have to sort through all this mail and figure out what's important and make sure I don't miss things. But I feel like there's a bunch of other use cases like that where people are like, okay, I didn't just generate a pretty picture or or Yeah. Or answer some question that I had. I like actually

Speaker 1: wanna save iRL spam filter.

Speaker 2: Yeah. An IRL spam filter. The spam Yeah. Spam filter robot. Yeah.

Speaker 6: Yeah. I think there's like two really really cool things about this project which sort of shows that direction that things are going. One is that this arm I don't know if you know about like too much about the prices of, like, classic robotics equipment. It's, like, thousands and thousands of dollars Yeah. At the moment. And, like, this arm that I'm using here, like, right behind me, this is a Hugging Face s o 100 robot, which is, like, open source, fully three d printable. You just need to get the actuators, which are, like, much cheaper. I think it at the moment, which is, I think, more expensive just because of supply chain issues. It's, like, around $200 or so, but, like, that's, like, incredibly cheap for robotic equipment and what you're able to do with it. So, like, I can see a world, like, quite soon where similar to and someone put this really well on Twitter. Similar to how there's this whole three d printing craze where everyone went and bought three d printers and ran software. Like, everyone's gonna buy these sort of cheap before we get, like, really industrial equipment for, like, personal use and personal product. Buy these plastic cheaper robot arms that you can just, like, sort of clip onto a table and just put stuff in front of and plug them into, like, agents that are already you can already do things with that are, like, already out there. Like, Astra is something that people can just pull up codecs and start controlling robots with right now, which is super sick. And we've also been seeing, like, at least on Twitter, I've noticed a lot of, like, sort of academic researchers at at different institutions start to, like, realize the that you can do this with these models and start doing, like, initial explorations and things with it, is super sick to see. And there's so much more Yeah.

Speaker 7: Was space. There was a

Speaker 2: YC company on the show yesterday that has managed to build a a humanoid robot for under $2,000. Oh, my gosh. Yeah. That feels like a price point that people would experiment with. So I if I if I know I can get a robot for like $2 and connect it to Codex and then tell it, like it basically lowers the stakes a lot where I could be like, hey, go every weed like this that you can find in my yard, go like pluck it out and and try to put it in a in a in a bag or whatever. Mhmm. And like that that's basically like the gardening robot. Yeah. And it could like flounder and fail, but I have like pretty high confidence that even right now, Astra would be able to identify like hundreds of like a specific type of of weed and probably and there's of course like safety The Nat Friedman. Safety concerns with that. But yeah. Yeah. The Nat Friedman like leaf robot like that feels like we're we're here. The models It can do it it's not it's not cheap but we're

Speaker 1: What is what is the next medium that you want to explore? Are you gonna get into whittling? Ceramics? I want to see you whittle a bench as a benchmark. Whittle bench.

Speaker 6: That's really good.

Speaker 1: But whittling a spoon or something. I don't know. It just feels like three d is the next thing.

Speaker 2: You really want to give the robot John wants to give the robot a knife. I don't think that's don't think read room, John. Read the room. Okay. Okay.

Speaker 1: Yeah. Maybe we'll stick

Speaker 2: to to Inter OpenAI intern gives robot a pocket knife. We'll do

Speaker 1: some ceramics, maybe get it on the pottery wheel, make a nice vase, something like that. But, mean, is there anything where you're like, oh, yeah. Okay. This is where this goes next.

Speaker 6: I think I was having the exact same thought, which is like, giving a robot a knife is like probably a very bad idea. But I I do wanna like sort of see if it can help do like cooking tasks or do various tasks that are like things that you do in your life that would be really sick if you could have a robot help you out here and there.

Speaker 1: Yeah. Yeah. That's interesting. Yeah. I I I'm so I'm so interested to see the because there's a whole class of tasks where you can't wait a full minute. I was testing on like a a real time strategy game and you can pause the game, but if you're not

Speaker 3: Mhmm.

Speaker 1: Moving at a certain APM, even on easy mode, like you will just get smoked. And so but but it feels like with new chips and Cerebras and Spark models and stuff like the the speed up is gonna come but it's just that's gonna unlock a whole new, you know, whole new host of of capabilities. What what what advice do you have for for young people that wanna get into DIY, you know, like this this type of work? I mean, you mentioned that one Hugging Face device that you have behind you. Are there any other devices or tools, tool kits that you recommend as places to get started?

Speaker 6: Yeah. I think that the Hugging Face robot is, like, an incredible tool. You can also three d print, like, a completely new embodiments and stuff. There's lots of open source projects online where people, like, change it around to get better grippers and things you can do with that. Just, like, also just don't give up after the first attempt. Like this was the very first

Speaker 3: Woah.

Speaker 6: Painting that the Woah. That's upside down even. I couldn't even tell. You can see the Golden Gate Bridge here. Can kind of see the ground it was trying to do. Yeah. And just like if you keep going and and you can you can sort of see the progression as it as it improves.

Speaker 1: Wow. That's amazing. Yeah. You need to frame those next That's to each that that's incredible.

Speaker 6: I'm thinking putting them all all all five of the progression in like a frame and calling it self improvement.

Speaker 2: Because That's Yeah.

Speaker 6: The model, this was this was a thread. The model just was able to

Speaker 3: Wow.

Speaker 6: Figure things out with a few pointers here and there, how to make get better and better at painting.

Speaker 1: Who's who's gonna sign it? Is it you? Do you have Codex sign it? Astra sign it? Who

Speaker 6: what's I've the also given the robot I've given the robot a a pen and it's gonna try. I don't know how well it's gonna do, but we'll see.

Speaker 1: Some axe or something. Well, thank you so much for coming on the show.

Speaker 2: So cool. Very cool. Come back on soon. Yeah. Yeah. Thank you for feeling the physical AGI.

Speaker 1: For sure. For sure.

Speaker 2: Very cool.

Speaker 1: Have a great rest of your day. Goodbye. Cheers. Let me tell you about public.com investing for those who take it seriously. They got stocks, options, bonds, crypto, treasuries, and more with great customer service.