Julius AI launches Deep Finance benchmark and finds Thinking Machines' open-source model tops American AI for finance tasks

Speaker 2: Speaking of finance. Speaking of finance. Hi, finance. We got Rahul Sonwalkar from Julius AI.

Speaker 3: There he is.

Speaker 2: Fucking absolutely diced and shredded as always. How are doing, Rahul?

Speaker 7: Doing fantastic. How are you guys?

Speaker 2: We're doing great.

Speaker 3: It's great to see you.

Speaker 2: Always good to have you on the show. Take us through the latest in your world, in Julius' world in San Francisco. Take us wherever you wanna go.

Speaker 7: Alright. So we just released Deep Finance. Mhmm. It's the first ever long horizon high finance benchmark.

Speaker 2: Okay.

Speaker 7: So the goal is to see how good models do on real world big deals that you would see in investment banking

Speaker 2: That's good. And

Speaker 7: private equity.

Speaker 2: We love this.

Speaker 7: It's giving... Money users. Exactly.

Speaker 2: Yeah.

Speaker 7: So we add... We we work with experts who have experience from firms like Evercore Sure. JPMorgan UBS, and Harvard Business School

Speaker 5: Yeah.

Speaker 7: To help build this benchmark so we can understand how good the frontier models are.

Speaker 2: Yep.

Speaker 7: On an actual real deal.

Speaker 3: Yeah. So... So how does it like like... It's very cool, but like I'm very curious how this... Like the the complexity in a deal ends up coming, you know, you... Let's say post term sheet, I don't know where you kind of try to start in in...

Speaker 2: Yeah. Where does this start? Where does it end?

Speaker 3: The complexity comes from like the, you know, emotional dynamics between all the different individuals involved and combining all these different preferences. And so how... Did you have to like simulate the interactions that come after you go from like a term sheet into the actual, you know, let's say like purchase agreement in the case of, you know, an m and a deal?

Speaker 7: Exactly. So what we start with is real bare bones data room with a lot of messy data. Mhmm. You know, what what does an actual deal look like? An actual deal looks like, hey. We are looking at this company. Here's a bunch of files and a bunch of data. Go figure out if it's worth pursuing this deal. Mhmm. What does that look like? Can we take this to the investment committee? Mhmm. It's a super high level open ended instruction. Mhmm. And we wanna evaluate these models on is, well, can they fully understand the business Mhmm. In a way that a private equity or investment banking professional would? Do they just look at revenue and just assume that, okay, that's the revenue? Or do they actually go trace the revenue down to the to the numbers, into the sources, and see if it all adds up? Mhmm. In real world, you see a lot of discrepancy.

Speaker 2: Yeah.

Speaker 7: You see, like, the management report one number, and the audit report another number, and then we actually do your own math, you find out that it's something else.

Speaker 2: Yeah.

Speaker 7: And the models deal with that complexity. Mhmm. So these workflows are actually designed to to reflect that robot complexity. So there's, you know, a lot of emails in the tasks. There's raw data. There's some financials. And then the models are tasked with coming up with a... You know, financial model, like a DCI, an LBO model Sure. And come up with a memo to go pursue a deal or not.

Speaker 2: How do you think about the the business impact for Julius on this? I mean, seems like you're getting close to maybe you already have built like an RL environment around this. You could sell that to a big lab. You can use it to partner with other model companies to fine tune things. You can integrate directly into your product, build a harness around it. Like, there's so much that you can do once you have the data, once you understand the problem. How are you planning to to actually deploy it for financial gain?

Speaker 7: Absolutely. To be honest with you, don't know. Okay. So we just launched Julius research. And the goal of Julius research is to go pursue interesting problems that are relevant to Julius and we think are gonna help impact the model progress on on on knowledge work.

Speaker 1: Mhmm.

Speaker 7: Right? And so one of the things we did with with this launch is we specifically did not measure the models on our own harness. We use the native CLI. So for open AI models, we use codex.

Speaker 2: Sure.

Speaker 7: For Clot models, we use ClotCode. Yep. And just to be independent and really understand where the models are. Mhmm. And so that's the first step, is push with the curiosity and see if we can build the first ultra long horizon benchmark for finance workflows, and we did. And now what we're doing is understanding where do those different models stack up, what are the different idiosyncrasies in how these models pursue these tasks. Yeah. So one of the things we noticed is models that are really good at coding, you know, they tend to search a code base. Right? They didn't search a code base to understand it. That doesn't always work in finance. Yeah. If you just search for revenue in the documents Yep. And you see a number, you can't just assume that's the revenue. You have to see if it all adds up.

Speaker 2: Yeah.

Speaker 7: And so...

Speaker 2: Well, many many modern deal makers aren't even looking at the revenue. They just ask, like, who else is in over a text message, and then that's their criteria. So you you should fine tune a model on that. They just ask ask ask who's who's leading and then and that's the entire that's the entire criteria.

Speaker 3: Oh, you want me in the round? Which round?

Speaker 2: Which one? Am I in the trunch? We'll figure it out. Yes. Can you get me up to speed on the latest and greatest in Thinking Machines world? Sort of a quiet company. There's a leak that it sounds like they're they're closing around 5,000,000,000 at 40, incredibly impressive. Some of the open source models have been really good. I heard that they're the biggest partner to to Reality's fund, believe. Is that right? But it seems like there's some traction there. You have some data. What's the latest in TML world?

Speaker 7: Yeah. So we actually... What's interesting is that they released InkLink too small yesterday

Speaker 2: Yeah.

Speaker 7: Which is the same day we released our evals. Yep. And so our evals weren't public when they trained this model.

Speaker 2: Interesting.

Speaker 7: And when we ran their model on our eval, we found that it's actually the best American open source model Yeah. And second best open source model overall

Speaker 2: Yeah.

Speaker 7: After Kimi. And so that's really, really impressive. Yeah. Especially if you see the leap from Inclink one to Inclink two Yeah. In a matter of months, that's really impressive. I think things are really really starting to work there. And especially if they're working with Reality's fund

Speaker 2: Yeah.

Speaker 7: I can see why understanding how good their models are at finance Sure. Is important. And also, it's open source. So that doesn't mean that this is just how good Eagling Too Small is gonna be. You can actually both train it to be even better.

Speaker 2: Yeah. And a lot of financial institutions might want to have data sovereignty. They might already already... They might already have their own infrastructure in many cases. And so it's interesting that you're seeing like the bleed over of whatever they're doing with their very secretive financial partners that they don't talk about. It's sort of leaking in their open source models which is great because then that's something that everyone can benefit from and it also serves as top of funnel. Exactly. To It's see. Well Exactly. I mean Yeah. Please. No. No.

Speaker 7: I was gonna say, you know, especially with American open source models, I think that ecosystem really needs to catch up. Yeah. And it's so impressive to see that it's even better than Nemotron from NVIDIA. Yeah. Because that gives institutions, even government agencies, and private companies more conviction that they can actually both train these models and and not... Yeah.

Speaker 3: Since it's impressive because you assume that that, like distillation isn't like the the the main sort of like advantage Yeah. Of of of a team. Right? Just because obviously NVIDIA and Thinking Machines are going have way more scrutiny around how they're actually creating these models. Impressive. Major white bill. Major white I like it. Well, thank you so much for calling.... These days.

Speaker 2: We need it

Speaker 3: When you see one, grab a glass of water Yeah. Take the white Yeah. Hey.

Speaker 2: Have a great rest Good your

Speaker 3: to see dude.

Speaker 2: Have a great weekend. Talk to you soon, Rahul. Goodbye. Cheers. Let me tell you about CrowdStrike.