Thomson Reuters CTO Joel Hron: legal industry is on fire for AI, RAG and domain experts key to hallucination control

And so one last thing was that we would compile the names and photos of everyone who got a job at Goldman Sachs and it was just black and white portraits of all of them listed like vertically and it's was titled our best and brightest. So something like that for you guys. There we go. Let's see it. Love it.

Well, very very excited for you to be at Arena. Who who's gonna take the companies public if not Goldman Sachs? I don't know. Who's going to issue the the massive uh debt to back these corporate titans if not Goldman Sachs? Well, they are the best and brightest. They're doing important things. They are.

Anyway, this has been fantastic. Thank you so much for stopping by. Great to see you. Thank you. Cheers. Talk soon. Uh up next, we have Joel from Thompson Reuters coming in the studio. Uh he was supposed to be on AI day. He's buddies with Swix. was introduced to me by Sean Swix who's been on the show earlier.

Uh we're going to dig into how Thompson Reuters is using artificial intelligence and what's going on in the enterprise. Welcome to the stream Joel. How are you doing today? Guys, how's it going? It's going great. Good to meet you.

Uh would you mind kicking us off with an introduction on yourself and then I'd love to know a little bit about the shape of Thompson Reuters business. People are obviously familiar with the brand, but uh on a day-to-day basis, what's actually uh going on the technology side? Yeah, you bet.

Um, so I've been at Thompson Reuters a few years. I joined via acquisition uh as a CTO of an AI company, TR Acquired a few years back and have been with the company since then. I think a lot of people know Thompson Reuters from the news business.

The obviously that's the most public facing part of the business, but from a from a revenue perspective and a technology perspective, it's really I would say a relatively small percentage of of what we do.

You know, we're one of the top two or three software providers in the world in the legal industry, in the tax industry, compliance, audit, and risk. So we really cover I would say a pretty broad expanse of the information services uh industry.

Um for a lot of years we were really seen as a as a content provider or an information provider within these businesses. But uh software and in particular like search has been sort of a foundational piece of the technology that has kind of underpinned delivering that information to customers.

again across across the sectors legal, tax, compliance, etc. Yeah. Okay. So, walk me through uh of those verticals, which one is seeing the most acceleration from the current suite of AI tools?

Which ones uh kind of maybe next up at bat and not really uh the models aren't really having as much impact as you expected so far, but where are we getting the most leverage in AI? Yeah, it's a great question.

You know, if you look at all of those industries, you wouldn't necessarily look at anyone and and immediately say they are uh particularly eager to adopt technology or aggressive in in that sense.

But, you know, for the last two two and a half years, the legal industry has been on fire and I'm sure you guys have talked about it. There's a a ton of capital coming into that to the legal tech industry in general. Yeah.

Um but really from the customers too have been extremely hungry and eager to use this technology and it's having profound impact on uh the way those professionals think about uh doing their work and operating their businesses and and the underlying businesses business models themselves.

I think uh you're starting to see more um more progress being made in industries like tax though. I think sort of the quantitative capabilities of the models kind of lagged the the language capabilities of the models. Mhm. Um but I think that's improved.

But I also think if you look at like the tax or the audit industry that particularly workflow kind of heavy and I I think as agents have really taken off in the last you know six to nine months and the tool call and capabilities of the models have improved uh it's really opened the door to you know deliver like real true automation and productivity uh with some of these agentic frameworks and reasoning based models.

Yeah. Talk to me about I mean I imagine you're not focused on pre-training a massive model to get to the frontier. You're probably a buyer of that capability, but we're seeing a lot of deals get done. Open AI's fine-tuning models for clients above $10 million. They have they have even cheaper offerings.

Palunteer is doing fine-tuning work now. We just heard Thinking Machines Mir Marudi is doing some fine-tuning reinforcement learning for businessto business applications. um what are you a buyer of right now? What are you focused on uh building internal applications around?

Where do you want the various uh pieces of the building blocks of a great artificial intelligence enabled product to sit? Yeah. Um this might be a little bit of a long-winded answer.

I would say first and foremost when we look about look at shipping product into the market quickly we're certainly a buyer of these models and and we're close partners with Anthropic OpenAI Google and others in that regard you know when those companies are working on their next versions of models a lot of times they come to us to to help them evaluate and test certain capabilities of the models because the the type of work that's done in in the legal industry in particular is is is challenging.

It's not a simple like rhetorical response. It's it's uh a deeper level of reasoning that happens in terms of the the types of problems that we're solving. So I would say first and foremost like we're we're we're buyers of the technology. We have done you know fine-tuning engagements with several of these providers.

super excited about the sort of RFT uh trend that may be coming with with many of them in terms of being able to apply more RLbased methods to to that fine-tuning. Um but we're also I I would say builders in some sense.

We're we're not you know pre-training models of the size of of uh GPT4 or or or claude uh sonnet but um but we do engage in that.

So about a year and a half ago, we acquired a small company with a couple folks from from Deep Mind who have come and and really built up our deep learning expertise and deep learning training expertise within our team.

You know, we've got about one and a half trillion tokens of our own proprietary content uh in in legal and tax and news and and like I said, that's been sort of one of our defining assets for a long period of time. Yeah.

uh as RL really becomes kind of the the main mechanism for driving further improvements in the model, uh we see our talent, you know, our 4500 domain experts across legal and and tax and otherwise uh really being pivotal in that process of driving performance into the model.

So, uh, we're certainly a buyer, but we're definitely trying to exercise, uh, the data and the content that we have proprietary to us, but also the the human experts that we employ to drive that.

Can you give me some extra color on on all the different value kind of on the table with one of these deals between a big company and a foundation model provider? Because it can seem so simple. It's like I'm I'm I'm buying an API, but there's actually a lot more to it.

And we were talking about this in the context of uh Apple potentially partnering with Anthropic or OpenAI. And there's actually kind of a, you know, an odd relationship there where if Apple's sending a bunch of users to chat GPT, that could improve the model and that could make their business stronger.

They could wind up servicing ads in the responses like Google does. And you could wind up with a situation where, you know, Google is paying Apple, not the other way around. You could see ChachiPT eventually paying Apple for the right to be on the Siri button. Um, but in your business, you have over a trillion tokens.

That's pretty valuable data. There's a question of, you know, what what value are you bringing to that side of the deal? And so, how are you thinking about doing a great deal that sticks and is aligned economically over the long term with a foundation model provider?

Yeah, it's a really it's a really hard question to answer and certainly you have to kind of like have some crystal ball to project out what the next two years are going to look like and it's it's hard to project two weeks to to do that effectively. We got one here in the studio for this exact reason. Helps quite a bit.

It helps a lot. Yeah, it's important. I'll borrow that next time I'm out there.

Um, so you know for us I think like I said our people and our experts and the data that they create ultimately I think is is sort of core to our identity and core to what we believe is valuable to us and we try to really hold on to that and extract as much value from it into our IP as we can.

I think where where there's mutual benefit like I said before is um we provide a lot of insight and direction to these foundation model providers in terms of like what the models are failing at sort of what kind of capabilities.

you know, they have these public benchmarks that say, you know, every week one model's better than the other, but they don't really represent the real world. And in particular, they don't represent the real world of like how professionals do work on a day-to-day basis.

And we spend a tremendous amount of time building real world evaluations into those kinds of things. Uh, and not only that, but we spend a tremendous amount of time and energy grading and evaluating those on like real world rubrics for how a lawyer would interpret this. So, it's not like it's a right or wrong answer.

It's is it helpful? Is it harmful in some way to a lawyer? These are types of things that like create a require a tremendous amount of just domain understanding to to do well.

And so that insight is super valuable to the foundation model providers to understand how they need to adjust their data sets and it benefits us on the flip side because we get more performant models uh out the back end that that are that are uh better suited for our tasks.

H how much time are you guys spending just on preventing hallucinations when you talk about domains like law and tax?

You know, it's not uncommon for a a lawyer or a tax attorney to hallucinate themselves, put, you know, put the wrong number somewhere, you know, have have the right but but usually there's, you know, hopefully checks and balances in terms of someone else reviewing it, the client reviewing it, making sure that it's getting to the right place.

Um, but that feels like the number one of the bigger issues kind of holding back LLMs from, you know, unlocking the full value. So I imagine it's it's a big focus for you guys, but but what are your thoughts? Yeah, I mean all all the time.

I mean like I think that that's one of the core things that we do, you know, and you're right that hallucinations existed before AI was involved here and they exist now and they no matter how good these models will get, they will exist in the future.

I think there's there's some element of like um uh that that that will be the fact. I mean foundationally the way the models operate like it it is a propensity and it can happen.

Uh so but we spend a lot of time trying to build guard rails trying to build uh methodologies to to prevent that or at least mitigate it in in as as many ways as we can.

Uh I would say this is where though the the UX of the applications really becomes like more and more critical like creating the environment where a lawyer like knows what they need to validate knows how to do that validation uh and can do it quickly and efficiently so that they're still saving time in whatever they're doing I think is a critical part of the the design of the applications and how they operate.

And as much time as we spend on the science side trying to build algorithms that prevent that, we spend just as much time on the design side trying to make sure we build experiences that allow lawyers to to build that that confidence and trust that that they need. Very cool. Well, thank you so much for stopping by.

We'll have to have you back when there's more news from your world. And I we hope you have a great day and have a have July 4th, too. Yeah. Appreciate the We'll talk to you soon. Cheers. Uh, Jordy, how did you sleep last night? I think you might have me beat. I was a little rougher. I got an 81, but how did you do?

Go to eight asleep. com/tbpn. Get a pod five. Actually, unbelievable. Did I beat you? Did I beat you? Play the sound effect. You know, you know the reward I get. 81. What do you get? I can't believe it. 52. 52. Oh my god. Wow. I I'm said I said for what it's worth, you have put on a fantastic performance this show.

I would have