Harjot Gill of CodeRabbit: GPT-5 nearly doubles code review performance, conversion to paid customers expected to jump

Yeah. Yeah. Yeah. Yeah. That' be great.

Cool stuff.

Um, well, in the meantime, before our next guest, let's tell you about eight Sleep Get a Pod 5-year warranty, 30-day risk-free trial, free returns, and free shipping. And we will uh have our next guest join us from Code Rabbit. How are you doing? Good to meet you. I'm good.

Good to meet you as well. Thanks for having me here.

Uh what's your reaction to GPT5? How long have you been playing with it? What are the biggest improvements that you've noticed?

Yeah, I would say mind-blowing, right? I mean, we have been playing, our team has been playing for like a few weeks now. U tested a few snapshots and it's amazing. It's a generational leap. We would say like we have been using open AI models like I don't know how much you know about Kodak. It's been like couple of years we have been on open AI anthropic um and our product is a very reasoning heavy product like one of the very few use cases where you are have a PhD style problems where we have to do code reviews that's what code raid does like users open up pull request our agent uh uses reasoning models to find issues like race conditions or security issues and so on

um so yeah so we've been testing GPD5 on some of the hardest pull requests we have in our golden data data set. So we maintain a data set where we track progress of different models and progress of AI in general. So we have like many problems that no model is able to use uh solve so far like I mean GPD5 but so far it has the highest score. We would say it's like almost 2x better than the next 03 or solid or opus at this time.

What's the customer value there? You think that uh like all the customers will just notice that the product gets better? Are you going to upsell folks? Are you going to uh like how do you play this given that this model is now in public availability? Every company, every competitor can can access it as well.

Yeah, there's no upsell. Like that's the thing with AI. I mean, for the same price or even like better prices, you're getting much more AI, much better A B A B A B A B A B A B A B A B A B A BI like that's the whole idea how fast this space is evolving. Um so yeah, from the pricing point of view, we don't see like this to be like a separate plan or something in our product. I mean the pro for the same price per month customers will now just get better quality of results with code rabbit.

Um what's next for the business? Um what kind of customers are you going after? Who do you think has been on the fence and this release is going to be the thing that gets them to actually jump into the world of AI? Yeah, we could track the topline metric like one of the things we track very closely in the company is like how many signups to the paid customers we get

and that number has been constantly improving since GPD4 uh GPD4 turbo GPD4 actually dipped. So there was a time when GP4 was almost like a Windows Vista of Elius like um it's funny like how we kind of trusted the evals and we thought it's the same model but in in a way it was imperial in many ways. Um then we saw a huge improvement after 01 came out. O1 preview was a game changanger for us. Even at that time our conversion doubled actually right I mean so we went like more like close to 30% success in getting the paid users and now with JPD5 we are hoping we can see another big jump in the in the number of people who start becoming paid customers and and how many people churn. So those are the real numbers like one is like wibes like how people like respond to the model and we get angry tweets or not though that's the other part but the other thing is like the actual revenues uh whether it moves the needle for us and that needs to be seen like one of the things we have seen even though you test these models in a lab they be it's not like a huge data set but once you actually are in the wild you see hallucinations some of those scale issues at scale pop up so those are something we'll still be observing over the next few days to see whether it's like spar only um 80% of the cases, but then if the false positive rate, the hallucinations are too high, then also it's not a great model, but that remains to be seen.

Yep, that makes a lot of sense. Uh well, thank you so much for stopping by. Congratulations on a new new tool in the tool chest,

new toy.

Uh we will talk to you soon. Have a great rest of your day.

Cheers.

Goodbye.

And let me tell you about public.com