Cloudflare's Matthew Prince launches default AI crawler blocking, aims to build Spotify-like content marketplace for publishers

going absolutely nuts and spending a huge amount on it. Enthropic revenue is up 4x since the start of this year. Um and and like cloud code only launched in February. Uh and now cursor has poached the top two people from cloud code like the creator and the and the PM.

Um I like it's it's breaking news but like in terms in terms of in terms of like talent trades this is a really interesting one because it's at the engineering level not the model level and it's starting to spread out. It's starting to be like extremely competitive here. Uh and I think it's very exciting. Wow.

Well yeah we'll have to have you back soon since uh this is this is the top story every day now. Uh this is this is the real life. Uh thank you so much for joining. Great to see you. Have a great rest of your day. And next up we have Matthew Prince from Cloudflare coming in to TBPN. Welcome to the stream.

Some huge news from Cloudflare. Uh we'll let him tell us the story. How are you doing? Good to meet you. That's me Juke. Uh welcome to the stream.

Um I would love to start with um the news today and then I want to go into some of the the history of the company and and uh when you faced similar uh junction points and kind of uh turning points in the past. Uh so why don't you uh kick us off with uh the big announcement the big news today from Cloudflare? Sure.

So about 18 months ago we started getting approached by a number of content creators saying that they faced a new new threat uh online. Um, we spend most of our days, most of our business trying to fight cyber hackers, Iranians, Russians, North Koreans. And so I was surprised when the new threat was AI companies.

And frankly, I kind of rolled my eyes uh at publishers. I was like, publishers always complaining about, you know, the next new technology. Uh, and and and and they said, just pull the data, look at what is is going on.

And it was really striking um that uh over time uh you could watch as the sort of deal that publishers had made with Google you know starting 30 years ago which was you can copy all of my content and in exchange you send us send us traffic that that deal just didn't hold up anymore uh where they were copying the content but because they were providing answer boxes AI overviews uh they just weren't sending nearly as much traffic and and that was the good news.

uh as you looked at OpenAI, it was about uh uh 750 times harder than with the kind of Google of old to get traffic. With something like Anthropic, it's 30,000 times harder. And the reason why is because people are reading the derivatives, they're not reading the content.

It'd be effectively like if I asked an AI system, you know, summarize what happened on the technology brothers show today. I have less incentive to actually go and watch your actual show.

And that means that if you're making money through advertising, if you're making money through subscriptions, if you're just, you know, getting value out of the ego of knowing that people are watching your show, uh that's going away in an increasingly AIdriven web. And and we worried about that um quite a bit.

And more and more as we talked to publishers, we realized there was something that we could do about it because Cloudflare runs one of the world's largest networks. A huge amount of the internet already sits behind us.

uh we could partner with literally publishers, everyone from Ade to Ze Davis and everything in between uh to say that let's change the the way that that uh the internet works and let's change the sort of model where if you're going to take content, you should actually compensate the content creators for it.

And so starting on July 1st, uh we flipped the script. We made it as a default that if uh you're using Cloudflare um that you uh can by default for free uh block any crawlers from taking your content without compensating you.

And I think now the hard work be begins where we have to figure out now the business model around how we get compensation back to these content creators and make sure that folks like you who are who are creating real original content can continue to be compensated for the hard work that you're doing. Yeah.

Well, we used to joke saying that uh the the labs are welcome to scrape our content because we want to influence uh the the outputs of the of the model. But no, it's a it's a serious it's a serious issue.

We we were covering one of Ben Thompson's pieces about this whole debacle a few weeks ago and it was funny because at the time we were like this is there's so many different stakeholders here. Who could possibly try to figure out a solution to this problem?

And I obviously the answer you guys are uh a key player in this and you have the scale in order to kind of make changes happen. Can you Yeah.

Can you I guess like break down more kind of like the the kind of the market structure between and and how you see this actually working out and who deals have been done in over the past year because I know uh didn't News Corp do a bespoke deal with OpenAI for that exact thing?

it sounds like you are more equipped to kind of handle like the longer tale but also some of the bigger publishers. So talk to me about kind of the state-of-the-art of what those deals look like when they're onetoone and then what things might look like going forward.

Yeah, so I think that um first of all uh I I think for some time to come uh large publishers will do deals with large AI companies uh where there might be places that people need help are either small publishers or small AI companies. um you know and figuring out what has to happen uh there.

But I think even in the case of places where deals have been struck, what the challenge is is, you know, someone like OpenAI is paying a lot of publishers for their content and doesn't actually object to the the basic idea that they should pay for the content that they're using.

Um but but they do object to the fact that they're paying where all of their competitors are getting content for free. any market that exists has to have scarcity. And the problem, the reason why there hasn't been a market in this place space is that there isn't scarcity.

And so what changed yesterday is scarcity was in introduced really for the first time. And I I think the closest analogy is if you think back to, you know, before iTunes launched, um we were all downloading music on Napster and, you know, all the the free um pirate services.

That's effectively how the AI companies are taking data from any content creators today. And what you need is an iTunes. You need someone to come in and say, "Okay, let's provide a way where people can pay for that content and they can go back. " Now, now that's evolved over time.

I think our first version in the marketplace will probably be relatively naive and we'll figure it out. We no one's paying 99 cents for songs anymore. Uh we've moved to something that's closer to a subscription model like a Spotify. Um and and and I think that that's what what we will evolve to have over time.

And and if we do it right, you know, I think that we can have a large number of sellers of content obviously, but also a large number of buyers of content. Both the big sort of general purpose models, but then also very specific models.

And we've got to make sure that new entrance uh can come into this market and have have a vibrant way of of participating in competing. And so I think that for now, most of the deals will be one-on-one bespoke deals between publishers and AI companies.

You'll need to have some way of enforcing those deals and making sure that if you're charging someone, they can get access, but their competitors can't. And then over time, I think we'll develop something that looks much more like a robust marketplace where um uh content providers can set sort of what their price is.

is there can be a bid for maybe getting that that content exclusively for some period of time or or or whatever else and that's going to develop uh over over the next uh over the coming months. Is that the correct I is that the correct model to think that the the content producer would set the price?

I mean, that's certainly what happens on Substack and, you know, the the New York Times and the Wall Street Journal set their price, but I've always thought that this might evolve a little bit more like YouTube or like Spotify where there is a pool of dollars that are going into Chat GPT subscriptions.

I personally pay $200 a month and occasionally I fire off queries that probably land on the Wall Street Journal or the Financial Times and I would be happy for them to send 10 cents or a dollar or split off 50% of what I'm paying.

So you have $100 to kind of, you know, parcel out in the pool and then you see how much inference pulled what data from what what sources and and kind of have it all be driven by uh the like kind of the same volume uh uh like the volume algorithm that you see on on uh on YouTube or Spotify.

Does that seem like kind of what the long-term um outcome might look like or do we need to also get to some sort of like ad driven marketplace because that's ultimately where YouTube how YouTube prices these things?

Yeah, you know, I think it's going to be um I I think that we there's there's a lot that we still have to figure out. I I I last week um traveled up to Stockholm to meet with Daniel A, the uh the founder of of Spotify, and and it was, you know, it was really um it was it was interesting.

He said, you know, the the ultimate model that Spotify settled on uh I think it's I don't want to put words in his mouth, but but it was basically like it was it was sort of the most naive of all of the different options that he had come up with and and it is works as it is a pool of funds and then they divide it up based on on basically how many minutes of of content you you consume and there are different rates for music versus podcasts versus audio books and um and and it was fascinating to to see.

So you could imagine a version where Cloudflare effectively negotiates on behalf of all of the content that sits behind us with the big AI companies. People can opt in or opt out of being a part of that pool. The more content that's part of that, the more valuable it is to the AI companies.

And then we would look at how we could split that up across how much crawling is actually um done across that. I think one of the real keys will be how can we give hints to the AI companies on what is the content which is the most valuable to them.

So for instance, if Taylor Swift releases a brand new song and maybe does an interview about it, that's incredibly valuable content. If you're a I don't know chatbot for teen lonely teen girls, right? On the other hand, um it's probably not so valuable if you're sort of a chatbot trying to be the world's best doctor.

you know may maybe there's some value in in sort of bedside manner but but largely you know that's not going to be a big deal in your in your care.

On the other hand, if a research study comes out on acetaminophen's effects on on uh you know different types of chemotherapy, very little value for the chatbot for teen girls, but incredibly valuable for the doctor.

And so what we have to do is actually I think on a on an LLM or a AI model by AI model basis be able to give them signals back that say this is the sort of thing that you should be paying attention to, that you should be indexing, that you should be actually um scraping and paying for.

Um and and the same way that in you know the music industry we've got reviews, we've got different types of music that that people are you know you might be into country or you might be into classical and and and it kind of allows you to compartmentalize that.

I think that whole ecosystem has to develop but once that does um something very much like Spotify could be the right answer uh long term and again I think we're in a unique position to pull it together. Talk briefly about stable coins.

I remember in this Ben Thompson piece he was people have been suggesting this idea that stable coins could be some part of the solution to to or backbone for a new uh business model for the internet.

At the same time while I'm bullish on stable coins someone like a cloudflare can set up the infrastructure to just use traditional financial rails to process payments.

So, I'm guessing that it's not part of of of what you're This was Ben's uh reaction to the the the original sin of the internet was advertising and this idea that Mark Andre when he built the browser didn't build in payments because it was this multi-party discussion to figure out how to actually integrate payments into HTTP didn't happen.

the first MCP spec that's the model context protocol spec for AI agents to interact with each other doesn't include payments but he was kind of saying that it might be on the horizon but would love your take yeah you know I think um the answer doesn't have to be 100% one or or the other uh and so I would guess that there are there are going to be um very traditional kind of traditional um financial rails that will be ways to pay pay people out but there will also likely be something that is uh blockchain based or or stable coinbased Uh and and again, I think we're looking at um all of the different providers uh that are in this space, including potentially creating our own uh stable coin that would be part of how these these transactions um you know, take place.

And I think you're exactly right that um there's a natural extension from what we're doing here to how we figure out around what the payment rails look like for MCPs because as your agent is going around interacting with things um that's also going to be something that is is either um you know charged for or that you that you actually um pay get paid to do.

Uh and and there's got to be a way for those those transactions to take place.

And to the extent that you can reduce the transaction costs for those transactions as far as possible, uh then that's that's a pretty interesting uh way of of uh of of being able to um process, you know, very high volume, low dollar amount uh uh transactions. So I think that's right.

My only one quibble is that I think that the original sin of the internet is not that it didn't include payments. It's that it uh was that there was no privacy uh built into it and your IP address reveals far too much about you as and who you are. We don't even have to get to cookies, but that's a No, we totally can.

What about we we we've talked about content creators, publishers. What about uh the the apps of the world, the Ubers, the Instacarts that actually have really meaningful advertiser businesses, you know, businesses built on top of them?

Uh do you expect there to be any type of of this functionality rolled out to them over time or or is that something you think they're thinking about quite a bit?

Because if I can go into chat GPT and that's my default daily assistant, let's say in a few years from now and I'm like, "Hey, order me groceries and order me this Uber, uh, you know, I'm not in the app seeing ads potentially, you know, being upsold and and that could be a problem for those types of companies.

" Yeah, totally. And I think that I mean um, you know, robots don't click on ads.

uh and and so and so the the sort of model that that part of the um you know revenue stream to support these services uh has to come from from somewhere uh and if today you know Uber is able to um monetize the time where I'm I'm waiting for a car to come by showing me an advertisement um that that means that they can effectively charge me less for the ride.

Um, and so it may be that if that ride is booked through an agent instead and they don't get that that advertising experience, and again, I don't know enough about um, you know, Uber's business model to know how how important that is to them, but if they don't get that, that maybe it is that there's some sort of premium that an agent booked ride receives versus a human booked ride.

And again, I think that that's um that that that's just we've got to figure out what that what that uh what that looks like. I think that's going to be less up to Cloudflare.

uh I think it's going to be you know our job will be how do we provide those payment rails how do we provide the systems how do we make it so so that we can have guard rails on as agents interact uh uh with applications that are there that you as the application provider can set those those rules and controls and then and then agents can interact within within that system and MCP is is sort of the leading candidate uh for providing that but you're you're right that that's a sort of draft version of that that specification and there are a lot of things that have to you know, mature in order for that to be something that is the the real um kind of agentto agent uh system that runs that is likely to run what will be an increasingly AIdriven web.

Yeah. Can you talk about any previous crucible moments for either Cloudflare or just like the internet history broadly? This feels like a moment where you're potentially trying to I mean deal with a shift in the in the overall internet structure or the incentive structure.

Uh but uh it's a reaction but it's also an action that should reshape the the uh the the relationship between different parties on the internet.

uh are there different moments in the history of Cloudflare or the history of the internet that you think uh we could draw on as historical analogies um to learn from to see how this might play out?

You know, I think that um you know that Google really uh 30 years ago defined what was the business model for the web, which was uh you you create content, search drives traffic, and then you monetize that content through advertising or or subscriptions or or ego.

Um and ego and monetization, not quite the right thing, but you drive value from it in one of those those three ways. Um, and that that's held up for the last 30 years.

I I I I think that this is um the first time that I've I've seen something where I thought, wow, there there are really going to be um very very very fundamental changes to the structure of how the internet works if we move from a searchdriven model to an AIdriven uh model.

And and I think that um what I hope is that we can actually make this something where um what gets rewarded is not what gets rewarded today, which which largely is, you know, who can write the headline that produces the most um you know, uh sort of inflammatory response uh in in all of us.

You know, who who basically drives the most cortisol uh you know, gets the most clicks and and therefore the most ad revenue or the most subscriptions. That's that's not a that's not a healthy way of looking at the world.

I think you know the alternative is if you imagine that kind of all of the AI models together are a good kind of assemblage of of an approximation on human knowledge.

If we can identify where the holes are and then create incentives for content creators to actually fill in those holes that actually is going to move humanity forward. It's going to make the AIS better. It's going to it's going to be a much more interesting thing.

And that's I think how we're going to we're going to be um creating it. I I can't think in the last 30 years of of a a as seinal moment. I think there are some things that are along the way. I think 2016 was a really a big turning point year um for for a lot of reasons.

It was sort of the year that it went, you know, technology went from being able to do no wrong to being able to do no right. Uh and and sort of rise in populism um around the world. You have you have Brexit in the EU. You have the first Donald Trump election.

You've got Xi and Modi consolidating power in Asia, the Filipino election.

Um, and I think that that was a point in time where, you know, also driven by um Snowden uh shortly before that in 2015, uh, that that a lot of the world lost faith in kind of the US model of internet governance and it wasn't quite ready to adopt the Chinese model, but that's that's definitely been a shift uh, which is substantial and I think we're still watching that uh, play out.

that's that I think is probably still remains the biggest threat to the overall internet.

But this shift to AI as well um you know again I I think is is is a real challenge but it's also an incredible opportunity if if again we can make the rewards incentives for content creators not be who can produce the most most cortisol but who can produce the biggest sort of incremental improvement in human knowledge.

um that that would be an incredible win and uh and something that that um you know I I don't quite know how to get there yet, but I I think yesterday or today and uh you know we took in July 1st we took a a real step in uh in in moving in in in that direction and making sure they compensate content creators can be compensated for for really producing uh information which is valuable to advance human knowledge.

What do you think about the future of investigative journalism? Uh there's been a big shift in just content creation into independent creators, but a lot of that is actually downstream or separate from the type of work that investigative journalists do.

I'm thinking of like Seymour Hirs hanging out at a local army bar for a year before he gets one scoop that he can write a book about. and and he and these investigative journalists, they tend to, you know, John Keroo was on the payroll of the Wall Street Journal for years.

He was writing stuff, but then he blows the Theronos case wide open, eventually a book and then a movie.

Um, and and I and I imagine that with artificial intelligence, the value of a scoop could actually go up because you could use an LLM to read through the clickbait repurposed version of the article to find, wait, their sourcing actually, this person posted it on the internet earlier.

They're the one that actually did the hard work to get the scoop. they should get 90% of the value and 10% should go to the repurposed ragebait version as opposed to the economic model now which is maybe you get 90% more clicks on the on the viral you know repurposed content and the scooper actually gets 10%.

Do you think that's possible or or how do you think that investigative journalism will evolve? Yeah, this I I don't know this is an answer to your question but but it but it's um I thought it was I thought it was interesting.

As I said, I met with Daniel A um up in up in in Stockholm and um and and he told me he said, you know, we were talking about this and he said, you know, one of the things we do at Spotify is we actually surface all of the different queries that are sent to Spotify that we don't feel like we have a really good response for.

Um, so if somebody searches for, you know, I want a a a song to a reggae kind of beat with it's all about how much it sucks when your sister steals your car and your dog.

Um, like not to get too specific, I'm not again I don't know what what what these things are, but there are people there musicians who take this list and they actually write songs based on what people are looking for. No way. and that that they make tens of millions of dollars a year doing just exactly that.

And there's something about that that I think is really beautiful. Um as opposed to the alternative of just you know just trying to do a little of more meto of what everybody else is doing like taking a different twist using signals from the market in order to go find that.

So you know again that's not investigative journalism by any means but I think it is actually advancing kind of the sort of good of humanity. It's taking what a place where there's a a missing need and it's filling that missing need.

And so I am hopeful um you know and and relative I mean I'm a relatively optimistic person and and I'm hopeful that if we get this right that we might be on a golden age of of content creation where people actually are rewarded for doing things that advance human knowledge as opposed to just doing things that produce as much cortisol as possible.

Yeah. The the idea of a knowledge bounty, right? It's like, hey, there there's there's groups of people or a single entity online that will just pay you to produce really high quality information around this specific topic.

And this the internet always did this sort of organically, but it would just be somebody spinning up a blog or maybe an Instagram account or something like that and they would just start blogging about something that was niche and then an entire network would form around it and other people would contribute to it.

So, I think there's quite a lot of precedent.

Uh but it would be exciting to like put that reward out before and not saying hey you have to do this and then maybe you'll get some ads associated with it later but like basically like the the pool of capital is here and you just have to deliver that knowledge that that people already want and we know it because they're searching it on LLM and things like that.

Can I I think that the um I think I think I'll tell you the sort of black mirror kind of dystopian outcome uh if we don't get this right which is I don't think that I don't think journalism goes away. I don't think that research goes away.

Um, but you could imagine a world in which we go back to something kind of like the Medachche where there are five big AI companies and they basically employ each of them all of the journalists, all the researchers, all the academics to be able to do the research to feed their systems because you're still going to have to have original content.

People are still going to have to be able to produce that in order for the AI systems to, you know, fill in the holes in their kind of effective block of Swiss cheese that that is that is their model.

Um, but but I I think that that's that's a bad that's a pretty bad world where where again because you'd then probably have like the conservative AI and the liberal AI and I mean all these things that we've kind of that the internet kind of broke apart.

You could imagine AI as a system that is just this massive consolidator uh in the future. And I I think that's a bad outcome.

And I think a better outcome is if we can figure out um how how can we reward people for again driving knowledge um but but but not get to a place where all of them have to be employed by by one of the five you know big yeah I actually talked to somebody very high up at one of the labs who said that was was saying that like if you take the total uh the total salaries of all the journalists in America it would be a fraction of what the AI labs are spending right now just acquiring ing data and so it it's not a financial question.

It is a political cultural question. It's interesting.

That's also the argument why I think that and and I've been really positively um surprised as we've talked with because we've talked with all the big tech companies, all the AI companies about this and they and they all like with with maybe one or two very small exceptions um they all say yes we understand over the long term we have to pay for content.

And so they get that. What what what what's key though is it has to be a fair playing field. Um and that fair playing field everyone sees that slightly differently. So if you're a new incumbent you want to say well Google can't be advantaged.

If you're Google you have to say you know well we have it has we have to be respected for the ecosystem that we've helped build and flourish and we should be able to you know this. So everyone's got a slightly different take on this. But I I'm actually encouraged that that the the dollars are there.

Uh that there is an acceptance that this is is uh is something that needs to happen and that if you if you if you ask them, everyone says yes, we need to do this as long as you can create a level playing field. And again, that's where I think our role comes in is how can we create that level playing field?

How can we create scarcity? And then how can we make sure that content creators are being compensated for the the hard work that they're doing? Yeah. Zooming out just for a second, what is the state uh how's the internet been for the last few weeks?

It's been a busy uh month in or last month in geopolitics, I'm and I know that makes your job a bit harder or at least I would imagine uh things get a little bit busier. Keeps it interesting.

I mean, it's whatever whatever you see sort of in in a in the sort of kinetic side, the physical side of a conflict is is usually replicated in the in the cyber um side of the conflict. And so um you know we as the hostilities um ramped up between Israel and Iran.

First of all we saw very clear signals that that something was coming ahead of time which is it's interesting how there's sort of a digital residue which shows when uh there's going to be uh kinetic action uh uh taken and so and so we were we were watching uh that that pretty carefully.

uh big attacks um targeting both um Iran uh one of Iran's largest banks was hacked by what looked like a group of um is Israeli um activists um the largest cryptocurrency exchange in Iran was hacked and all its funds were drained u again looking like it was either by the Israelis or or Israeli um sympathetic uh attackers.

Iran has has tried to do the same thing against Israel. Israel's um cyber uh defenses are much stronger.

Um but we're starting to see that Iran is targeting other um uh other largely US uh financial services firms uh critical infrastructure firms uh to try and see to sort of try and strike back from what what is um you know h how do they strike back against the US in a way that um that that is harmful but doesn't start World War II?

And I think that increasingly um unfortunately cyber will be a big piece of that. And so we're um you know we're we're doing everything we can to to stay in front of that, make sure that anyone who needs it has has protection and that we can keep keep the lights on everywhere in the world. Last question for me.

Um there's a number of larger overarching tech narratives. I'm interested in if you're losing sleep over any or tracking any in in particular focus. Um the AI talent wars and kind of pay inflation for top performers.

Is that an issue or something you've been tracking or uh the difficulty to build larger data centers and infrastructure the GPU short of the chip wars? Any any of those topics kind of particularly relevant to you or your business or or kind of keeping you up at night?

Yeah, you know, I think we're we're not trying to build a foundational model LLM. So So we're not we're not as much competing um for the the bleeding edge AI researchers.

I will say that we've I've been um really amazed at at both from an academic side and then also just you know pure talent side how many people are interested in working on that problem of something like how do you score content in order to be able to create a marketplace and so and so that that's something that's been um you know pretty exciting.

We're we're very excited about the uh the the people who are coming to uh apply to work for Cloudflare to to work on problems like that and like today has been a record uh applicant day for us uh on the on the Congratulations. Record.

Sorry, I hit I hit the air air horn for you on the record applicant day, but it's good. It's good news. Uh and then, you know, I think um you know, we we've uh we tend to put a little bit of equipment in a lot of places and so that we don't tend not to have kind of massive multi- Yeah.

You're not running into like power constraints or rare earth element constraints or water, right, constraints? Yeah, we we we do in but in in in kind of a micro way where you know we put GPUs at the edge of our network. we have to live with a power envelope that we're given by an ISP.

And so we're doing things to try and figure out how to just get the best power efficiency out of the the GPUs that are there. The answer to, you know, more and more need of GPUs can't be everyone has to turn up their own nuclear power station.

Um so so I do think the place where we're pushing suppliers and AMD is doing some really interesting things in the space. Qualcomm is doing some interesting things in the space.

fruit company down in Certino is doing some interesting things in the space or how do you focus on yes performance but performance per watt and and we've seen this game before with with Intel saying oh that doesn't matter like just just get the most performance possible you can water cool things all kinds of things that that tends to not be the winning strategy over the long term and so we do think that there will be a renewed focus on just energy efficiency around GPUs and and we're really pushing uh that in a way that because because again we can't build our own data centers in the business that we're in.

Um, and then and then chip shortage. The chip shortages are funny. Like it's there's never been a chip shortage in any time in the history of of silicon that doesn't turn into a glut. Um because it's Hey, but this time it's different. Yeah, this