Right now, an AI crawler is probably reading your blog — and not for the first time. According to data released by Cloudflare, more than half of all AI crawler traffic worldwide is spent revisiting pages it has already scraped once. Your newsletter archive, your columns, your blog posts are likely sitting on the servers of multiple AI companies, and you have never once been paid for it.

On July 1, Cloudflare announced a date that could change that math. The deadline is September 15, 2026.

Background: Cloudflare is a US-based global ICT company that speeds up websites and protects servers from cyberattacks. More than 20% of the world's websites use its service, which functions as a reverse proxy — a "shield" sitting between users and origin servers.

On September 15, the crawlers get split apart

Cloudflare's new policy requires AI companies to clearly separate their search-indexing crawlers from their AI training and agent crawlers. Any crawler that isn't separated out by the deadline will be blocked by default on publisher sites that run ads. The policy applies to new Cloudflare sign-ups, new sites from existing customers, and the entire free-tier user base. Given how much of the world's web traffic passes through Cloudflare's infrastructure, the reach of this policy is broader than it might first appear.

Until now, most AI training crawlers have used the same or similar identifiers as search-indexing bots. For publishers, it has been nearly impossible to tell which bot was crawling for search visibility and which was harvesting data to train a large language model. Unless a publisher explicitly blocked a specific crawler identifier in robots.txt, the door was effectively open to almost everyone. Cloudflare is using that very opacity as the basis for its policy: only identifiable crawlers get through, and everything else gets blocked.

Alongside this, Cloudflare has proposed a "Pay Per Use" revenue model. The AI search services Ceramic.ai and You.com are joining as early partners, and publishers will receive a share of revenue whenever their content is actually used in AI search results or accessed through premium tiers. In effect, Cloudflare is positioning itself as more than a CDN provider — as a broker for content transactions. Cloudflare has also disclosed data showing that Google has access to roughly twice as much content as other AI companies, an implicit invitation for publishers to judge for themselves whether today's playing field is actually fair.

Why Google isn't on board with this framework

Google is the company that has publicly pushed back on this policy. Google says its "Google-Extended" bot already gives publishers a separate opt-out for AI training, and that opting out doesn't affect a site's visibility in search results. Its argument: it already operates separated mechanisms, so Cloudflare's unilateral deadline unfairly lumps it in with everyone else.

There's a reason this pushback is hard to dismiss as mere defensiveness. For Cloudflare's policy to actually work, AI companies need to follow the classification standard Cloudflare has set. But if a major player like Google can claim it already has its own separate standard and stay outside this framework, the policy's real effect on the biggest data collector is limited. A publisher-protection policy that doesn't cover the largest participant risks becoming just that — a policy with a hole in the middle.

Cloudflare's own business context matters here too. If content-distribution deals get struck on top of Cloudflare's infrastructure, Cloudflare collects a new brokerage fee. This is a policy where "protecting publishers" and "testing a new revenue model" operate at the same time. I don't think these two motives necessarily conflict. But it's worth keeping in mind that Cloudflare siding with publishers isn't purely a matter of goodwill.

What solo Korean publishers should check before September 15

There's a long-standing observation in competitive strategy: businesses that accurately recognize how valuable their own assets are, and are willing to pay a deliberate cost to protect that value, tend to survive longer at the negotiating table. On the other side are those who try to save on the cost of defending a position and end up losing the position itself. The fact that your content is being repeatedly scraped by AI companies is itself a signal that it's valuable. Whether you receive that signal passively or act on it is a separate choice.

Many Korean newsletter publishers, bloggers, and content directors either don't use Cloudflare at all, or use it without ever having checked their AI-crawler settings.

If your site runs on Cloudflare, you can find the "AI Scrapers and Crawlers" option under the security menu in your dashboard. This feature has been available on the free tier since the second half of 2024, and a single toggle lets you block major AI training crawlers all at once. After September 15, unseparated crawlers will be blocked by default anyway — but checking your settings now is the more proactive move.

Even without Cloudflare, you can use robots.txt. The identifiers for major crawlers — GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), and others — are all publicly known, and adding them to a block list doesn't require any technical expertise. This isn't a complete defense: there's no technical way to stop a crawler that simply ignores robots.txt, and newly launched crawlers won't be caught by an existing block list. But there's a meaningful difference between having taken no action at all and having explicitly stated your position — a difference that matters if and when a negotiation eventually comes.

It's unlikely that the Pay Per Use model will become a realistic revenue source for small Korean publishers anytime soon. Ceramic.ai and You.com are services built around English-language markets, and it will take time before Cloudflare's revenue-sharing network meaningfully covers Korean-language content. But without making some attempt to put a price on your own content, it's hard to stop the market from settling on a price of zero.

If you're someone whose writing keeps getting scraped by AI again and again, this is a good moment to ask yourself a question: is it okay if my writing is used to train AI, and if so, under what conditions? If you don't decide the answer in advance, someone else will eventually decide it for you.

September 15 is a deadline set by a technology policy. But the question it raises goes well beyond a settings menu. What is your content worth right now?