Cloudflare recently announced a significant update to its bot management system, which will take effect on September 15, 2026. The new update offers greater control granularity over AI traffic, but it contains a critical pitfall for SEO professionals and website administrators: the new default settings, or blanket blocking of “Training” bots, could lead to the complete and unintended blocking of multi-purpose crawlers such as Googlebot and BingBot.
In fact, incorrect blocking of search engine crawlers in Cloudflare has happened in the past as well, but the current update makes the risk more tangible than ever for anyone who does not ensure that their security settings are properly configured.
What Was the Situation in the Past?
Until now, Cloudflare offered a relatively binary solution under a dedicated “Block AI bots” button. This tool was originally designed to block single-purpose bots that crawled websites solely for data harvesting (Scraping) and training language models (LLMs), without providing any added value to the website, such as referring traffic.
Cloudflare’s September 2026 Update: The Multi-Purpose Bot Trap
Starting in mid-September, Cloudflare is moving away from the blanket approach and switching to a four-part model that includes separate control over three types of bots:
- Search: Classic bots whose purpose is to direct visitors to the website (allowed by default).
- Agent: Bots that perform tasks on behalf of users (blocked by default on new domains that display advertisements).
- Training: Crawlers that collect data for model training (blocked by default on new domains that display advertisements).
This is where the problem lies from an SEO perspective: The technological reality today is complex, and major bots such as Googlebot, Applebot, and BingBot are multi-purpose crawlers – they crawl both for traditional search engines and to collect data for their companies’ model training (such as Gemini and Copilot).
According to Cloudflare’s new policy, these multi-purpose crawlers will be evaluated according to the strictest rule. This means that if you have configured your site to block “Training bots” – or if the system has blocked them for you by default (Default) – your website will be completely blocked to Googlebot and Bingbot, including for their regular search functionality.
The Impact on Crawling, Indexing, and GEO
From the perspective of website crawling and reading, the implications are broad and affect two main areas:
- Classic SEO: Blocking at the CDN/DNS level prevents the bot from reaching the server and rendering the DOM (website code). The direct result is damage to the Crawl Budget and website indexing.
- GEO (Generative Engine Optimization): In an era in which we are working to make our content accessible to AI-based answer engines (such as ChatGPT or Perplexity), uncontrolled blocking of Agent and Training bots could be a self-inflicted wound. Such blocking will make your brand disappear from AI systems that retrieve live and up-to-date information from the web in real time in order to build their answers (a technology known as RAG). The new, targeted filtering capability offered by Cloudflare requires us to make a strategic decision: which AI bots are we willing to allow access to our website in order to ensure that our content continues to appear and stand out on these new platforms.
Action Items: What Should You Do Now?
Ahead of September 15, it is essential to ensure that your digital assets are protected from unintended blocking:
- How Can You Tell Whether Your Website Is Hosted on Cloudflare? If you are unsure, you can easily check this by examining the domain’s Name Servers (using free Whois tools) or by examining the server’s HTTP Headers using the browser’s developer tools (look for values such as
Server: cloudflare). - Check the Settings (For Those With Cloudflare Access): Log in to the Settings area (Security) and navigate to the new Bot Management options. Explicitly make sure that you opt out of the stricter defaults for Training crawlers that also crawl for search purposes, in order to prevent Googlebot and BingBot from being blocked.
- Contact the Technical Team: Our recommendation for anyone who does not manage the server infrastructure themselves is to contact the website developer or hosting company as soon as possible. Ask them to verify that the new AI Traffic settings in Cloudflare are not blocking, even by default, search bots or multi-purpose bots that you rely on.
- Continuous Monitoring in Google Search Console: As an additional indicator, it is recommended to closely monitor the “Crawl Stats” report in GSC throughout September. A sharp drop in crawl requests or an increase in 403/5xx errors could indicate that the bot is being blocked at the Cloudflare level.
Our Recommendation: Do not leave your crawling strategy to the default settings of external systems. Make sure that your website’s availability aligns with your SEO goals, both in traditional search and in the new AI engines.
