How many Czech websites block AI crawlers

We counted how many major Czech websites and e-shops block ChatGPT, Claude, and Perplexity. Many blocks never appear in robots.txt—the firewall or e-shop platform decides.

Orange turnstile with its arm lowered between two wall blocks, with a line of five gray robot figurines in front of it
MarketingReading time: 11 min

A quarter of the largest Czech websites keep out at least one AI bot. In our November 19, 2026 study, 208 of 819 reachable sites in the top 1,000 .cz domains (25.4%) and 284 of 1,223 e-shops (23.2%) block AI crawlers. Often only the firewall blocks them (lower estimate: a third to half of ChatGPT blocks), while robots.txt says nothing. If you want ChatGPT, Claude, or Perplexity to mention your business in answers, checking AI bots in robots.txt is not enough—your firewall or e-shop platform may decide.

How many Czech websites keep AI out: a quarter of the largest

AI bots crawl websites either to train models (GPTBot, ClaudeBot) or to answer queries with links (OAI-SearchBot, Claude-SearchBot, PerplexityBot). A robots.txt file (such as vasweb.cz/robots.txt) is a public sign on the door that well-behaved bots respect. A firewall (your host or Cloudflare) acts like a guard: it turns a bot away even when robots.txt says nothing.

We tested 1,000 .cz domains from the Tranco ranking (819 returned a page, which forms our percentage base) and separately 1,223 e-shops among 5,120 ranked .cz domains.

What we foundTop 1,000 (819 sites)E-shops (1,223)
Blocks at least one AI bot208 (25.4%)284 (23.2%)
GPTBot disallowed in robots.txt97 (11.8%)96 (7.8%)
GPTBot rejected by firewall83 (10.1%)137 (11.2%)
of which not disallowed in robots.txt4982
AI search bot disallowed in robots.txt41 (5.0%)27 (2.2%)
AI search bot rejected by firewall65 (7.9%)84 (6.9%)
llms.txt file85 (10.4%)294 (24.0%)

Firewall figures are always a lower estimate: we count only explicit rejections where a normal browser and a control request passed in the same minute. Sites disallowing GPTBot in robots.txt include idnes.cz, mapy.cz, vinted.cz, denik.cz, aktualne.cz, lidovky.cz, irozhlas.cz, and hn.cz.

Heavily protected sites returned no page at all, so they are missing from the 819-site base. Across all 807 readable robots.txt files in the top 1,000, however, 127 (15.7%) disallow GPTBot. Protected sites block more often, so our headline numbers likely understate the total.

Firewalls block AI crawlers that robots.txt never mentions

In the top 1,000 domains, 146 websites (of 819) block GPTBot: 63 in robots.txt only, 34 in both, and 49 (34%) at the firewall only (lower estimate). Among e-shops, 178 stores (of 1,223) block it, including 82 (46%) at the firewall only (lower estimate).

Bar chart of GPTBot blocking on top 1,000 websites and e-shops: robots.txt, both, firewall
GPTBot blocking: top 1,000 .cz domains (146 sites) and e-shops (178 stores). Firewall: lower estimate (Guap Media, November 19, 2026).

E-shops block GPTBot more often via firewall (137 of 1,223, lower estimate) than in robots.txt (96 of 1,223). Earlier studies read only robots.txt. Yet 191 of 819 large websites (23.3%) run through Cloudflare, which accounts for 55 of 83 firewall blocks of GPTBot on large sites and 79 of 137 on e-shops (lower estimates). Servers or hosts reject the rest.

On 29 top-1,000 sites and 42 e-shops, Cloudflare also inserts a "BEGIN Cloudflare Managed content" block into robots.txt that always disallows GPTBot. It does this when AI blocking is active in the zone settings—from the outside, we cannot tell if the owner turned it on knowingly.

On e-shops, the platform decides, not the merchant

On e-shops, the platform—not the merchant—often controls robots.txt for AI bots. On Shoptet, 259 of 291 stores include Content-Signal: 232 disallow AI training (ai-train=no), and 190 set exactly "search=yes, ai-train=no".

Cloudflare's Content-Signal states whether AI may use content for search (search), answers (ai-input), or training (ai-train). Still, it is only a statement—just 4 of 291 Shoptet stores disallow GPTBot in robots.txt. Whether bots respect it is up to them.

Platforms similarly shape adoption of llms.txt, which summarizes site content for AI models. We found it on 294 of 1,223 e-shops (24.0%), compared with just 85 of 819 large websites (10.4%):

  • Shopify: 33 of 37 (89%)
  • Shoptet: 86 of 291 (30%)
  • Custom builds: 139 of 670 (21%)
  • PrestaShop: 8 of 87
  • WooCommerce: 7 of 48

We discuss whether llms.txt pays off in our article on AI search.

Bar chart: share of e-shops with a Content-Signal line and an llms.txt file by platform
Content-Signal line in robots.txt and llms.txt file on e-shops by platform (platforms with at least 30 stores). Source: Guap Media study, November 19, 2026.

Your platform or host also determines firewall rejections of GPTBot (lower estimate):

  • PrestaShop: 26 of 87
  • Custom builds: 86 of 670
  • Shoptet: 4 of 291
  • Shopify: 3 of 37

Running an e-shop on a hosted platform? Check robots.txt to see what it configured for you. We compare setup options in our e-shop platform comparison.

Robots.txt rarely disallows AI search bots, firewalls do it more often

Only 41 of 819 large websites (5.0%) and 27 of 1,223 e-shops (2.2%) disallow search bots for ChatGPT, Claude, and Perplexity in robots.txt. At the firewall, however, at least 65 large websites (7.9%) and 84 e-shops (6.9%) reject OAI-SearchBot or PerplexityBot—a lower estimate in both cases.

Providers separate their bots like this:

  • OpenAI: GPTBot gathers content for potential training. According to OpenAI's documentation, disallowing OAI-SearchBot removes a site from ChatGPT search; OpenAI applies changes within roughly 24 hours.
  • Anthropic: ClaudeBot collects content for training, Claude-SearchBot handles search, and Claude-User loads pages for users. According to Anthropic, blocking the latter two can reduce visibility.
  • Perplexity: According to Perplexity, PerplexityBot powers search and is not used to train models.

In robots.txt, news and classifieds sites disallow search bots most often, including idnes.cz, denik.cz, zive.cz, ceskenoviny.cz, and sreality.cz. These are public lines in their robots.txt files.

Across the 819 reachable sites, firewalls most often reject ClaudeBot (92), GPTBot (83), PerplexityBot (59), and OAI-SearchBot (14)—each a lower estimate. Blanket AI blocking can therefore drop a site from cited answers even when the owner only meant to block training.

Compared with 2024: 97 sites disallow GPTBot today, Apify found 21

Today, 97 of the 1,000 Tranco .cz domains disallow GPTBot in robots.txt (11.8% of 819 reachable sites, 9.7% overall). In 2024, Apify found 21 using a different traffic-based ranking; Czech media outlet Lupa.cz reported "a mere three percent" on September 2, 2024. That is four to five times as many, though from different website lists. Apify also measured before Cloudflare's managed robots.txt, which now blocks 29 sites.

According to Czech media outlet Britské listy, Piperic scanned the entire Czech web in July 2026 and found any AI bot disallowed on 2.9% of 224,452 websites and 1.9% of 28,103 e-shops, excluding firewalls. Among the largest sites, 152 of 819 websites (18.6%) and 177 of 1,223 e-shops (14.5%) disallow at least one training bot in robots.txt. The two figures measure different things.

StudyWhenWebsitesWhat it measuredResult
Apify (via Lupa.cz)September 20241,000 most visitedrobots.txtGPTBot disallowed on 21 sites
Piperic (via Britské listy)July 2026224,452 activerobots.txt, llms.txt2.9% disallow an AI bot
Guap MediaNov 19, 2026819 of top 1,000 and 1,223 e-shopsrobots.txt, llms.txt, firewallGPTBot disallowed on 97 sites, AI bot blocked on 208

In a Studio Silex check (2026), 5 of 14 websites returned a 403 error; the authors attributed that to bot IP verification rather than AI blocking. We classify those as "indistinguishable", not as blocks.

How to check if your website lets AI bots in

You can check whether your website lets AI bots in within a few minutes in three steps:

  1. Read robots.txt. At vasweb.cz/robots.txt, look under "User-agent" for GPTBot, ClaudeBot, OAI-SearchBot, Claude-SearchBot, and PerplexityBot. "Disallow: /" beneath them blocks the whole site ("User-agent: *" with "Disallow: /" blocks all bots, including Google). External services often add "BEGIN Cloudflare Managed content" and Content-Signal lines automatically.
  2. Check your firewall. In Cloudflare, open Security → Settings → AI Crawler and Agent access policies (formerly Block AI bots). Per Cloudflare's documentation, it keeps separate rules for Search, Agent, and Training. On other hosts, ask your provider which AI bots the firewall rejects.
  3. Send a test request as a bot. Run the two commands below in your command line (Terminal on macOS, PowerShell on Windows) and compare the first response line against a request without the -A part. Code 200 means allowed; 403 or 401 means rejected.
curl -I -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" https://www.vasweb.cz/
curl -I -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" https://www.vasweb.cz/

Repeat this check after every domain move, host switch, or platform change—that is when settings shift without your knowledge. We cover what else to watch in Website redesign without losing traffic.

Training and search are two separate choices

Decide on AI bot access in two steps based on whether you want to protect your content or get AI recommendations. The result should match your choice, not a default setting:

  • Model training (GPTBot, ClaudeBot, CCBot, etc.): disallowing stops future models from training on your content. According to OpenAI, disallowing GPTBot does not affect ChatGPT search.
  • Cited search (OAI-SearchBot, Claude-SearchBot, PerplexityBot): if disallowed, these services will not find or link to you in answers.
  • User-triggered fetches (ChatGPT-User, Claude-User, Perplexity-User): these open a page at a person's request, so providers note that robots.txt may not apply to them.

Learn how to appear in AI answers in our AI search article, and review the fundamentals in What you can do for your company's SEO. We can help improve your visibility through our marketing services.

How we measured and what an outside scan cannot see

We ran our scan on November 19, 2026, from a single computer in Prague on a residential IP address. Each website received at most eight sequential requests and we bypassed nothing—22,459 requests in under 15 minutes.

Website lists. Rankings come from Tranco list L5KW4 dated November 18, 2026 (a 30-day average from five sources). It measures domain popularity rather than traffic, and the top 1,000 .cz list includes technical domains. In total, 181 of 1,000 domains returned no page (130 rejected the browser, 51 did not respond) and are excluded from shares.

E-shops. From 5,120 .cz domains, we selected sites showing a platform signature and a cart, or a cart plus at least two of five sales signals. In a manual check of 40 sites, all were stores except at most two borderline cases. This includes hosts with carts, marketplaces, and ticket sellers; it misses non-.cz stores (.com, .eu), JavaScript-rendered sites, and sites rejecting browsers. "Custom and other" also covers platforms not recognized from HTML.

What we fetched. The homepage, robots.txt, and llms.txt as a browser, then the homepage five times as GPTBot, a control Googlebot, OAI-SearchBot, ClaudeBot, and PerplexityBot.

Robots.txt. Parsed under RFC 9309. We count only full-site disallows as blocks; partial disallows and sites closed to all bots including Google are tracked separately.

Firewall. Only explicit rejections (401, 403, 406, and 451) when both the browser and Googlebot passed in the same minute count as blocks—making this a lower estimate. Including 429 and 5xx codes (which can also indicate overload), 99 of 819 large websites and 193 of 1,223 e-shops reject GPTBot. Sites rejecting both GPTBot and Googlebot fall under "indistinguishable" (23 of 819 and 45 of 1,223) and are not counted as blocks.

What an outside scan cannot see:

  • What a real bot receives. A firewall may allow verified IPs even after rejecting us, and Cloudflare may block AI bots even after letting us pass. Only server logs show the exact result.
  • Intent. One click or a default setting can turn on Cloudflare's managed robots.txt, AI bot rules, or a platform's Content-Signal line.
  • Change over time. Rules change; our scan captures one day.
  • Compliance. Robots.txt is a request and Content-Signal is a statement, not a lock.

Downloadable data. We publish only public files for all 1,900 sites in our share bases (rank, robots.txt verdicts for 22 AI bots and Googlebot, llms.txt, Content-Signal, and Cloudflare's managed block) in a CSV table. We do not publish per-site firewall data because external checks cannot verify how a site treats a real bot.

If you would like to review the AI bot settings on your website with us, tell us what you need.

Frequently asked questions

How do I check if my website blocks ChatGPT?

Open vasweb.cz/robots.txt and look for "Disallow: /" under GPTBot and OAI-SearchBot. Check your firewall too (AI Crawler and Agent access policies in Cloudflare) and send a test request with the bot's header. In our study, robots.txt said nothing on 49 of the 146 large websites that keep GPTBot out.

How do I block AI crawlers on my website?

Add "User-agent:" with the bot's name and "Disallow: /" beneath it in robots.txt. That blocks the site only for bots that follow the rules; a firewall is more reliable. Consider the bot type first: disallowing search bots removes you from cited answers, whereas disallowing training bots does not.

What are GPTBot and OAI-SearchBot?

Both belong to OpenAI (ChatGPT). GPTBot collects data to train models, and disallowing it does not affect search. OAI-SearchBot handles search—if you disallow it, your site will not appear in ChatGPT answers.

What does the Content-Signal line in robots.txt mean?

Content-Signal states whether AI may use content for search (search), answers (ai-input), or training (ai-train). Setting "ai-train=no" declines training, though it does not block the bot. Most Shoptet e-shops had this line, yet only 4 of 291 disallowed GPTBot.

More articles

Handpicked topics for you

AI search optimization: appear in Google & ChatGPT

AI Overviews, AI Mode, ChatGPT, and Seznam AI answer directly in search. See how they pick sources, what not to pay for, and what to fix in 30 days.

Reading time: 9 min

What you can do to help your business with SEO

Read our tips on boosting your search rankings and winning new customers for free

Reading time: 5 min

Choosing an e-commerce platform: Shopify, Shoptet or custom

Most new online stores only need a hosted platform. Learn how to spot when you outgrow it and when custom e-commerce is an unnecessary expense.

Reading time: 9 min