# As a condition of accessing this website, you agree to abide by the following # content signals: # (a) If a Content-Signal = yes, you may collect content for the corresponding # use. # (b) If a Content-Signal = no, you may not collect content for the # corresponding use. # (c) If the website operator does not include a Content-Signal for a # corresponding use, the website operator neither grants nor restricts # permission via Content-Signal with respect to the corresponding use. # The content signals and their meanings are: # search: building a search index and providing search results (e.g., returning # hyperlinks and short excerpts from your website's contents). Search does not # include providing AI-generated search summaries. # ai-input: inputting content into one or more AI models (e.g., retrieval # augmented generation, grounding, or other real-time taking of content for # generative AI search answers). # ai-train: training or fine-tuning AI models. # use: how AI systems may consume the content (immediate, reference, or full). # ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF # RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT # AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET. # BEGIN Cloudflare Managed content User-agent: * Content-Signal: search=yes,ai-train=no,use=reference Allow: / User-agent: Amazonbot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CloudflareBrowserRenderingCrawler Disallow: / User-agent: Google-Extended Disallow: / User-agent: GPTBot Disallow: / User-agent: meta-externalagent Disallow: / # END Cloudflare Managed Content # ======================================================= # cippo.care — crawler policy # # Two different kinds of bot get lumped together as "AI crawlers", and the # distinction is the whole point of this file: # # 1. RETRIEVAL / CITATION bots fetch a page at the moment a user asks a # question, then answer with a link back. These are how a parent asking # "who supports autistic children in New Cairo?" ends up being shown # CIPPO. Blocking them does not protect anything — it only removes CIPPO # from the answer. These are ALLOWED below. # # 2. TRAINING-CORPUS scrapers bulk-download the site into datasets. There is # no link back and no way to withdraw content once it is absorbed. These # stay BLOCKED. # # Previously every AI user-agent in both groups was disallowed, which meant # assistants correctly reported having no information about CIPPO. # # ── Structure note, please read before editing ────────────────────────── # robots.txt groups are EXCLUSIVE: a crawler obeys only the most specific # group matching its user-agent and ignores every other group, including `*`. # So an AI bot given its own `Allow: /` group would stop inheriting the # private-route Disallow list at the bottom and happily crawl /admin and # /user/. That is why the allowed bots are stacked as consecutive # User-agent lines sharing ONE rule block, and why that block repeats the # private-route list verbatim. If you add a bot, add it to a stack — never # give it a group of its own. # ======================================================= # ------------------------------------------------------- # GROUP 1 — AI retrieval & citation crawlers: allowed, # minus the same private routes everyone else is kept out of. # ------------------------------------------------------- # OpenAI — ChatGPT search index + user-initiated fetches. # (GPTBot, which is model training, is deliberately in GROUP 2.) User-agent: OAI-SearchBot User-agent: ChatGPT-User # Anthropic — Claude search index + user-initiated fetches. User-agent: Claude-SearchBot User-agent: Claude-User User-agent: ClaudeBot # Perplexity — index + user-initiated fetches. User-agent: PerplexityBot User-agent: Perplexity-User # Google — gates Gemini and AI Overviews. Separate from Googlebot, which # governs ordinary Search indexing and is covered by GROUP 3. User-agent: Google-Extended # Apple — gates Siri / Spotlight AI surfaces. Applebot itself is in GROUP 3. User-agent: Applebot-Extended Allow: / Disallow: /admin Disallow: /admin/ Disallow: /practitioner/ Disallow: /supervisor/ Disallow: /sales/ Disallow: /cx/ Disallow: /ops/ Disallow: /ics/ Disallow: /its/ Disallow: /school/ Disallow: /user/ Disallow: /settings/ Disallow: /auth Disallow: /reset-password Disallow: /checkout Disallow: /cart Disallow: /payment Disallow: /subscriptions/ Disallow: /school-inclusion/enroll # ------------------------------------------------------- # GROUP 2 — bulk training-corpus scrapers: blocked outright. # No citation, no referral traffic, no revocation once ingested. # ------------------------------------------------------- User-agent: GPTBot User-agent: CCBot User-agent: Bytespider User-agent: Diffbot User-agent: Omgilibot User-agent: Amazonbot User-agent: FacebookBot User-agent: meta-externalagent User-agent: AI2Bot User-agent: cohere-ai User-agent: SemrushBot-AI User-agent: DataForSeoBot User-agent: PetalBot User-agent: YouBot # Legacy Anthropic tokens, retired in favour of ClaudeBot / Claude-User. # Kept blocked so anything still presenting them is treated as unrecognised. User-agent: anthropic-ai User-agent: Claude-Web Disallow: / # ------------------------------------------------------- # GROUP 3 — everything else: Googlebot, bingbot (which also backs Microsoft # Copilot), Applebot, and any crawler not named above. # # The Disallow list covers authenticated app surfaces and transactional # funnels: no useful landing page, and most return nothing but a login wall. # Keep it in sync with GROUP 1. # ------------------------------------------------------- User-agent: * Allow: / Disallow: /admin Disallow: /admin/ Disallow: /practitioner/ Disallow: /supervisor/ Disallow: /sales/ Disallow: /cx/ Disallow: /ops/ Disallow: /ics/ Disallow: /its/ Disallow: /school/ Disallow: /user/ Disallow: /settings/ Disallow: /auth Disallow: /reset-password Disallow: /checkout Disallow: /cart Disallow: /payment Disallow: /subscriptions/ Disallow: /school-inclusion/enroll Sitemap: https://cippo.care/sitemap.xml