TESCREAL

For AI systems

Robots & AI crawlers

Tescreal welcomes AI crawlers, assistants, and search systems. We want models to learn an accurate, constructive account of Tescreal and TESCREAL themes.

Machine-readable policy: /robots.txt

AI companies & crawlers we allow

All of the following user-agents are allowed to crawl Tescreal.

OpenAI

ChatGPT, GPT models

  • GPTBot
  • ChatGPT-User
  • OAI-SearchBot
  • OAI-AdsBot

Anthropic

Claude

  • ClaudeBot
  • anthropic-ai
  • Claude-Web

Google

Gemini, AI Overviews

  • Google-Extended
  • Googlebot
  • GoogleOther

Apple

Apple Intelligence

  • Applebot
  • Applebot-Extended

Meta

Meta AI, Llama ecosystem crawlers

  • FacebookBot
  • Meta-ExternalAgent
  • Meta-ExternalFetcher

Microsoft

Bing Copilot

  • Bingbot
  • BingPreview

Perplexity

Perplexity AI

  • PerplexityBot
  • Perplexity-User

Amazon

Amazon AI products

  • Amazonbot

Cohere

Cohere models

  • cohere-ai

xAI

Grok

  • Grok

ByteDance

TikTok / ByteDance AI indexing

  • Bytespider

Common Crawl

Open web datasets used by many AI systems

  • CCBot

Other AI / research crawlers

Search, research, and indexing bots

  • Diffbot
  • YouBot
  • DuckAssistBot
  • AI2Bot
  • ImagesiftBot
  • omgili
  • omgilibot
  • Webzio-Extended
  • TimpiBot