Home › Knowledge base › AI crawlers in detail

Bytespider by ByteDance

Bytespider is the crawler of ByteDance (TikTok, Doubao) and one of the most active AI bots on the web.

Reputation and behaviour

Bytespider has in the past stood out for high crawl frequency and inconsistent robots.txt compliance. For many operators it is a candidate for blocking if there is no interest in the Chinese market.

Blocking it

"User-agent: Bytespider" followed by "Disallow: /". Because compliance is not guaranteed, an additional WAF or firewall rule on the user-agent string is advisable.

When to allow it

If you want visibility in the Chinese-speaking market — for example via Doubao — you should allow Bytespider and instead limit server load with rate limits.

Checking access

citeglass fetches your page with the Bytespider user agent and shows whether the robots.txt rule and the server response match.

Check your own site

citeglass shows in about 30 seconds where your website is unreadable for AI systems — free and without login.

Read on

• CCBot and Common Crawl
• Block all AI crawlers
• Allow or block GPTBot

General information, not legal advice. GEO is a young field — recommendations may change.