Cloudflare’s 15 September changes classify crawlers by Search, Agent and Training behaviour. Because some crawlers are multi-purpose, blocking training activity can also block a crawler involved in Search.
Cloudflare’s 15 September changes classify crawlers by Search, Agent and Training behaviour. Because some crawlers are multi-purpose, blocking training activity can also block a crawler involved in Search.