Verified record · from the answer index

robots-txt

When the user wants to configure, audit, or optimize robots.txt. Also use when the user mentions "robots.txt," "crawler rules," "block crawlers," "AI crawlers," "GPTBot," "allow/disallow," "disallow path," "crawl directives," "user-agent,"

skill:kostja94/marketing-skills/robots-txt · published by kostja94 · source repository