Robots.txt — Definition & Meaning
What is Robots.txt?
robots.txt is a plain-text file at your domain root (yoursite.com/robots.txt) that tells crawlers which URLs they can and cannot request. It controls crawl budget, blocks staging paths, and points crawlers at your sitemap — but it does not prevent indexation.
Robots.txt — Definition & Meaning is part of LemRank, the rank intelligence platform for SEO agencies and in-house teams: daily rank tracking, SERP change detection, AI Overview and LLM visibility, competitor gap analysis, site audits and white-label client reporting.