What does robots.txt control?
- A Whether a page is indexed
- B Whether crawlers fetch a URL
- C Whether a page ranks
- D Whether a page is cached
Answer
Whether crawlers fetch a URL
robots.txt controls crawling only. A blocked URL can still appear in results if other pages link to it, because Google cannot fetch the page to see a noindex tag.





