What are canonical tags, robots.txt and XML sitemaps, and how do they differ?

Three technical controls that are frequently confused, and confusing them causes serious problems.

• robots.txt — controls crawling. It tells crawlers which paths not to fetch. Critically, it does not control indexing: a blocked URL can still appear in results if other pages link to it, showing without a description. Never use robots.txt to keep a page out of the index — because blocking the crawl means Google cannot see the noindex tag either.
• noindex meta tag — controls indexing. This is the correct way to keep a page out of results. The page must remain crawlable for it to work.
• Canonical tag — handles duplication. It tells Google which version of similar pages is the preferred one, consolidating ranking signals onto it. Essential for e-commerce filters and parameters, printer-friendly versions, and pages reachable by multiple URLs. It is a hint rather than a directive; Google may choose differently.
• XML sitemap — aids discovery. A list of URLs you want indexed, submitted through Search Console. Useful for large sites, new sites, and pages with few internal links. It only includes canonical, indexable URLs — listing a noindexed page sends contradictory signals.

Note: The classic mistake is blocking a directory in robots.txt to remove it from search, then finding the URLs still listed. Say that, and it demonstrates you understand the crawl-index distinction rather than reciting definitions.