Add a robots.txt file

robots.txt is a tiny text file at the root of your site (yourdomain.com/robots.txt) that tells search-engine crawlers which paths they are allowed to fetch. It is also the official place to point crawlers at your sitemap. Most sites should have one even if it lets everything through.

A missing robots.txt is not catastrophic, Google will crawl your site anyway. But shipping one gives you control over what gets indexed (admin paths, search-results pages, staging URLs) and is the single best way to advertise your sitemap to all crawlers at once.

1. 1

   Decide what to block

   For most service-business sites: block admin paths, the cart/checkout, internal search-result pages, and any test directories. Allow everything else.

2. 2

   Create the robots.txt file

   Place it at the root: yourdomain.com/robots.txt. Keep it simple, the format is line-based and case-sensitive.

   # Allow all crawlers everywhere except these paths
   User-agent: *
   Disallow: /admin/
   Disallow: /cart
   Disallow: /checkout
   Disallow: /search?
   
   # Tell crawlers where the sitemap lives
   Sitemap: https://apexcarservice.com/sitemap.xml

3. 3

   Next.js: use app/robots.ts

   The App Router has a built-in convention. A robots.ts file in /app gets served at /robots.txt.

   // app/robots.ts
   import type { MetadataRoute } from 'next';
   
   export default function robots(): MetadataRoute.Robots {
     return {
       rules: [
         {
           userAgent: '*',
           allow: '/',
           disallow: ['/admin/', '/cart', '/checkout'],
         },
       ],
       sitemap: 'https://apexcarservice.com/sitemap.xml',
     };
   }

4. 4

   Avoid the classic catastrophic mistake

   Never ship this to production:

   # THIS BLOCKS YOUR ENTIRE SITE FROM GOOGLE.
   # Common bug from copying a staging robots.txt to production.
   User-agent: *
   Disallow: /

Visit https://yourdomain.com/robots.txt, the file should load and show what you wrote. In Search Console > robots.txt Tester (or the URL Inspection tool), confirm important URLs are "allowed" and only the paths you intended are blocked.

• Google: robots.txt introduction
• robots.txt specification (RFC 9309)