robots, sitemap and llms.txt

A crawler asks for /robots.txt and /sitemap.xml before it reads a page, and a model asks for /llms.txt. Each comes from a file beside the root layout, named for what it answers:

The same names and shapes as Next, so a port copies them across unchanged.

import type { MetadataRoute } from '@rsc-kit/core/metadata';

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: '*', allow: '/', disallow: ['/api/', '/studio'] },
      { userAgent: 'GPTBot', disallow: '/' },
    ],
    sitemap: '/sitemap.xml',
  };
}

import type { MetadataRoute } from '@rsc-kit/core/metadata';
import { db } from '@/lib/db';

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const posts = await db.post.findMany({ select: { slug: true, updatedAt: true } });

  return [
    { url: '/', changeFrequency: 'weekly', priority: 1 },
    { url: '/pricing' },
    ...posts.map((post) => ({ url: `/blog/${post.slug}`, lastModified: post.updatedAt })),
  ];
}

import type { MetadataRoute } from '@rsc-kit/core/metadata';

export default function llms(): MetadataRoute.Llms {
  return {
    title: 'Example',
    summary: 'What the site is, in one sentence.',
    sections: [
      { title: 'Pages', links: [{ title: 'Pricing', url: '/pricing', description: 'Per restoration, no subscription' }] },
    ],
  };
}

A relative url in any of them is made absolute with the root layout’s metadataBase; without one, a relative url is a build error that says so. Any of the three may return a string instead, served as written.

A sitemap you do not write

With no sitemap.ts, the build writes /sitemap.xml itself, from what it already knows: every page it stored and every url generateStaticParams listed, each with the build as lastModified. Left out: anything under a middleware.ts (a guard means not for everyone), a page the build could not render, and the not-found page. The build’s table says so:

○  /sitemap.xml
   written by the build: 5 urls; a sitemap.ts beside the root layout replaces it

It needs the root layout’s metadataBase for the host; without one the line says it was not written. A sitemap.ts replaces it entirely — the moment you want a page the build cannot see (a post per database row without generateStaticParams) or a changeFrequency, write one.

How fresh

Three choices, and the function decides — the same rule every route follows:

import { connection } from '@rsc-kit/core/request';

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  await connection(); // per request — the same mark a page uses

  const posts = await db.post.findMany({ select: { slug: true, updatedAt: true } });

  return posts.map((post) => ({ url: `/blog/${post.slug}`, lastModified: post.updatedAt }));
}

Without the connection() line the same function runs once at build and the answer is stored; the build’s table shows ○ for a stored one and ƒ for one that runs per request. Reading the database at build is fine — it is the request that makes a route dynamic, not the data.

How they are served

Each file becomes an api route, and that decides the rest. A sitemap.ts that reads nothing per request — the database counts as nothing, the request does not — is answered once at build and stored, like any frozen route, and served from the file after. One that reads cookies() or awaits connection() stays dynamic and runs per request. The build’s table lists them with the other routes and says which.

They run no middleware, on purpose: a guard on the root layout’s directory would otherwise answer a crawler’s request for robots.txt with a 401.

The urls are typed like every other route, so route('/sitemap.xml') is a link the build checks.

Files as written

A hand-written file beside the root layout is served at the root as it is: robots.txt, sitemap.xml (or sitemap-posts.xml), llms.txt, llms-full.txt, and the others a site is asked for there — humans.txt, security.txt, ads.txt. In development they are read from app/; a build copies them beside the client output. A file and a function for the same url is a build error naming both.

Coming from Next

app/robots.ts and app/sitemap.ts carry across unchanged. app/llms.txt in Next is usually a route.ts; here it is llms.ts with a shape, or a file as written. Next’s generateSitemaps() for a sitemap split across files is not here: a sitemap-posts.xml written by hand, or a route.ts under app/sitemap/, covers it.