How our Next.js 404 page called the CMS on every request

For about a year I looked at our monitoring and saw a huge amount of 404 calls. And every time I thought: ok, we run a big e-commerce site with millions of product URLs. Products go offline all the time, crawlers keep the old links and try them again and again. So lots of 404s sounded normal to me and I never looked closer.

Turns out these were mostly not crawlers. It was our own 404 page, and it was rendered on pretty much every page request.

The setup

We run a Next.js App Router application that serves several markets. The content of the 404 page lives in our headless CMS (Payload), so the content team can change texts and links without a deployment. The `not-found.tsx` sits at the top of our main route group and renders a template, which fetches that document from the CMS:

1
2
3
4
5
6
7
8
9
10
11
// not-found.tsx
export default async function NotFound() {
  return <Default404Template />;
}

// Default404Template.tsx
export default async function Default404Template() {
  const data = await getPayloadContent('/full-content-page/404', MARKET);

  return <PayloadFullContentPageTemplate {...data} />;
}

Looks harmless. The fetch only runs when somebody lands on a page that does not exist. At least that is what I thought.

What actually happens

While working on another issue, I had to check the 404 page. Reading through the template I got curious how often this component actually renders. The answer: on every request.

The reason is how the App Router builds its component tree. For every route segment Next.js creates a `LayoutRouter` and hands it the not-found element of that segment as a prop. This is from the Next.js source (version 16):

1
2
3
4
5
6
7
8
9
// node_modules/next/dist/server/app-render/create-component-tree.js
createElement(LayoutRouter, {
  parallelRouterKey: parallelRouteKey,
  error: ErrorComponent,
  template: templateNode,
  notFound: notFoundComponent,
  forbidden: forbiddenComponent,
  unauthorized: unauthorizedComponent,
});

`LayoutRouter` is a client component. A server component that you pass as a prop to a client component gets rendered on the server and is sent along in the RSC payload. That way the client can show the 404 right away when a `notFound()` happens, without another round trip to the server. Nice for the user, but it also means our `NotFound` component ran for every page below that segment. Product page, category page, search page, it did not matter. Every single render fetched `/full-content-page/404` from the CMS, and nobody ever saw the result.

So the 404 calls in the monitoring were literally calls for the 404 page document. They looked exactly like what I expected to see, and that is why I never questioned them.

The fix: fetch the 404 page at build time

The 404 content changes maybe once or twice a year. Fetching it on every request makes no sense, so I moved it into the build.

We already had a small script that runs before every build and dev start. It fetches the main navigation for the current market from the CMS, writes it to a JSON file and generates a small accessor module next to it. I added the 404 document to the same script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
// scripts/generate-layout-config.js

async function generateNotFoundPageConfig(payloadUrl, market) {
  const notFoundPageUrl = `${payloadUrl}/api/public/${market}/full-content-page/404`;
  console.log(`Fetching 404 page from ${notFoundPageUrl}`);
  const notFoundPageConfig = await fetchJson(notFoundPageUrl);
  await writeConfigFile(`notFoundPage_${market}`, notFoundPageConfig);
}

async function generateForMarket(market) {
  const payloadUrl = getPayloadUrl(market);

  await Promise.all([
    generateFooterConfig(market),
    generateMainNavigationConfig(payloadUrl, market),
    generateNotFoundPageConfig(payloadUrl, market),
  ]);
}

The generated accessor then looks like this:

1
2
3
4
5
6
7
8
9
10
// src/generated/layout/index.ts (generated, not committed)
import notFoundPage_de from './notFoundPage_de.json';

const notFoundPage: LayoutConfigMap = {
  "de": notFoundPage_de,
};

export function getNotFoundPage(market: string): Record<string, unknown> {
  return notFoundPage[market] ?? notFoundPage['de'];
}

And the template reads the JSON instead of calling the CMS:

1
2
3
4
5
6
// Default404Template.tsx
export default function Default404Template() {
  const data = getNotFoundPage(MARKET);

  return <PayloadFullContentPageTemplate {...data} />;
}

The 404 page still renders on every request, Next.js does not care. But now it only reads a JSON file that is already in the bundle.

The trade-off

Changes to the 404 page in the CMS now only go live with the next deployment. For a page that changes this rarely that is totally fine for us.

You could also keep the runtime fetch and cache it, for example with `fetch(url, { next: { revalidate: 3600 } })` or the `'use cache'` directive. That cuts down the calls a lot as well. But you would still request the 404 document for pages nobody sees as a 404. A normal product page that answers with a 200 still asks the CMS for the 404 page, and that is pretty confusing for anyone looking at the monitoring later. That is exactly what fooled me for a year. With the build-time version these requests are gone completely, and the 404 numbers in the monitoring mean what they say again. And since we already had the build script for the navigation, it was the easier choice anyway.

Conclusion

The fix went live a few months ago and has worked great since then. The CMS now gets one request for the 404 page per build instead of one per page render, and the 404 calls are gone from the monitoring.

Lesson learned: In the App Router, `not-found.tsx` is not only rendered when there is a 404. It renders for every page in its segment, same as `forbidden.tsx` and `unauthorized.tsx` if you use them. Treat these files like a part of your layout and keep uncached fetches out of them.

And the second one: if a number in your monitoring looks strange, check once where it comes from, even if you are pretty sure you know the answer. I was sure for a year.