I Deleted 12 Dead URLs From My Sitemap. Here's Why Google Punishes You for Them.

You have redesigned your site at least once. Pages got merged, routes got cut, old work got pruned. Quick question: did your sitemap get the memo?

Mine didn't.

In June I ran a full audit on webight.com, my own studio's site. The sitemap listed 21 URLs. 12 of them were case-study pages under /work/ that I had deleted from the codebase 5 weeks earlier. Every one of those 12 returned a 404. Two more URLs, /privacy and /terms, loaded fine but carried a noindex tag. So out of 21 URLs I was handing Google, 14 were either dead or telling Google to go away.

I build sites for clients in India, the Gulf, the US, Southeast Asia, and the Netherlands, and this was sitting on my own domain. Embarrassing. Also useful, because the way this bug got in is the exact way it gets into almost every Next.js project I look at.

One big idea today: every URL in your sitemap is a claim. "This page exists, it returns 200, it is the canonical version, and I want it indexed." A sitemap with 12 dead URLs is you making 12 false claims to the one crawler whose trust decides your traffic.

What a dirty sitemap actually costs you

I'll be precise here, because SEO content loves to scare you into buying an audit. Nobody at Google flips a penalty switch because your sitemap has 404s in it. The cost is quieter, and it lands in 3 places.

Crawl budget first. Googlebot reads your sitemap, queues the URLs, and fetches them. A 404 answer doesn't end it either, because URLs sometimes come back, so Googlebot retries dead URLs for a long time. Every retry is a fetch that could have gone to a page you actually care about. On a 7-page studio site this is small. On a 5,000-URL store it's the difference between your new products getting crawled today or next month.

Second, the file itself loses weight. Google's documentation is blunt about what belongs in a sitemap: canonical URLs that return 200 and are indexable. Google also cross-checks what you declare against what it finds. The lastModified field is the documented example: if your dates keep claiming changes that didn't happen, Google starts ignoring the field. Same logic applies to the whole file. Feed it junk for months and your sitemap stops functioning as a discovery channel, which was its entire job.

Third, Search Console turns into noise. Every dead entry shows up as "Submitted URL not found (404)". Mine had a wall of them. And when 12 fake errors sit in that report, the one real error you needed to see this week is buried under them. That's how real problems survive: hidden inside reports you stopped reading because they cried wolf too long.

How 12 dead URLs got into my sitemap

The case studies used to live at /work/[slug], one page per project. During a redesign I collapsed all of them into a single /work page and deleted the route file. The build passed. TypeScript said nothing. Every internal link got updated.

And sitemap.ts kept serving the old list, because nothing in Next.js connects your sitemap to your routes. sitemap.ts is just a function that returns whatever you wrote. What I had written was hardcoded:

// src/app/sitemap.ts, the version that lied
import type { MetadataRoute } from "next";

export default function sitemap(): MetadataRoute.Sitemap {
  const baseUrl = "https://www.webight.com";

  // These pages were deleted from the codebase weeks before
  // this file was touched again.
  const caseStudySlugs = [
    "boutique-hotel-booking-bot",
    "zulal-performance-rebuild",
    // ...10 more slugs, every single one a 404
  ];

  const caseStudyRoutes: MetadataRoute.Sitemap = caseStudySlugs.map((slug) => ({
    url: `${baseUrl}/work/${slug}`,
    lastModified: new Date(),
    changeFrequency: "monthly",
    priority: 0.7,
  }));

  return [
    { url: baseUrl, lastModified: new Date(), changeFrequency: "weekly", priority: 1.0 },
    { url: `${baseUrl}/about`, lastModified: new Date(), changeFrequency: "monthly", priority: 0.8 },
    { url: `${baseUrl}/services`, lastModified: new Date(), changeFrequency: "monthly", priority: 0.9 },
    { url: `${baseUrl}/work`, lastModified: new Date(), changeFrequency: "weekly", priority: 0.8 },
    { url: `${baseUrl}/pricing`, lastModified: new Date(), changeFrequency: "monthly", priority: 0.8 },
    { url: `${baseUrl}/contact`, lastModified: new Date(), changeFrequency: "monthly", priority: 0.7 },
    { url: `${baseUrl}/blog`, lastModified: new Date(), changeFrequency: "weekly", priority: 0.5 },
    { url: `${baseUrl}/privacy`, lastModified: new Date(), changeFrequency: "yearly", priority: 0.3 },
    { url: `${baseUrl}/terms`, lastModified: new Date(), changeFrequency: "yearly", priority: 0.3 },
    ...caseStudyRoutes,
  ];
}

Deleting the route file threw no error because no code imported it. The sitemap threw no error because it's a list of strings, and strings don't 404 at build time. They 404 in production, silently, for Googlebot only, while every human visitor sees a perfectly healthy website.

See, anything maintained by memory will drift. That's an engineering principle older than the web: keep two copies of one truth and they will disagree eventually. My route files were one copy of "which pages exist". My sitemap was a second copy, maintained by hand. The drift was guaranteed the day I wrote it. The only open question was when.

The noindex contradiction

The 2 legal pages were a different bug in the same file. /privacy and /terms are live, but both carry robots: { index: false } in their metadata, on purpose. Boilerplate legal text that looks like ten thousand other privacy pages has no business ranking, and I'd rather Google spend its attention on pages that earn money.

But they were still listed in the sitemap. Which means I was saying two things at once. The sitemap entry says "index this, I put it on the list for you." The meta tag on the page says "do not index this." Google obeys the noindex every time, so no ranking harm done. The crawler still has to fetch the page to discover the contradiction though, and Search Console files each one under "Submitted URL marked 'noindex'", one more warning burying real signal.

The rule is simple. A URL is either worth indexing, in which case it goes in the sitemap and has no noindex, or it isn't, in which case it gets the noindex and stays out of the sitemap. There is no third state. If you find a URL on both lists, you haven't decided yet, and Google hates pages you haven't decided about.

The fix

Here's the file after the audit, complete, as it runs on webight.com today:

// src/app/sitemap.ts
import type { MetadataRoute } from "next";

export default function sitemap(): MetadataRoute.Sitemap {
  const baseUrl = "https://www.webight.com";

  // Only indexable, existing routes belong here. /privacy and /terms are
  // noindexed and /work/[slug] pages no longer exist, so they are excluded.
  const staticRoutes: MetadataRoute.Sitemap = [
    { url: baseUrl, lastModified: new Date(), changeFrequency: "weekly", priority: 1.0 },
    { url: `${baseUrl}/about`, lastModified: new Date(), changeFrequency: "monthly", priority: 0.8 },
    { url: `${baseUrl}/services`, lastModified: new Date(), changeFrequency: "monthly", priority: 0.9 },
    { url: `${baseUrl}/work`, lastModified: new Date(), changeFrequency: "weekly", priority: 0.8 },
    { url: `${baseUrl}/pricing`, lastModified: new Date(), changeFrequency: "monthly", priority: 0.8 },
    { url: `${baseUrl}/contact`, lastModified: new Date(), changeFrequency: "monthly", priority: 0.7 },
    { url: `${baseUrl}/blog`, lastModified: new Date(), changeFrequency: "weekly", priority: 0.5 },
  ];

  return staticRoutes;
}

21 entries down to 7. Every remaining URL returns 200 and wants to be indexed. And notice the comment in the file. Write the why down where the next person (probably you, in 8 months) will edit the list, or the legal pages will quietly creep back in.

For the deleted case studies that's the whole fix, because the pages are gone for good. If your detail pages still exist and render from a data file, do one better and derive the sitemap from the same source the pages render from:

import type { MetadataRoute } from "next";
import { caseStudies } from "@/lib/case-studies";

export default function sitemap(): MetadataRoute.Sitemap {
  const baseUrl = "https://www.webight.com";

  return [
    { url: baseUrl, lastModified: new Date(), priority: 1.0 },
    { url: `${baseUrl}/work`, lastModified: new Date(), priority: 0.8 },
    ...caseStudies.map((cs) => ({
      url: `${baseUrl}/work/${cs.slug}`,
      lastModified: cs.updatedAt,
    })),
  ];
}

Now deleting a project from the array removes the page and the sitemap entry in the same commit. Drift becomes impossible instead of inevitable. One source of truth, that's it.

Test your own sitemap in 90 seconds

You don't need a crawler subscription for this. Paste this into a terminal:

curl -s https://www.yourdomain.com/sitemap.xml \
  | grep -o "<loc>[^<]*</loc>" \
  | sed -e "s|<loc>||" -e "s|</loc>||" \
  | while read -r url; do
      code=$(curl -s -o /dev/null -w "%{http_code}" "$url")
      echo "$code  $url"
    done

It pulls every URL out of your sitemap and prints the status code next to it. You want a column of 200s. Anything else gets handled like this:

• 404: the page is gone. Remove the entry, or restore the page if deleting it was the mistake.
• 301 or 308: the entry points at an old address. Update it to the final URL, because sitemaps should never make Googlebot follow redirects.
• 200: one more check. Run curl -s "$url" | grep -i 'name="robots"' and make sure nothing says noindex. If it does, pick a side.

Then go to Search Console, open Sitemaps, and resubmit. The fake errors drain out of the Pages report over the following weeks, and what's left in that report is real.

For the record, the sitemap cleanup was one item in the June audit that also fixed a canonical mismatch, killed a 4,190ms hero render delay, and removed 22.5 KiB of render-blocking CSS. Mobile scores went from 90, 96, 96, 100 to 96, 100, 100, 100. The sitemap fix contributed exactly zero of those points, because Lighthouse never reads sitemap.xml. No tool you run casually does. That is precisely why this bug lives for months on sites whose owners check their scores every week.

So open yourdomain.com/sitemap.xml right now and read it line by line, or run the 7-line script if the file is long. If every URL is alive and indexable, good, you spent 90 seconds buying certainty. If you find a dead one, you've been feeding Google false claims since the day you deleted that page, and now you know how to stop before lunch.

Found dead URLs in yours? Tell me how many, I read every reply.