# Submitted URL marked 'noindex': find where the noindex comes from and fix it

> Your sitemap lists a URL that tells Google not to index it. Find the noindex (meta tag, X-Robots-Tag, CMS, staging leak) and fix it in Next.js or WordPress.

Source: https://seofix.ai/blog/submitted-url-marked-noindex  
Published: 2026-11-03  
Updated: 2026-11-02  
Author: SEOFix Team  
Category: Technical SEO  
Tags: Technical SEO, Guides  
Length: 2,157 words, 10 min read

"Submitted URL marked 'noindex'" means a URL you listed in a sitemap answered Google with a `noindex` rule, so Google didn't index it. Two parts of your site disagree: the sitemap says "index this", the page says "don't". Decide which one is right, then fix the other. The noindex can come from a `<meta name="robots">` tag, an `X-Robots-Tag` HTTP header, a CMS setting, or a staging configuration that reached production, and `curl` plus Google's URL Inspection will tell you which in a few minutes.

One naming detail first, because it confuses people searching for this. "Submitted URL marked 'noindex'" is the name many guides still use. Today's Page indexing report lists the reason as **URL marked 'noindex'** and tells you whether a URL was submitted through a filter at the top of the report. Choose **All submitted pages** in that dropdown and the URLs you see under "URL marked 'noindex'" are the sitemap ones this post is about. Google defines "submitted" as a URL "listed in a sitemap or sitemap index that was submitted using either the Sitemaps report or a robots.txt file on your site", even if Google also found it some other way ([Page indexing report](https://support.google.com/webmasters/answer/7440203)).

Google's documentation below was checked on 2 November 2026.

## What Google says the status means

From the [Page indexing report help](https://support.google.com/webmasters/answer/7440203):

> "When Google tried to index the page it encountered a 'noindex' directive and therefore did not index it. If you do not want this page indexed, congratulations! If you do want this page to be indexed, you should remove the 'noindex' directive."

The same page lists this exact situation as a cause of error spikes: "you might have submitted a sitemap that includes URLs that are blocked for crawling by robots.txt, noindex, or a login requirement."

Google honours a noindex sent either way. Its [noindex documentation](https://developers.google.com/search/docs/crawling-indexing/block-indexing) says the rule can be a `<meta>` tag or an `X-Robots-Tag` response header, that "they have the same effect", and that a noindex inside robots.txt "is not supported by Google". When Google crawls a page and sees one, it "will drop that page entirely from Google Search results, regardless of whether other sites link to it."

So there are two possible outcomes, and only one involves the page:

| You want the page in Google? | What's wrong | Fix |
| --- | --- | --- |
| Yes | The noindex | Remove it at its source, then ask Google to recrawl |
| No | The sitemap | Drop the URL from the sitemap; the noindex is doing its job |

The second row is common and harmless. Thank-you pages, internal search results and tag archives often end up in auto-generated sitemaps. Take them out and the status stops appearing for them.

## Step 1: confirm it's still there, and where

Start with what Google saw, then what your server sends now.

### URL Inspection

Click the inspect icon next to an example URL in the report. Under **Coverage \> Indexing \> Indexing allowed?** Google shows the noindex it found on its last crawl. Then click **Test live URL** and look at **Availability \> Indexing \> Indexing allowed?** ([Page indexing report](https://support.google.com/webmasters/answer/7440203)). If the live test says indexing is allowed, the noindex is gone and you only need Google to recrawl (Step 4). If it's still there, keep going.

The live test is also the only check that uses Googlebot itself. Use **View tested page** to see the rendered HTML Google got, which matters when JavaScript adds the tag (below).

### curl the header and the HTML

The noindex is in one of two places. Check both:

```bash
URL="https://example.com/jobs/data-analyst-dubai"

# 1. The HTTP header
curl -sI -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "$URL" \
  | grep -i '^x-robots-tag'

# 2. The robots meta tags in the HTML (robots and googlebot)
curl -s -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "$URL" \
  | grep -io '<meta[^>]*name=["'"'"']\(robots\|googlebot\)["'"'"'][^>]*>'
```

Run it with a normal browser user agent too. If the results differ, something on your server or CDN is varying the response by user agent, and that's your lead. Remember that sending Googlebot's user agent from your laptop isn't Googlebot: IP-based bot rules won't treat you the same way. URL Inspection settles those cases.

Look for both `name="robots"` and `name="googlebot"`. Google supports both tokens in the robots meta tag ([robots meta tag spec](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag)), and a `googlebot`-only noindex is easy to miss when you search the source for "robots". The same spec says that when rules conflict, "the more restrictive rule applies", so `index` in one tag doesn't cancel `noindex` in another.

### Check the raw HTML against the rendered HTML

If `curl` finds nothing but URL Inspection still reports a noindex, JavaScript is adding it after load. Google's own JavaScript guide shows this pattern as a way to keep error pages out of the index: a script that appends `<meta name="robots" content="noindex">` when a product doesn't exist ([JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics)). If your app does that for "not found" states, a slow or failing API call during Google's render can mark a real page as not found.

## Step 2: find the source

Once you know whether it's a header or a tag, the list of suspects is short.

### A staging or preview setting that reached production

This cause can deindex a whole site. Teams add a noindex to staging so it can't compete with production, then the switch controlling it is wrong in production: an environment variable copied from staging, a build that ran with the wrong env file, a config flag defaulting to "noindex" when it's unset.

On Vercel, the platform adds `X-Robots-Tag: noindex` to every preview deployment from a non-production branch, and to the previous production deployment when you promote a new one ([Vercel KB](https://vercel.com/kb/guide/are-vercel-preview-deployment-indexed-by-search-engines)). That's fine. The problems start when a team adds its own rule and inverts the condition. Vercel's own example only adds the header when `VERCEL_ENV` is `preview`:

```javascript
// next.config.js: noindex only on preview builds
module.exports = {
  async headers() {
    if (process.env.VERCEL_ENV !== 'preview') {
      return []
    }
    return [
      {
        source: '/:path*',
        headers: [{ key: 'X-Robots-Tag', value: 'noindex' }],
      },
    ]
  },
}
```

Compare that with a check written as `process.env.NODE_ENV !== 'production'` or `!process.env.ALLOW_INDEXING`. Those turn the noindex on in production whenever the variable is missing. Write the condition so that a missing variable means "index": the safe default for production is no header.

### Next.js metadata

In the App Router, the robots meta tag comes from the `robots` field of `metadata` or `generateMetadata` ([Next.js docs](https://nextjs.org/docs/app/api-reference/functions/generate-metadata)):

```typescript
// app/jobs/[slug]/page.tsx
import type { Metadata } from 'next'

export async function generateMetadata(
  { params }: { params: Promise<{ slug: string }> }
): Promise<Metadata> {
  const { slug } = await params
  const job = await getJob(slug)
  return {
    // index expired jobs: false; live jobs: true
    robots: { index: job?.status === 'live', follow: true },
  }
}
```

Two things to look for. First, the same docs say robots metadata defined in an earlier segment is overwritten by the last segment that defines it, so a `robots: { index: false }` in a layout applies to every page below it unless a page sets its own. Grep for `index: false` and `noindex` across `app/`, including layouts. Second, a data-dependent rule like the one above will mark every page as noindex if `getJob` fails or returns a different status string than you expect. Log the decision, or test a known-live URL after each deploy.

### Next.js `notFound()` during streaming

This applies to streamed App Router pages. When `notFound()` runs before streaming starts, Next.js returns a `404`. When it runs after streaming has started, "the status code remains `200` and the not-found UI renders in the response", and Next.js "injects a `<meta name="robots" content="noindex" />` tag in both cases" ([not-found.js docs](https://nextjs.org/docs/app/api-reference/file-conventions/not-found), [loading.js status codes](https://nextjs.org/docs/app/api-reference/file-conventions/loading)).

So a sitemap URL for a record your page component can't load at request time (deleted, unpublished, or a slow database call that throws and is caught as "not found") comes back as `200` with a noindex. In Search Console that is "URL marked 'noindex'", not "Not found (404)". Two fixes:

- If the record is really gone, remove it from the sitemap. Generate `sitemap.ts` from the same query and status filter the page uses, so the two can't drift.
- If the record exists, find out why the page decided it didn't. Check for timeouts and errors swallowed into `notFound()`.

Next.js recommends running the existence check before the response streams (for example in Proxy) when you need a real `404` status.

### WordPress: the site-wide checkbox

**Settings \> Reading \> Search engine visibility \> Discourage search engines from indexing this site.** Since WordPress 5.3, ticking it outputs a `noindex,nofollow` robots meta tag in every page's `<head>` ([WordPress Reading settings](https://wordpress.org/documentation/article/settings-reading-screen/)). In code, core's `wp_robots_noindex()` adds noindex whenever the `blog_public` option is off ([developer reference](https://developer.wordpress.org/reference/functions/wp_robots_noindex/)). It's the WordPress version of the staging leak: ticked during the build, never unticked at launch, or reset when a staging database is copied over production.

Check it from the command line with WP-CLI:

```bash
wp option get blog_public    # 1 = visible, 0 = discouraged
wp option update blog_public 1
```

### WordPress: per-post and per-type SEO plugin settings

If only some URLs are affected, it's usually the SEO plugin. In Yoast SEO, a single post is noindexed with **Allow search engines to show this content in search results?** set to **No** in the Advanced tab, and a whole content type or taxonomy with **Settings \> Content types \> Show X in search results** switched off ([Yoast help](https://yoast.com/help/how-do-i-noindex-urls/)). Other plugins have the same two levels. When the noindex covers exactly one post type, look at the type-level switch first.

### Server and CDN headers

An `X-Robots-Tag` you didn't write in the app usually comes from the web server or the edge:

```bash
# nginx / Apache config in the repo or on the server
grep -rni 'x-robots-tag' /etc/nginx/ /etc/apache2/ 2>/dev/null
grep -rni 'x-robots-tag' . --include='*.conf' --include='.htaccess' --include='vercel.json' --include='netlify.toml' --include='_headers'
```

Also check CDN transform or header rules, and any middleware that adds headers to all responses.

## Step 3: fix it at the source, by template

Fix the code or setting, not individual URLs. If 3,000 job pages are affected, there's one template or one condition behind them. A good fix has two halves:

1. **The page.** Remove the noindex from the template, header rule or setting, for the pages that should rank.
2. **The sitemap.** Make the sitemap and the robots rule use the same source of truth. If your template noindexes expired jobs, your sitemap query must exclude expired jobs. When one function decides "should this URL be indexed?" and both the `<head>` and `sitemap.ts` call it, this status can't come back.

```typescript
// lib/indexable.ts: one rule, used by the page metadata and the sitemap
export function isIndexable(job: { status: string; expiresAt: Date }) {
  return job.status === 'live' && job.expiresAt > new Date()
}
```

Purge the CDN cache for affected paths after deploying. Google sees what the cache serves.

## Step 4: tell Google, then wait

After the deploy:

1. Re-run the `curl` checks on a few URLs from different parts of the site.
2. Run URL Inspection's live test on one or two. If "Indexing allowed?" now says yes, click **Request Indexing** for your most important pages only. Google says there's a daily limit on index requests ([URL Inspection help](https://support.google.com/webmasters/answer/9012289)).
3. Open the "URL marked 'noindex'" issue in the Page indexing report and click **Validate fix**. Google ["immediately checks a few pages"](https://support.google.com/webmasters/answer/9216203) and stops the validation if any still carry the noindex, so do this after the CDN purge, not before.

Recrawls follow Google's schedule. Its noindex page notes that "depending on the importance of the page on the internet, it may take months for Googlebot to revisit a page" ([block indexing](https://developers.google.com/search/docs/crawling-indexing/block-indexing)). Validation "typically takes up to about two weeks, but in some cases can take much longer" ([validation details](https://support.google.com/webmasters/answer/9216203)). Our post on [Validate fix vs Verify fix](https://seofix.ai/blog/search-console-validate-fix-vs-verify-fix) covers the states and why to confirm the fix yourself before starting it.

If the opposite happened (you added a noindex on purpose and the page still gets clicks), that's a different problem: [noindex page still getting traffic](https://seofix.ai/blog/noindex-page-still-getting-traffic).

## Don't block the URL in robots.txt as well

A common reflex when cleaning up is to add a `Disallow` rule for URLs you've just noindexed. Don't do both. Google's docs are explicit: "If the page is blocked by a robots.txt file or the crawler can't access the page, the crawler will never see the noindex rule, and the page can still appear in search results" ([block indexing](https://developers.google.com/search/docs/crawling-indexing/block-indexing)). URL Inspection makes the same point: for a robots-blocked URL, "Indexing allowed?" always says "Yes", because Google can't read the tag ([URL Inspection help](https://support.google.com/webmasters/answer/9012289)). Noindex and remove from the sitemap; keep crawling open.

## How SEOFix flags it

SEOFix checks both sides of the conflict in an audit. `NOINDEX_IN_SITEMAP` lists every sitemap URL that carries noindex, and `NOINDEX_PAGE` reports the directives behind each noindex page, reading `<meta name="robots">`, `<meta name="googlebot">` and an `X-Robots-Tag` header that applies to Google ([indexability checks](https://seofix.ai/help/checks-indexability), [sitemap checks](https://seofix.ai/help/checks-structure)). Issues are grouped by URL template, so a staging leak shows up as one task on `/jobs/[slug]`, not 3,000 rows. `NOINDEX_PAGE` can be rechecked right after a deploy with [Verify fix](https://seofix.ai/help/verify-fix), before you start Google's validation.

## FAQ

**Does a noindex page in my sitemap hurt the rest of the site?** Google doesn't describe a site-wide penalty for it. It's a contradiction you're asking Google to resolve on every crawl of the sitemap, and it buries the real problems in the report. Clean it up.

**Should I remove noindexed pages from the sitemap even if the noindex is correct?** Yes. A sitemap should list the URLs you want indexed.

**The live test says "Indexing allowed: Yes" but the report still lists the URL.** The report reflects Google's last crawl. Request indexing for the important URLs and validate the issue; the report updates as Google recrawls.

**Can I put noindex in robots.txt?** No. Google says specifying `noindex` in robots.txt "is not supported" ([block indexing](https://developers.google.com/search/docs/crawling-indexing/block-indexing)).

---

Want to know which of your sitemap URLs carry a noindex, grouped by template? The first SEOFix audit is free (up to 500 pages).