Why Isn't My Page on Google? A Beginner's Checklist for noindex, robots.txt and Canonicals
目次
Key Takeaways: Check Three Settings, Then Check Google's Verdict
"I published that page a month ago and it's still not on Google." If you've just inherited a website, you'll say this sooner or later. After going back through Google's documentation, I'd boil the troubleshooting down to four steps.
- First, confirm the page really isn't indexed — Paste the URL into Search Console's URL Inspection tool. It tells you whether the page is in Google's index, the database Google serves results from. Often the page is indexed and simply ranks low, which is a different problem
- noindex means "keep this page out of search" — When Google finds a noindex rule on a page, it drops the page from its results. The classic cause is a staging setting that survived a site launch or redesign
- robots.txt means "don't come in"; canonical means "this is the main version" — robots.txt limits what crawlers (Google's page-reading bots) can access. It isn't a way to hide pages. A canonical tag tells Google which URL to treat as the main one when several pages are near-duplicates
- Even with clean settings, Google may not index a page — Google doesn't promise to index everything. Search Console's Page indexing report tells you why each URL was left out, so you can fix what's fixable
Let's start with the vocabulary, then work through each check and the mistakes I see most.
1. "Not on Google" Can Mean Two Different Things
Here's the thing: "my page isn't showing up" usually describes one of two very different situations.
| Situation | What it means | Where to look |
|---|---|---|
| Not indexed | The page isn't in Google's database at all | The settings and errors in this article |
| Indexed, but ranking low | The page is in the database but doesn't show near the top | Content quality and keyword targeting |
Think of the index as a library catalog. If a book isn't in the catalog, no amount of searching will bring it off the shelf. This article is about the first case: pages missing from the catalog. For the full pipeline from crawling to ranking, see "How Search Engines Work."
How do you tell which one you have?
The most reliable way is Search Console's URL Inspection tool. Search Console is Google's free dashboard for site owners. Paste a URL into the search bar at the top and you'll see one of these:
- URL is on Google — It's indexed. If you can't find it in search, you're most likely dealing with a ranking problem
- URL is not on Google — It isn't indexed. Work through the checklist below
You can also type site:example.com into Google, but Google itself says the site: operator doesn't necessarily return every indexed URL. Treat it as a rough signal and let URL Inspection make the call.
New to Search Console? "Your First Hour in Google Search Console" covers the three screens to open first.
References: URL Inspection tool - Search Console Help Get your website on Google Search - Google Search Central
2. noindex: "Please Keep This Page Out of Search"
noindex comes first on the checklist. In my experience, it's the setting people overlook most when a page suddenly vanishes.
What noindex is
noindex is an instruction, placed either in the page or in the server's response, that says "don't show this page in search results." There are two ways to write it.
| Method | Where it goes | Typical use |
|---|---|---|
<meta name="robots" content="noindex"> | In the page's HTML <head> | Regular web pages |
X-Robots-Tag: noindex | In the HTTP response header | PDFs, images and other non-HTML files |
According to Google's documentation, when Googlebot (Google's crawler) crawls a page and finds noindex, it drops that page from Google Search results entirely, regardless of whether other sites link to it. If it's there by accident, even your best article won't show up.
How it ends up there by accident
The usual suspects:
- A "discourage search engines" setting switched on during a redesign, then left on at launch
- In WordPress, the "Discourage search engines from indexing this site" box under Settings → Reading is still checked
- An SEO plugin rule meant to noindex tag or category archives also catches your posts
Google's site-move guide explicitly lists removing leftover noindex rules and robots.txt blocks from development as a launch check. It's that common.
How to check
The URL Inspection result shows whether noindex is blocking the page. For a site-wide list, use the Page indexing report covered below.
After you remove noindex, nothing changes until Googlebot recrawls the page. Google notes that, depending on a page's importance, that can take months. If you're in a hurry, use "Request indexing" in URL Inspection to ask for a recrawl.
References: Block Search indexing with noindex - Google Search Central Site moves with URL changes - Google Search Central
3. robots.txt: "Please Don't Come In"
robots.txt gets confused with noindex constantly. Honestly, understanding the difference between the two prevents a large share of indexing headaches.
What robots.txt is
robots.txt is a plain-text file at the root of your site (https://example.com/robots.txt). It tells crawlers which URLs they may or may not access.
Google describes its main purpose as avoiding overloading your site with requests. And it's blunt about what robots.txt is not: it is not a mechanism for keeping a web page out of Google.
Why it can't hide a page
This is the part that trips up beginners. If other sites link to a page you've blocked in robots.txt, Google can still learn the URL exists. It may then show that URL in results with no description, because it was never allowed to read the page.
robots.txt is a "Staff Only" sign on a door. Nobody walks in, but anyone outside can still see the building is there.
Using both together backfires
It gets trickier when you combine them.
| Setup | Result |
|---|---|
| noindex only | Google reads the page, sees noindex, and drops it from results |
| robots.txt block only | Google doesn't read the page. With links pointing to it, the bare URL may still appear |
| robots.txt block + noindex | Google can't read the page, so it never sees noindex. The page may stay in results |
Google states plainly that crawlers must be able to access a page for noindex to work. To reliably keep a page out of search, use noindex and don't block it in robots.txt. Password protection or deleting the page also work.
Flip side: make sure the pages you want in search aren't blocked. A single Disallow: / line tells crawlers to stay away from your entire site, and it's just as easy to forget at launch as a stray noindex.
References: Introduction to robots.txt - Google Search Central Block Search indexing with noindex - Google Search Central
4. Canonical Tags: "This Is the Main Version"
The third check is the canonical tag. Unlike the first two, it isn't about hiding anything.
What a canonical is
When the same or nearly the same content lives at several URLs, you have duplicates. For example:
https://example.com/shoesandhttps://example.com/shoes?color=red- The same page loading on both
http://andhttps:// - A printer-friendly version of an article
Google picks one of these as the canonical, the main URL it shows in results. This process is called canonicalization, and <link rel="canonical" href="..."> is how you tell Google which URL you'd prefer.
It's a hint, not a command
What surprised me when I first read Google's guidance is how carefully it frames this. Redirects, canonical tags and sitemaps are all described as signals, and none of them is required. Redirects are the strongest signal, rel="canonical" is also strong, and sitemap inclusion is weaker.
So Google may pick a different URL than the one you declared. URL Inspection shows the "User-declared canonical" and the "Google-selected canonical" side by side. Check that they match.
When a canonical makes a page "disappear"
Two typical cases:
- Page A's canonical points to page B by mistake — A template bug that points every page's canonical at the homepage is more common than you'd think. If Google follows that hint, page A itself becomes hard to find
- It's working as intended — If parameter URLs are being folded into the main URL, that's healthy
Google also warns against using robots.txt or noindex for canonicalization. noindex, in particular, blocks the page from Search entirely.
References: How to specify a canonical URL with rel="canonical" and other methods - Google Search Central URL Inspection tool - Search Console Help
5. Other Common Reasons
Your three settings can be spotless and a page still won't be indexed. Google lists low-quality content, robots meta rules that block indexing, and site designs that are hard to index as common reasons. These are the ones beginners hit most.
| Reason | What's happening | First step |
|---|---|---|
| The page is brand new | Google hasn't discovered it yet | Wait a few weeks. Link to it from other pages on your site |
| Not found (404) | The URL is wrong or the page is gone | Fix the URL. If the page moved, add a 301 redirect |
| Login required | Googlebot can't log in, so it can't read the page | Make public pages viewable without a login |
| Server error (5xx) | Your server failed to respond | Check your hosting and server health |
| The URL redirects | The URL forwards to another URL | Normal, as long as the destination is indexed |
Google says it can take several weeks for it to notice a new site or changes to an existing one, and that it finds new pages mainly through links and sitemaps. Make sure your new page isn't an orphan with no internal links pointing to it. For sitemaps in depth, see "XML Sitemap Strategy 2026." For moved pages, "What Is a 301 Redirect?" covers the basics.
References: In-depth guide to how Google Search works - Google Search Central Get your website on Google Search - Google Search Central
6. Reading Search Console's Page Indexing Report
Inspecting URLs one at a time gets old fast. For a site-wide view, open Pages (the Page indexing report) in Search Console's left menu.
Every "not indexed" reason in one list
The report charts indexed versus not-indexed pages and lists the reasons pages weren't indexed. Here's how the labels map to this article.
| Reason in the report | What it means | Action needed? |
|---|---|---|
| URL marked 'noindex' | The page has noindex | Remove it if you want the page in search |
| URL blocked by robots.txt | robots.txt is blocking crawling | Unblock it if you want the page in search |
| Alternate page with proper canonical tag | Correctly folded into its canonical | No |
| Duplicate without user-selected canonical | No canonical declared; Google picked another URL | Add a canonical if you have a preference |
| Duplicate, Google chose different canonical than user | Google disagrees with your canonical | Double-check your canonical |
| Not found (404) | The page doesn't exist | Add a 301 if the page moved |
| Page with redirect | The URL forwards elsewhere | No, if the destination is indexed |
| Crawled - currently not indexed | Crawled but not indexed yet | Review the content. Google says resubmitting isn't needed |
| Discovered - currently not indexed | Known but not crawled yet | Check internal links and sitemaps, then wait |
Don't chase 100%
Here's where it gets interesting. Google's own help page says you shouldn't expect every URL on your site to be indexed. The goal, in its words, is to get the canonical version of every important page indexed.
Admin screens, thank-you pages and internal search results being unindexed is normal. Don't panic at the total. Scan the list for pages you actually want in search.
References: Page indexing report - Search Console Help
7. Common Misconceptions and Mistakes
Let's turn all of that into the mistakes that actually happen.
Misconception 1: "Blocking a page in robots.txt removes it from Google"
Not necessarily. robots.txt only stops crawling. A linked URL can still show up with no description. If you want it gone, use noindex and leave it crawlable.
Misconception 2: "noindex plus robots.txt is extra safe"
It's the opposite. A blocked page can't be read, so Google never sees the noindex. Use noindex alone.
Mistake 1: Organic traffic flatlines after a redesign
A staging noindex or Disallow: / made it to production. Running your homepage and key pages through URL Inspection right after launch catches this in minutes.
Mistake 2: Every canonical points to the homepage
Hard-coding a canonical into a template makes every page declare "the homepage is the main version." Check that each key article's user-declared canonical is its own URL.
Mistake 3: Hammering "Request indexing"
A request asks Google to crawl. It doesn't guarantee indexing, and Google's help page says as much. Fix whatever is blocking the page first.
8. How to Check Your Own Site
Search Console's free tools cover the first pass. If you'd like a page-level settings audit in one place, LinkSurge's AI Ready audit checks robots.txt, the canonical setting, whether a meta description exists, and more, item by item. It was built to assess readiness for AI search, but it flags these basic misses too.
Once indexing is sorted, the next question is how your pages actually perform in results. LinkSurge's rank tracker records where your pages rank for the keywords you register, on a schedule.
LinkSurge
linksurge.jp
SEO・AIO・GEO統合分析プラットフォーム。AI Overviews分析、SEO順位計測、GEO引用最適化など、生成AI時代のブランド露出を最大化するための分析ツールを提供しています。
Quick Check
Q1: Can a page blocked by robots.txt ever appear in Google's results?
Answer: Yes. If other sites link to it, Google may show the bare URL without a description. To keep a page out of search, use noindex instead.
Q2: If you set a canonical tag, will Google always use that URL as the canonical?
Answer: No. A canonical tag is a strong signal, not a command. Compare the user-declared and Google-selected canonicals in URL Inspection.
Q3: Should you fix every "not indexed" URL in the Page indexing report?
Answer: No. Google says not every URL will be indexed. The goal is for every important page to be indexed, so check that those pages aren't on the list.
Frequently Asked Questions
How long does it take for a new page to show up on Google?
There's no fixed timeline. Google says it can take several weeks to notice a new site or changes. Linking to the page from elsewhere on your site and listing it in your sitemap help Google find it.
I removed noindex, but the page still isn't showing. Why?
The change takes effect only after Googlebot recrawls the page, which can take a while for some pages. Run a live test in URL Inspection, and if it passes, use "Request indexing" to ask for a crawl.
How should I keep a page out of search?
Add noindex and make sure robots.txt doesn't block the page, or Google won't see the rule. For content that people shouldn't see either, such as members-only pages, use password protection.
If a site: search doesn't show my page, does that mean it isn't indexed?
Not necessarily. Google says the site: operator doesn't return every indexed URL. Use URL Inspection in Search Console for a definitive answer.
Conclusion: Confirm Indexing First, Then the Three Settings
When a page won't show up, resist the urge to rewrite it. Check whether it's indexed with URL Inspection. If it isn't, go through noindex, robots.txt and canonical in that order. Still stuck? Let the reasons in the Page indexing report point the way.
If every setting checks out and the page still isn't indexed, look at the content itself and at how many internal links reach it.
Up next in the series: "Title Tags and Meta Descriptions," which covers how your page looks once it's in results. Missed the previous article? Read "How Search Engines Work." Stuck on a term? Try the "SEO Glossary for Beginners." The full reading order lives in the "SEO, GEO and AIO for Beginners: A Reading Order From Zero."
LinkSurge's AI Ready audit checks basic settings such as robots.txt and canonical tags page by page.
