# How Search Engines Work: Crawling, Indexing and Ranking Explained > How does Google find your pages and decide where they rank? A beginner's walkthrough of crawling, indexing and serving results, based on Google's official documentation and explained with a library analogy. | Field | Value | |-------|-------| | Published | 2026-10-13 | | Category | Beginner | | Author | Aro Ogata | | Tags | beginner, SEO basics, SEO, crawling, indexing, how search works, Googlebot | | URL | https://linksurge.jp/blog/en/how-search-engines-work/ | --- ## Key Takeaways: Search Engines Find, Store, Then Choose "I published the page a week ago. Why can't I find it on Google?" It's the first wall almost every beginner hits. In my experience, most of these cases make sense within minutes once you know how search engines actually work. Based on Google's official documentation, here's what to know. 1. **Search works in three stages** — Google discovers and fetches pages (crawling), understands and stores them (indexing), then picks and orders the best matches for each search (serving, which is where ranking happens) 2. **Not every page makes it through all three** — Google states plainly that it doesn't guarantee it will crawl, index or serve any page. If a page stalls at any stage, it won't appear in results 3. **You can't pay for rankings** — Google says it doesn't accept payment to crawl a site more often or rank it higher. Rankings are decided by automated systems 4. **Results differ from person to person** — The same query can return different results depending on location, language and device Let's walk through each stage, using a library as our analogy. --- ## 1. Think of Google as a Giant Library First, the big picture. Google Search is fully automated. Most pages in its results weren't submitted by anyone; Google's software found them by exploring the web. Here's how the three stages map onto a library. | Stage | Google's term | Library analogy | |:--|:--|:--| | 1 | Crawling | Visiting bookstores worldwide to collect new books and new editions | | 2 | Indexing | Reading each book, cataloging it, and shelving it | | 3 | Serving results (ranking) | Hearing a patron's question, then handing over the most useful books, best first | The stages run in order. If a book is never collected, it's never cataloged. If it's never cataloged, no one is ever handed it. Google's own guide makes the same point: not every page goes through all three stages. When a page won't show up, the fastest diagnosis is figuring out where it stalled. > References: > [In-Depth Guide to How Google Search Works - Google Search Central](https://developers.google.com/search/docs/fundamentals/how-search-works) --- ## 2. Stage 1: Crawling — Finding and Fetching Pages Crawling is when a search engine's software visits a page and downloads its content. That software is called a crawler, and Google's is named Googlebot. ### How does Google find new pages? There's no central registry of every web page. So Google has to keep looking for new and updated pages. It finds them in two main ways. - **Following links** — When Google already knows a page, it can discover new pages linked from it. Google's guide names this as one way new pages are discovered - **Reading sitemaps** — A sitemap is a file listing the URLs on your site. Submitting one gives Google another way to discover your pages Here's the practical takeaway: a page nothing links to is hard to find. Linking a new article from other pages on your site does real work. For sitemap planning, see "[XML Sitemap Strategy 2026](/blog/en/xml-sitemap-strategy-2026/)." ### Found doesn't mean fetched Google's systems decide which sites to crawl, how often, and how many pages. Googlebot doesn't crawl every page it discovers. Common reasons a page isn't crawled: - **robots.txt blocks it** — robots.txt is a file that tells crawlers which parts of your site they may access. A wrong rule can lock Googlebot out - **It requires a login** — Googlebot can't see members-only pages - **The server is struggling** — Google avoids overloading sites. If your server returns errors like HTTP 500, crawling slows down ### Does Google see content loaded by JavaScript? Many sites use JavaScript to load content. During crawling, Google renders pages with a recent version of Chrome and runs the JavaScript it finds. Rendering can still fail for various reasons. Putting your most important information directly in the HTML as text is the safer bet. > References: > [In-Depth Guide to How Google Search Works - Google Search Central](https://developers.google.com/search/docs/fundamentals/how-search-works) > [Learn About Sitemaps - Google Search Central](https://developers.google.com/search/docs/crawling-indexing/sitemaps/overview) --- ## 3. Stage 2: Indexing — Understanding and Storing Pages Pages that get crawled move on to indexing. The index is Google's enormous database of the pages it has processed. It's the library catalog. ### What does Google read on a page? Beyond the body text, Google analyzes elements like these. - **The title element** — the HTML tag that names the page, often used as the clickable headline in results - **Alt attributes** — text descriptions of images - **Images and videos** — the media on the page The basics apply: make it obvious, through text and headings, what the page is about. ### Similar pages get grouped, and one is chosen Google groups pages with similar content, then picks one to represent the group. That representative is called the canonical page. The canonical is the page that may be shown in search results. The others are treated as alternate versions that may appear in specific contexts, like mobile searches. If the same content lives at several URLs, Google may choose a different page than the one you wanted. This is one reason pages don't show up, which the series covers in "[Why Isn't My Page on Google? A Checklist](/blog/en/page-not-indexed-checklist/)." ### Indexing isn't guaranteed either This is the part people miss. Google doesn't index every page it crawls. Its guide lists reasons such as: - Low-quality content - Robots meta rules (page-level instructions to search engines) that block indexing, such as noindex - Site designs that make indexing difficult Honestly, most "I published it but it's not there" cases I've looked at stall here, or one step earlier at crawling. > References: > [In-Depth Guide to How Google Search Works - Google Search Central](https://developers.google.com/search/docs/fundamentals/how-search-works) --- ## 4. Stage 3: Serving Results — Choosing and Ordering Pages The final stage happens when someone searches. Google looks through its index for pages that match the query and that it judges to be high quality, then shows them. "Ranking" is the ordering that happens here. ### What decides the order? Google weighs many factors beyond the match between page and query. Its guide cites the searcher's location, language and device as examples. Search "bicycle repair shop" in Paris and in Hong Kong and you'll get different results. What's useful depends on who's asking and where. Google's "guide to Google Search ranking systems" describes some of the systems involved. There are systems that analyze links, systems that understand the meaning of words, and freshness systems that favor newer content for searches where recency matters. Ranking isn't one rule. It's many systems working together. ### Can you pay to rank higher? No. Google states that it doesn't accept payment to crawl a site more frequently or rank it higher, and that ranking is done programmatically. ### Results pages look different for different searches The features on a results page change with the query too. In Google's own example, a search for "bicycle repair shop" will likely show local results rather than image results. These days, an AI-generated answer (an AI Overview) may also appear above the links. A page can be indexed and still not shown, if it doesn't match the query or isn't judged useful enough. Being in the catalog doesn't mean you'll be handed to every patron. > References: > [In-Depth Guide to How Google Search Works - Google Search Central](https://developers.google.com/search/docs/fundamentals/how-search-works) > [A Guide to Google Search Ranking Systems - Google Search Central](https://developers.google.com/search/docs/appearance/ranking-systems-guide) --- ## 5. Common Misconceptions and Mistakes Once you know the three stages, the classic beginner mistakes make sense. Here are five. ### Myth 1: "Publish it and it's on Google right away." Crawling and indexing take time. Google's SEO Starter Guide says changes can take anywhere from hours to several months to be reflected, and suggests waiting a few weeks before judging an effect. Not showing up the next day is normal. ### Myth 2: "Submit a sitemap and every page gets indexed." A sitemap helps discovery. That's all. Google's sitemap documentation says there's no guarantee every item in a sitemap will be crawled and indexed. ### Myth 3: "Search Console says it's indexed, so it'll rank." In Search Console (Google's free site-management tool), the URL Inspection tool may say "URL is on Google." Google's help center notes that this doesn't guarantee the page will appear in search results. Passing stage 2 doesn't mean you'll be picked in stage 3. ### Myth 4: "The ranking I see is the ranking everyone sees." Results vary by location, language and device. Two people on the same team reporting "we're #1" and "we're #5" is common. When comparing rankings, use data recorded under the same conditions. Rank trackers like [LinkSurge](https://linksurge.jp/) measure under fixed settings, which makes trends comparable. ### Myth 5: "Leave every test page and old page live. It can't hurt." Large numbers of thin or near-duplicate pages make it harder for Google to pick the right canonical. Make it clear, through your site structure, which pages you actually want people to find. --- ## 6. How to Check Your Own Site Now check how far your own pages have gotten. 1. **Crawling and indexing status** — Paste a URL into Search Console's URL Inspection tool. It tells you whether the page is indexed and which canonical Google picked 2. **Serving** — Search for the keywords that page targets and see whether it appears 3. **Which queries you show up for** — You may be appearing for searches you never planned For the third, use Search Console's Performance report, or a tool like [LinkSurge](https://linksurge.jp/)'s [ranked keywords analysis](https://linksurge.jp/ranked-keywords), which lists the keywords a domain ranks for. Seeing which questions you're "picked" for in stage 3 is a strong hint about what to write next. For a beginner-friendly tour of Search Console, see "[Search Console: Your First Hour](/blog/en/search-console-first-hour/)" in this series. > References: > [URL Inspection Tool - Search Console Help](https://support.google.com/webmasters/answer/9012289) > [SEO Starter Guide - Google Search Central](https://developers.google.com/search/docs/fundamentals/seo-starter-guide) --- {{linksurge-card}} ## Quick Check ### Q1: Name the three stages of search, in order. Answer: Crawling (finding and fetching pages), indexing (understanding and storing them), and serving results (choosing and ordering pages for a query). ### Q2: If you submit a sitemap, will every page get indexed? Answer: No. A sitemap helps Google discover pages, but Google says there's no guarantee every item will be crawled and indexed. ### Q3: Why do two people searching the same thing see different results? Answer: Because Google also considers factors like the searcher's location, language and device when choosing results. --- ## Frequently Asked Questions ### What is Googlebot? Googlebot is the name of Google's crawler, the software that visits web pages and downloads their content. It discovers new and updated pages through links and sitemaps and passes what it fetches on to indexing. ### How long does it take for a new page to appear on Google? There's no fixed timeline. Google's SEO Starter Guide says changes can take from a few hours to several months to be reflected. Link the page from elsewhere on your site and check its status with Search Console's URL Inspection tool while you wait. ### Why is my page indexed but not showing in search results? Indexing is a prerequisite, not a guarantee. A page may not be shown if it doesn't match what people search for, or if other pages are judged more useful for that query. ### Can I pay Google to rank higher? No. Google states that it doesn't accept payment to crawl a site more frequently or to rank it higher. Rankings are determined automatically by its systems. --- ## Conclusion: Always Ask "Which Stage Is It Stuck At?" Most search problems become clearer once you ask where the page stalled. Was it never found? Found but not indexed? Indexed but not chosen? Each answer leads to completely different work. Start by running your most important pages through Search Console's URL Inspection tool. Once you know they're indexed, move on to improving how often they get chosen. The previous article in this series is "[SEO vs. GEO vs. AIO vs. LLMO](/blog/en/seo-geo-aio-llmo-differences/)." Up next: "[Why Do Search Rankings Change Every Day?](/blog/en/why-search-rankings-change/)" For the full reading order, see the "[SEO, GEO and AIO Beginner Curriculum](/blog/en/seo-beginner-curriculum/)," and keep the "[SEO Glossary for Beginners](/blog/en/seo-glossary-beginners/)" nearby. When you're ready for more, the "[Complete SEO Guide 2026](/blog/en/seo-guide-2026/)" and "[Topic Clusters & Internal Linking Strategy](/blog/en/topic-cluster-internal-linking-strategy/)" are good next steps. LinkSurge's [ranked keywords analysis](https://linksurge.jp/ranked-keywords) shows every keyword your domain currently appears for in search results.