Basic AI SEO Course · Lesson 6 of 8 6 of 8 lessons

Lesson 06 · Technical Foundations

Technical SEO Basics:
Crawlability, Indexing & Speed.

You can write the greatest content on the internet, but if search engines cannot crawl or render it, your visibility will be zero. Here is how the technical pipeline works and how to protect it.

~13 min read 📝 5-question quiz at the end

The Search Engine Pipeline: Crawl → Render → Index → Rank

Before a page ever appears in search results, it must pass through four distinct stages:

  1. Crawling (Discovery): Web crawlers (like Googlebot) follow links across the internet to discover new and updated URLs.
  2. Rendering (Processing): The crawler runs HTML, CSS, and executes JavaScript to construct the actual visual and semantic page.
  3. Indexing (Storage): The analyzed text, entities, images, and schema are filed away into Google's massive global index database (Caffeine).
  4. Ranking (Serving): When a user enters a query, ranking algorithms evaluate indexed documents to return the best results.

Crucial Distiction: Crawling ≠ Indexing
Just because Googlebot crawled your URL does not mean it added the page to its index. If the page is deemed low quality, duplicate, or inaccessible, it remains crawled but excluded.

The Gatekeepers: Robots.txt & XML Sitemaps

1. Robots.txt

Your robots.txt file lives at the root of your domain (e.g., /robots.txt) and instructs bots where they are and are not permitted to crawl:

User-agent: *
Disallow: /admin/
Disallow: /checkout/

Sitemap: https://aiseocourse.online/sitemap.xml

Warning: Disallowing a URL in robots.txt does not remove it from the index if other sites link to it. To guarantee a page is not indexed, use the <meta name="robots" content="noindex"> directive instead.

2. XML Sitemaps

An XML sitemap is a clean map of all canonical, high-value URLs you want search engines to prioritize. Submit this URL in Google Search Console to speed up new page discovery.

Canonicalization: Preventing Duplicate Content Confusion

If your website serves the same content across multiple URLs (such as HTTP vs HTTPS, www vs non-www, or e-commerce URL filters like ?color=blue), search engines get confused about which version to credit.

The solution is the canonical tag placed in the <head>:

<link rel="canonical" href="https://aiseocourse.online/courses/basic/">

This tells search engines: "Regardless of how a user arrived at this URL, treat this exact canonical address as the master version."

Core Web Vitals & Page Experience

Google officially measures user experience through three Core Web Vitals metrics:

  • LCP (Largest Contentful Paint): Loading speed. Measures how long it takes for the main content to render (target: under 2.5 seconds).
  • INP (Interaction to Next Paint): Responsiveness. Measures the latency of user interactions like button clicks and form taps (target: under 200 milliseconds).
  • CLS (Cumulative Layout Shift): Visual stability. Measures whether elements jump around unexpectedly while assets load (target: under 0.1).

Check Your Understanding, Lesson 6 Quiz

1. What are the four core phases of the search engine processing pipeline?

Question 1 of 5

2. What happens if you block a URL in robots.txt vs using a "noindex" meta tag?

Question 2 of 5

3. What is the role of a rel="canonical" tag?

Question 3 of 5

4. What does the Core Web Vitals metric "INP" measure?

Question 4 of 5

5. What is Cumulative Layout Shift (CLS)?

Question 5 of 5

—

Quiz Complete

Score at least 3/5 to mark this lesson complete and unlock Lesson 7.

Complete the quiz above with 3 or more correct answers to finish Lesson 6.