Indexing, robots.txt, canonical, redirects, Core Web Vitals — how a site's technical layer affects visibility.
Sitemap.xml is an XML file listing the URLs you want in search. It speeds up discovery of new and deeply nested pages but does not guarantee indexing. Include only canonical pages returning 200; the limit is 50,000 URLs or 50 MB per file. Reference the sitemap in robots.txt and submit it in Search Console and Yandex Webmaster.
The file at the site root sets crawl rules: which sections bots should not crawl. Important: Disallow blocks crawling, not indexing — a blocked URL can still appear in results without a description if other pages link to it. To remove a page from search, use meta robots noindex with crawling allowed. robots.txt is useless for protecting data: it is a public file anyone can read.
The rel="canonical" tag names the preferred URL among pages with identical or near-identical content, for example with UTM tags or sorting parameters. It is a hint, not a directive — the search engine may choose another URL. A classic mistake is a canonical pointing to a non-existent or blocked page: the search engine loses the main version and may drop the whole section from the index.
Indexing is adding a page to the search engine's database, after which it can appear in results. The chain is: discovery (links, sitemap) → crawling → rendering → quality and duplicate assessment → index. Crawling is not indexing: the Search Console status "Crawled — currently not indexed" means Google saw the page but did not consider it valuable enough.
404 means "page not found". It is the correct response for removed URLs; 410 says the page is gone for good and is usually dropped from the index faster. Two real problems: "soft 404s", where an empty page returns 200, and internal links to non-existent URLs. If a removed page has an equivalent, a 301 redirect is better.
A 301 is a permanent redirect: it moves users, bots and most of a page's signals to the new address. Use it when changing URLs or domains, moving to HTTPS and merging duplicates. Rules: redirect straight to the final address without chains, to a relevant page (not everything to the homepage), and keep it for at least a year. A 302 is temporary and unsuitable for permanent moves.
Three Google metrics based on real Chrome user data: LCP — main content load (good ≤ 2.5 s), INP — responsiveness to interactions (≤ 200 ms), CLS — visual stability (≤ 0.1). They are assessed at the 75th percentile of visits over 28 days. They are a ranking signal, weaker than relevance — but speed directly affects conversion.
Ideally, yes. Google can execute JavaScript, but rendering is deferred and resource-intensive, and many other bots — including AI crawlers — do not run JavaScript at all and see only the source HTML. If text, links and prices appear only after scripts run, some systems will miss them. Serving ready HTML through SSR or static generation is more reliable.
First find the source: www vs non-www, trailing slashes, sorting and UTM parameters, print versions, identical product cards. Then choose the main URL and the right tool: a 301 redirect for technical duplicates, canonical for parameters, Clean-param in robots.txt for Yandex. Internal links and the sitemap should point only to the main version.
HTTPS encrypts data between browser and server: without it, form contacts can be intercepted on public Wi-Fi and browsers label the site "Not secure". Google uses HTTPS as a ranking signal. When migrating, 301-redirect every HTTP URL to HTTPS, update internal links, canonicals and the sitemap, and enable the HSTS header.
Nothing found.
