Technical SEO Checklist for Business Websites (Including AI Search and llms.txt)

Technical SEO is the foundation everything else in search marketing stands on. You can publish excellent articles and earn strong backlinks, but if search engines cannot crawl, render and understand your pages, that effort is wasted. For business websites the stakes are higher than ever, because search no longer ends with ten blue links: answers increasingly appear in AI overviews, chat assistants and voice results that pull from the same underlying signals.

This checklist is the one the Creensight team uses when we audit or launch a business website. It covers the classic technical essentials — crawlability, indexation, speed, structured data — and the newer layer of AI search readiness, including llms.txt and crawler policies for AI bots. Work through it section by section; most items take minutes, and together they make your site easier to find, understand and cite.

1. Crawlability: let search engines in

Crawlers discover your pages by following links and reading sitemaps. Anything that blocks or confuses them reduces what can be indexed.

Checklist

• robots.txt is reachable at /robots.txt, returns status 200 and does not block important sections, CSS or JavaScript.
• Admin areas, carts, checkouts and internal search results are disallowed to save crawl budget.
• The robots.txt file references your XML sitemap with a Sitemap: line.
• Important pages are reachable within three clicks from the home page.
• Navigation uses real <a href> links, not JavaScript click handlers that crawlers cannot follow.
• No orphan pages — every page you want indexed is linked from at least one other page.

On WordPress, plugins such as Yoast SEO generate a sitemap index automatically (typically at /sitemap_index.xml) and let you exclude post types, taxonomies and media attachment pages you do not want indexed.

2. Indexation: control what appears in results

Crawling is not indexing. Search engines choose which pages to keep, and you can guide that choice.

Signal	Purpose	Common mistake
Canonical tag	Tells search engines the preferred URL among duplicates	Pointing every page to the home page
Meta robots noindex	Keeps a page out of the index	Left on after a staging site goes live
XML sitemap	Lists URLs you want indexed	Including redirected or noindexed URLs
HTTP status codes	200 for live pages, 301 for moves, 404/410 for removed	“Soft 404s” — error pages returning 200
hreflang	Maps language or country versions	Missing return links between versions

Use the Pages report in Google Search Console and the equivalent in Bing Webmaster Tools to see which URLs are excluded and why. “Crawled – currently not indexed” often signals thin or duplicate content rather than a technical fault.

Site architecture and URLs

• URLs are short, lowercase, hyphen-separated and descriptive: /services/web-development/ rather than /?page_id=123.
• One topic per page; avoid several pages competing for the same keyword.
• Breadcrumbs reflect the hierarchy and are marked up with BreadcrumbList schema.
• Internal links use descriptive anchor text that tells users and crawlers what the destination is about.
• Redirect chains are collapsed into single 301 redirects.
• HTTP redirects to HTTPS, and one host version (with or without www) redirects to the other.

3. Page speed and Core Web Vitals

Google’s Core Web Vitals measure real-user experience: Largest Contentful Paint (LCP) for loading, Interaction to Next Paint (INP) for responsiveness and Cumulative Layout Shift (CLS) for visual stability. Commonly cited “good” thresholds are LCP under 2.5 seconds, INP under 200 milliseconds and CLS under 0.1, measured at the 75th percentile of visits.

Fixes that move the needle

1. Images: serve WebP or AVIF, size them to their display dimensions, add width and height attributes and lazy-load anything below the fold. Do not lazy-load the hero image.
2. Fonts: host fonts locally, preload the one or two used above the fold and use font-display: swap.
3. JavaScript: remove unused plugins and third-party scripts, defer non-critical scripts and avoid heavy page builders where a lean theme will do.
4. Caching: enable page caching and a CDN; set long cache lifetimes for static assets.
5. Server: choose hosting with modern PHP, HTTP/2 or HTTP/3 and a nearby data centre for your audience.

Every third-party script is a small loan against your page speed. Before adding a chat widget, tracking pixel or font service, ask what it earns you and whether it can load later.

Our web development team builds WordPress themes with local assets, minimal plugins and clean markup precisely because speed problems are cheaper to prevent than to fix.

4. Mobile and accessibility

Search engines index the mobile version of your site first, so the mobile experience is the experience that counts.

• A responsive layout with a correct viewport meta tag.
• Tap targets at least 44–48 pixels, readable font sizes and no horizontal scrolling.
• Content on mobile matches desktop — do not hide important text on small screens.
• Images have meaningful alt text; headings follow a logical order (one H1, then H2s and H3s).
• Sufficient colour contrast and visible focus states for keyboard users.

Accessibility and SEO overlap heavily: clear structure, descriptive links and alt text help screen readers and crawlers alike.

5. Structured data

Structured data in JSON-LD format describes your content in a machine-readable way. It can make pages eligible for rich results and gives search engines and AI systems unambiguous facts about your business.

Schema type	Where to use it	Key properties
Organization	Site-wide	name, legalName, url, logo, address, sameAs, identifier
WebSite	Home page	name, url, potentialAction (SearchAction)
Article / BlogPosting	Blog posts	headline, datePublished, dateModified, author, image
BreadcrumbList	All inner pages	itemListElement with position, name, item
FAQPage	Genuine FAQ sections	mainEntity of Question and Answer
Product / Offer	Shop items	name, description, image, offers (price, priceCurrency, availability)

For an Indian company, adding the Corporate Identification Number (CIN) as an identifier and listing official social profiles in sameAs strengthens entity recognition. Validate markup with Google’s Rich Results Test and the Schema.org validator, and only mark up content that is visible on the page.

6. AI search readiness and llms.txt

AI assistants and AI-powered search features answer questions by retrieving and summarising web content. You cannot control exactly how they use it, but you can make your site easier to retrieve, understand and cite accurately.

Make your facts easy to extract

• State key facts plainly on your About and Contact pages: legal name, registration details, location, founding year, services and contact channels.
• Keep brand information consistent across your site, social profiles and business directories.
• Answer common questions directly near the top of relevant pages, then expand with detail.
• Use descriptive headings that match how people phrase questions.

Publish an llms.txt file

llms.txt is a proposed convention: a plain Markdown file at the root of your domain that gives language models a concise summary of your site and links to its most important pages. A typical file contains a title, a one-paragraph summary in a blockquote, and sections listing key URLs with short descriptions. Some sites also publish llms-full.txt with the full text of core pages. Support among AI providers is still evolving, so treat it as a low-cost, low-risk addition rather than a ranking factor.

Decide your AI crawler policy

Several AI crawlers identify themselves with user-agent tokens such as GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended (a control token for Google’s AI products). You can allow or disallow them individually in robots.txt. For most businesses that want brand visibility in AI answers, allowing reputable AI crawlers to read public marketing content is the sensible default, while keeping private areas disallowed.

7. Security, HTTPS and site health

• A valid TLS certificate on every subdomain, with automatic renewal.
• No mixed content (HTTP images or scripts on HTTPS pages).
• Security headers such as X-Content-Type-Options and Referrer-Policy.
• A security plugin or web application firewall, updated core, themes and plugins.
• Uptime monitoring with alerts — repeated downtime can lead to pages dropping from the index.
• A custom 404 page that returns a real 404 status and helps users find their way.

8. Duplicate content, parameters and e-commerce pages

Business sites with a shop or a filterable blog often generate many near-identical URLs without anyone noticing. Sorting options, tracking parameters, session IDs and pagination can multiply a few hundred real pages into thousands of crawlable variations.

• Point filtered and sorted product listings to the main category with a canonical tag, or keep them out of the index.
• Strip UTM and other tracking parameters from internal links; use them only on external campaigns.
• Give every product a unique description instead of copying the manufacturer’s or a sibling product’s text.
• Noindex cart, checkout, account and order-confirmation pages, including thank-you pages after form submissions.
• Make sure tag and author archives add value; otherwise noindex them to avoid thin duplicate listings.
• Check that print versions, AMP leftovers and staging subdomains are not indexed.

A crawl with a desktop SEO crawler will quickly show clusters of duplicate titles or descriptions, which are usually the clearest sign of this problem.

9. Monitoring and maintenance

Technical SEO decays. Plugins update, editors upload huge images, redirects accumulate. A light routine keeps problems small.

Frequency	Task
Weekly	Scan Search Console for new errors and manual actions
Monthly	Review Core Web Vitals, indexation trends and broken links
Quarterly	Full crawl with an SEO crawler; review redirects, canonicals and structured data
After every major release	Check robots.txt, sitemaps, noindex tags and page speed

Frequently asked questions

Does llms.txt improve Google rankings?

There is no evidence that llms.txt is a Google ranking factor. Its purpose is to help AI systems understand your site. It is quick to create, so many businesses publish one alongside their XML sitemap.

Should I block AI crawlers?

It depends on your goals. If you want your brand and expertise to appear in AI answers, allow reputable AI crawlers on public pages. If your content is your product, such as paid research, you may choose to block some of them.

How long does technical SEO take to show results?

Fixes such as unblocking pages or correcting noindex tags can show results within days of recrawling. Speed and structural improvements usually show their effect over several weeks.

Is WordPress good for SEO?

Yes, when built well. A lightweight theme, a reputable SEO plugin, sensible caching and disciplined content structure give WordPress sites a strong technical base.

Get a technical SEO audit

If you would like a second pair of eyes on your site, Creensight’s digital marketing and SEO team can run a full technical audit — crawlability, speed, structured data and AI search readiness — and give you a prioritised fix list. You can also browse fixed-scope packages in our shop. Contact us to get started.