A guide for beginners on technical SEO
Technical SEO plays a crucial role in SEO until it doesn’t. The crawlability and indexability of pages are essential for ranking, but when compared to content and links, other actions may have minimal influence.
What does technical SEO mean?
Technical SEO refers to the optimization of a website to facilitate its discovery, crawling, comprehension, and indexing by search engines such as Google. The primary objective is to increase visibility and enhance search engine rankings.
How complex is technical SEO?
Well, it varies. The basics are not too challenging to learn, but technical SEO can be intricate and difficult to comprehend. In this guide, I will attempt to simplify things as much as possible.
How does crawling function?
Crawling is the process by which search engines extract content from web pages and follow links to discover additional pages. There are various methods to regulate what gets crawled on your website. Here are some possibilities:
Robots.txt
A robots.txt file specifies which sections of your website search engines can and cannot access.
DID YOU KNOW?
Even if links point to inaccessible pages, Google may index them. This can be perplexing, but if you want to keep pages from being indexed, check out this guide and flowchart, which can assist you through the process.
Crawl rate
Many crawlers support a crawl-delay directive in robots.txt, which enables you to specify how frequently they can crawl pages. Unfortunately, Google does not follow this.[1] To adjust the crawl rate in Google Search Console, you’ll need to do it for Google.[2]
Access restrictions
If you want a page to be accessible to certain users but not search engines, you’ll probably need one of these three options:
A login system of some kind
HTTP authentication (which necessitates a password for access)
IP whitelisting (which allows only particular IP addresses to access the pages)
This type of configuration is ideal for internal networks, member-only content, or for staging, test, or development sites. It allows a specific group of users to access the page while preventing search engines from accessing it and indexing it.
How to monitor crawl activity
To see what Google is crawling, use the “Crawl stats” report in Google Search Console, which provides additional information on how your website is being crawled.
If you want to monitor all crawl activity on your site, you’ll need to access your server logs and possibly employ a tool to analyze the data more effectively. This can become quite sophisticated. However, if your hosting comes with a control panel like cPanel, you should be able to access raw logs and some aggregators like AWstats and Webalizer.
Crawl Configuration
Each website has a unique crawl budget, which is determined by a combination of how frequently Google wants to crawl the site and how much crawling the site can handle. Pages that are popular or frequently updated will be crawled more frequently, while less popular or poorly linked pages will be crawled less often.
If crawlers encounter difficulties while crawling your website, they will typically reduce their crawl speed or even halt the crawl until the issue is resolved.
After pages are crawled, they are processed and added to the index, which is the primary list of pages that can be returned in response to search queries. Let’s discuss the index in more detail.
Directives for Robots
A robots meta tag is an HTML code snippet that instructs search engines on how to crawl or index a specific page. It is inserted into the <head> section of a webpage and appears like this:
<meta name=”robots” content=”noindex” />
Canonicalization
When a website has multiple versions of the same page, Google will select one version to store in its index. This process is known as canonicalization, and the URL that is chosen as the canonical URL will be displayed in search results. Google uses a variety of signals to determine the canonical URL, including:
Canonical tags
Duplicate pages
Internal links
Redirects
Sitemap URLs
The simplest way to check how Google has indexed a page is to use the URL Inspection tool in Google Search Console, which will display the Google-selected canonical URL.
Prioritizing SEO tasks can be challenging as there are numerous best practices to follow, but certain modifications can have a more significant impact on your website’s rankings and traffic than others. Below are some of the top priority projects that I would suggest for SEOs.
Ensure that the pages you want to be discovered by users are indexable on Google. The preceding sections were focused on crawling and indexing for a reason. You can utilize the Indexability report in Site Audit, which is free in Ahrefs Webmaster Tools, to determine pages that cannot be indexed and the underlying reasons.
Over time, websites tend to change their URLs, causing old URLs to become obsolete. These old URLs may have links from other websites that are no longer valid and do not count towards your pages. However, you can reclaim lost value by redirecting these old URLs to current pages. It’s a quick and effective way to build links.
To identify opportunities to reclaim lost links, use Ahrefs’ Site Explorer. Simply enter your domain, go to the Best by Links report, and apply a filter for “404 not found” HTTP responses. Sort the results by “Referring Domains” to prioritize the most impactful opportunities.
By examining the initial URL in archive.org, it can be observed that the page was previously dedicated to Mother’s Day. By implementing a 301 redirect from the old page to the current version, you can regain 225 links from 59 unique websites. There are numerous other prospects to take advantage of.
To recover this lost value, it is necessary to redirect all old URLs to their current locations using a 301 redirect.
Including internal links on a website can assist in discovering pages and improve their ranking. Ahrefs’ Site Audit offers a tool named Internal Link Opportunities that identifies internal linking opportunities by searching for keywords that the site already ranks for.
For instance, if a page mentions “product reviews,” and there is a separate page on the site that talks about “product reviews,” the tool will suggest adding an internal link from the former page to the latter page to increase the website’s internal linking structure.
In order to improve your website’s search engine visibility, you can add schema markup to your website’s code. This code helps search engines better understand your website’s content, enabling it to display rich snippets, answer boxes, and other search features. To learn more about the different search features and the schema required for your website to be eligible, you can visit Google’s search gallery.
The page experience signals refer to the factors that have a lower impact on ranking, but are still essential to consider for enhancing the user experience (UX) of a website.
To ensure a good user experience, Google’s Page Experience signals include Core Web Vitals, which are speed metrics. These metrics consist of Largest Contentful Paint (LCP) for visual load, Cumulative Layout Shift (CLS) for visual stability, and First Input Delay (FID) for interactivity.
To ensure the security of communication between a browser and server, HTTPS is used which prevents attackers from intercepting and tampering with it. HTTPS provides confidentiality, integrity, and authentication to the vast majority of today’s web traffic. It is recommended to load your web pages over HTTPS instead of HTTP.
You can identify whether a website is using HTTPS by looking for the “lock” icon in the address bar.
To ensure that webpages are properly displayed and easily used on mobile devices, it is important to check their mobile-friendliness. You can use the “Mobile Usability” report in Google Search Console to assess how mobile-friendly your site is.
Interstitials are elements that obstruct the content of a website, often in the form of popups that cover the main content, and require user interaction before they can be dismissed.
Hreflang is an HTML attribute utilized to indicate the language and geographic focus of a webpage. Suppose you have several versions of the same page in different languages. In that case, you can use the hreflang tag to inform search engines such as Google about these variants, which helps them serve the appropriate version to their users.
These tasks may not significantly affect your website’s rankings, but they are essential for ensuring a smooth user experience and maintaining the health of your website.
Broken links refer to links on a website that no longer work or point to non-existent resources. These can be either internal, i.e., links to other pages within the same domain or external, i.e., links to pages on other domains.
To find broken links on your website, you can use the Links report in Site Audit. This tool is available for free in Ahrefs Webmaster Tools.
To identify redirect chains on your website, you can use the Redirects report in Site Audit. This tool is available for free in Ahrefs Webmaster Tools. Redirect chains are a sequence of redirects that occur between the initial URL and the final destination URL.
Google Search Console is a free tool provided by Google that helps website owners monitor and resolve issues related to their website’s appearance in Google search results. It allows you to identify and fix technical errors, submit sitemaps, and resolve structured data issues, among other things.
To determine the ease of use of a page on a mobile device, Google’s Mobile-Friendly Test is used. It also detects specific mobile usability issues such as the use of incompatible plugins and text that is too small to read.
The test allows you to see what Google observes when crawling your page. Furthermore, the Rich Results Test can be used to see the content that Google views for desktop or mobile devices.
Chrome DevTools is a web development and debugging tool that is built into Chrome. It can be used to identify and fix page speed issues, enhance webpage rendering performance, and perform a variety of other tasks that are useful from a technical SEO perspective.
The Ahrefs’ SEO Toolbar is a complimentary extension for Firefox and Chrome that delivers valuable SEO insights about the web pages and websites you browse.
It provides free functionalities including:
An on-page SEO report
A redirect tracer complete with HTTP headers
A broken link checker
A link highlighter
SERP positions
Furthermore, as an Ahrefs user, you also have access to:
SEO metrics for every site and page you visit, including Google search results
Keyword metrics like search volume and Keyword Difficulty directly in the SERP
SERP results export
PageSpeed Insights is a tool that evaluates the loading speed of your web pages. It provides a performance score and actionable suggestions to help improve loading times.
Important points to keep in mind:
Search engines won’t display your content in their results if it’s not indexed.
It’s crucial to fix any issues that affect your search traffic, but improving your content and building links are usually more important.
Projects that have the greatest potential to boost your rankings often involve improving indexing or link-related factors.