Website crawlers often visit your website during the entire life cycle of your website. They look for new content on your website, follow links, and collect data to better understand how it serves search intent.
These crawlers may encounter errors when browsing your website, such as duplicate content and broken links. Fixing these errors as soon as possible is very important to keep your website healthy and your ranking on Google.
In this guide, we have included 5 serious crawl errors and how to fix them.
Mistake 1: The internal link is broken
Internal links lead users from one page on your website to another page. Internal links are good for SEO, Can help establish the site structure and spread link rights.
Broken links may be caused by wrong URLs or deleted pages. In these cases, the website has not been updated to reflect the changes.
Why is a broken link a problem?
Broken links will prevent users from browsing your site, which can have a negative impact on the user experience (a key factor in your site’s ranking).
Broken links also prevent crawlers from browsing and indexing your website. Due to crawling millions of websites, each robot can only crawl a limited number of pages before it must move to the next website. This is often referred to as your “crawl budget.”
If your link is broken, robots will not be able to crawl your site and will consume your “budget.”
How to fix broken links
you can use it Site audit tool Identify your broken link. After running a site review, you can filter any link errors indicated by the tool:
To run an audit, you need to set up a project for your domain. Navigate to “Project” under “Management” in the main toolbar on the left:
If you have not set up a project for your site, please create a new project by selecting the “Add New Project” button in the upper right corner of the page:
You will be prompted to enter the domain of the website and name the project:
You will be taken to the project dashboard where you can select the site review tool:
Use site audit tools to:
- Delete the link to the deleted page.
- Determine which page the broken link should point to, and update the link to make it work.
- Run the site audit again to check if the update was successful.
Check our guidelines to avoid other internal linking issues The most common internal link building errors.
Mistake 2: Mixed content
When the code of a web page is loaded with HTTPS (Secure Hypertext Transfer Protocol), but the content on the page (images, videos, etc.) is loaded with HTTP, mixed content problems will occur.
Why is mixed content a problem?
When this issue occurs, users will receive a warning pop-up window indicating that they may be downloading unsafe content. This may prompt the user to leave the page, leading to a high bounce rate.
Mixed content can negatively impact the user experience and indicate to Google that there is a security issue with your page.
How to solve the mixed content problem
You can use the site audit tool to find any pages with mixed content issues:
You need to replace all insecure HTTP links on your website with newer and more secure HTTPS links, and then use the same HTTPS protocol link to embed your resources.
Mistake 3: Duplicate content
When web crawlers find that the content on your website is 85% or more similar to other content on your website, Google will mark it as duplicate content.
This can happen if you use the same content in multiple languages to serve different countries/regions around the world, or if you have multiple landing pages with similar content.
Why is duplicate content a problem?
When search engines see duplicate content, they may be confused as to which version to crawl or index, and may exclude both at the same time.
Search engines may think that you are trying to manipulate algorithms and may lower your ranking or prohibit your website from appearing in search results.
Linked assets may also be diluted due to duplication, which will affect your overall page authority score.
How to fix duplicate content issues
After running a site review, you can use the site review tool to see how many duplicate pages your site has:
To fix them, you can:
- Add canonical label Go to the page you want Google to crawl and index. The canonical tag indicates the original page to search engines, so they will know which page to display in the SERP.
- use 301 redirect From the copied page to the original page.
- Instruct Googlebot to use Google Search Console to process URL parameters in different ways.
The easiest way to avoid this problem is to create unique content for each page of the website.
Mistake 4: Orphaned pages
Orphan pages will not internally link to any other pages on your website. Users cannot access these pages through your website’s main menu, sitemap, or links to other pages.
The only way to access orphaned pages is through direct links.
Why is orphaned pages a problem?
If the search engine cannot find an orphaned page, the page will not be indexed on the SERP.
Adding an orphaned page to your site.xml file wastes your crawling budget because Google’s robots will take longer to find and crawl it.
How to fix orphaned pages
The site audit tool indicates orphaned pages in the site.xml file:
To do this, click the “Questions” tab and then the “Select Questions” button:
If your website is no longer needed or no longer useful, delete the orphaned page.
If you still want to keep this page, please add an internal link to it from another page on your website and make it discoverable in the menu system.
For more details on this process, please read about how to find Orphaned page with site audit.
Mistake 5: Lack of backlinks and toxic backlinks
The function of backlinks is similar to online recommendations. When an external website links to your website, you will get support. In the eyes of search engines, they are a sign of trust and authority.
Having a link from an external website with good page permissions will transfer some permissions back to your website.
Why is the lack of backlinks and toxic backlinks a problem?
The lack of backlinks means that your website has almost no page permissions, and you are not an authority in your field.
Toxic backlinks are links from websites, they will weaken your page authority and damage your SEO ranking. You need to avoid backlinks from mirror sites, low-privileged domains, or poorly laid out sites.
How do you solve the problem of missing backlinks or toxic backlinks?
You will be from Backlink audit tool The first thing to find is where both types of links exist.
If you have already created a project for your domain, you will be able to run the backlink review tool from your project dashboard. After opening, the tool will prompt you to set the crawl range, target country/region, etc.:
After the review is complete, you will be able to see how many backlinks your site has gained, the domains that point to your site, and which domains are harmful:
The backlink review tool “ranks” and sorts harmful backlinks from 0 to 100, where 0 is the best score.
Semrush performed more than 50 tests to arrive at a toxicity score. You can view these checks by hovering your cursor over the toxic link.
How you deal with these toxic links depends on the score and determining why they are toxic.
You can put them on the delete list to delete them manually, or you can ask Google to reject them so that they won’t affect the page ranking of your website.
If you find that some links have low toxicity scores, but you know that these sites are not toxic, you can add these links to your whitelist.
For more details on backlink review, please refer to our knowledge base.
Semrush tool to reveal site errors
There is an incredible range of Semrush tools to help you solve various technical SEO problems, including site review tools, backlink review tools, and content analyzers.
Website audit tool
This Website audit tool Check more than 120 pages and technical issues when crawling your website. Use this tool to check common issues such as broken links, hreflang implementation, crawlability, etc.
Monitoring the health of your website through site audits is an important process and should be done regularly. Site audit can help you find site errors early and often, and the site audit tool will provide suggestions to help you solve any problems it finds.
You can crawl up to 100 pages per month for free.View our site review tool guide on our website knowledge base.
Backlink audit tool
Backlink audit tool Check the number and quality of backlinks to your website. You can then whitelist or reject the backlinks you find to help improve the page authority of your website.
You can also further analyze toxic backlinks and use the tool to request removal of such links from external sites.
[create-campaign destination_url=” https://www.semrush.com/backlink_audit” show_input=”false” header=”Find and fix toxic backlins” text=”with the Backlink Audit tool” button_text=”Try for Free” bg_images=” https://static.semrush.com/blog/uploads/media/96/fd/96fd495930d058321d5f78f5554aff88/trial-superbanner-1.png” bg_button=”-success”]
The content analyzer tool works in 2 steps.The first one is content review Content review tool.
This tool reviews the content of your website by browsing the sitemap and checking the URLs and internal links in subfolders. It needs your sitemap or robots.txt file to run this report.
In the example above, the tool shows that there are 364 URLs that need to be reviewed under the “Blogs” subfolder.
After the review is completed, the tool will return the report:
You will have four categories to deal with:
- Rewrite or delete: Add pages that were published 24 months ago and viewed less than 15 times in the last month. You can choose to delete the page or refresh the content.
- Need to update: Pages that have been published more than 6 months and have been viewed less than 15 times in the past month are all placed here. You may need to refresh the content or find new ways to internally link to this content so that it becomes more useful.
- Quick review: These are the most recently published pages that have been visited more than 150 times in the past month. These pages are performing well, so please take notes for future content or use them as inspiration for refreshing old content.
- Bad content: There are fewer than 200 words on these pages. It is a page of less than 200 words. This is too little for Google. Try to repurpose social media content or expand to at least 600 words.
The world of search engine optimization is fiercely competitive and you need to gain all the advantages. Set yourself apart from the competition by solving critical errors that may affect your website.
If our audit tool returns a large number of errors, don’t be overwhelmed. Step by step, fix each error in a logical and stable manner, and run follow-up audits to ensure that your work has an impact.
For a more in-depth guide to crawlability errors, check out our article Fix crawlability issues.