Each search engine comes with its own proprietary web crawler like Bing with Bingbot and Google with Googlebot. These bots are also widely known as spider, robot or web crawlers. It is designed to scrape data from webpages, index and show in the search results pages. Because Google is still the most important search engine, it is important that your website is well-optimized for its web crawler, the Googlebot.
Optimizing WordPress Site for Googlebot
Google continuously improves its algorithm and web crawling performance. It means, you can no longer use outdated methods to fool web crawler to gain higher SEO performance. However, you do not need to break your head or hire a SEO agency for optimizing a WordPress site. There are some basic stuffs that you can do it yourself as a website administrator to improve SEO performance.
1. Check XML Sitemap
Sitemap is essential for Googlebot to quickly find pages on the website and it’s an important way to inform Google about newly published content. Fortunately, WordPress by default comes with XML Sitemap though there are lot of SEO plugins available to create your own. The problem is that some plugins do not redirect default WordPress Sitemap to the plugin generated one. Therefore, make sure that your website only has a single Sitemap, and you submit the correct Sitemap in Google Search Console. You may consider adding a separate Sitemap if you have a separate sections like deals with plenty of webpages and categories.
If you are using manual Sitemap, make sure that your Sitemap does not contain dead 404 pages that are changed or deleted by chance. You can use free broken link check tools and setup 301 redirects to fix the errors.
2. Improve Robots.txt
Check the root directory of your website to locate the robots.txt file. When Googlebot crawls your website, robots.txt is the first thing that it looks for. It is important that your robots.txt file is well optimized for SEO performance. By default, you do not need to add any directives in robots.txt file. However, it is good to insert a Sitemap link so that search engine bots can find the location of your XML Sitemap especially if it is stored outside root directory.
You can also use robots.txt file to block admin and other unnecessary folders so that Googlebot do not waste time in crawling those pages. In addition, your developer or hosting company might have added a disallow command accidentally, which blocks Googlebot from accessing a portion of your website. You can simply open robots.txt file using yoursite.com/robots.txt URL in the browser and confirm there are no mistakes in the file. If you find any problem, you need to use FTP or File Manage app from hosting account to modify and update the file. It is also possible to edit robots.txt file in WordPress using plugins like Yoast SEO.
Be aware that even after you correct this, it may take a couple of weeks or more before Googlebot recrawl your website and organic traffic starts to go up.
3. Use Clean URL Taxonomy
WordPress by default uses webpage’s title for the URL. Because the title has strong keywords, its use in the URL may lead to improved user experience and higher rankings. Googlebot will know about the topic of the webpage by assessing the keywords it uses. However, you can change the URL structure by going to “Settings > Permalinks” section and choose different setup. Make sure to use post name or any other simple structure without using autogenerated characters in the URL. In addition, you can make use of breadcrumb navigation to inform search engines and users about the exact location of your current page. this is useful when you have multiple categories covering different topics in the same site.
4. Use Canonicalization
For large websites, like e-commerce portals, duplicate webpages can be a big problem, because they may impair SEO performance. However, there are different reasons why duplicate webpages are needed, as an example if different languages are necessary. When you have multi-language webpages, make sure to use hreflang attribute to identify the correct language used on your page. For product and other duplicate pages, you can use canonical meta tag. This will inform Googlebot about the page to be considered for indexing while consider other pages as duplicate.
Most of the popular SEO plugins like Yoast allows to add canonical tag from the post editor. You have to provide the original content URL in the meta box which will be added in the page’s header section to instruct search engines.
5. Using Schema
It is easier for Googlebot to understand your website’s context if you use structured data. However, when using Schema, make sure that it follows the guidelines of Google. For better efficiency, JSON-LD is a recommended solution for structured data markup. In fact, Google has confirmed that website administrators should use JSON-LD for their markup language. Check out our article on how to add schema markup in WordPress sites.
6. Optimize Site Speed
Google has specified that site speed is an important indicator for ranking factors. If your website loads too slowly due to heavy visual content or unoptimized codes, Googlebot will flag it for a lower ranking. You can check the loading speed of your website using any page speed measurement tools like Google PageSpeed Insights. If your website still loads too slowly, find the possible causes and fix with the followings:
- Use caching plugins like WP Rocket for enabling page caching and browser caching.
- There are CDN like Cloudflare to deliver your content from the nearest location for your visitors.
- Reduce bloating on your site by using lightweight themes like GeneratePress and avoid using heavy page builder plugins.
- Remove unnecessary scripts and CSS from your site by using optimization plugins like Perfmatters.
Final Remarks
Unlike other website building platforms, WordPress offers complete solution to optimize your site for Googlebot and other search engines. However, you also need to avoid using plenty of plugins, bloated themes and use Google optimization guidelines for webmasters.