<img height="1" width="1" style="display:none" src="https://www.facebook.com/tr?id=149034655494227&amp;ev=PageView&amp;noscript=1">

The HubSpotters Guide To Get Their Web Pages Crawled Effeciently On Google

May 11, 2017 | Dev Shah

As a HubSpotter, how many times have you faced a situation that a live page doesn’t appear on Google Search? Google is dependent on its “web crawlers” that explore the web at a certain period of time to find sites that could be indexed and entered in their database.

A huge majority of sites that are indexed on Google are added automatically as they were easily identified by the crawlers. It means if your website is not crawled on Google yet, it is definitely not a crawler-friendly website.

Even if your website is crawled by Google, are you confident that Google will understand the content of the website in order to place it in front of relevant searchers?

Google has given us some quality guidelines that will enable webmasters to optimize the technical aspects so that Google crawls and understands the content of the website.


How To Help Google Find Your Web Pages Built On HubSpot?


Google has provided certain guidelines that could be achieved easily on HubSpot in order to ensure quick crawling of your CMS web pages. In this article we will discuss the major points from this guidelines and how you could fulfil them via HubSpot to ensure that your pages are crawlable.


Provide A Sitemap File

If you have a HubSpot hosted website, you could add /sitemap.xml to the end of your domain name to view the sitemap. In case you want to add or remove pages from your sitemap to enable quick and efficient crawling, you could go to Content > Content Settings and click Sitemaps in the left sidebar menu.




Once you enter the Sitemap page, you will be able to make changes to specify what should be included and excluded from the sitemap.




In the example given above, we have excluded the very first page as it is a sample. The rest of the pages have been selected to be included in the sitemap.


Submit URL In Both The Versions

If you are looking integrate your domain with HubSpot, your DNS should provide a CNAME to HubSpot. The CNAME record provides global performance that is vital to access sites faster as the websites based on CMS are delivered via HubSpot’s local CDN to the geographic location where the page was requested.

Fully hosting your website on HubSpot, will require you to set up 301 redirects from the “non-www version” (http://example.com) to the www version (http://www.example.com). This 301 redirect is provided by the DNS. In case you don’t get it from the DNS, you will have to contact HubSpot support.

If the root domain is not connected with HubSpot, you will find difficulties in getting the pages crawled.


Structured Data

Structured data is highly organized that makes it highly comprehensive for search engines to understand the intent of a particular page. There is an added benefit of a good UX for searchers right from the search engine.






The structured data markup is included in the file that will be served by the web. At this moment, HubSpot doesn’t provide support for schema markup. However, it has not deprived developers from implementing a custom HTML module that could be used to add rich snippets. Developers could use the Schema.org markup language to define the content of their pages. Here is the page that helps you in creating schema markup.



Check Robots.Txt

You could check if you have applied no-index to the robots.txt file for this particular page. Aplying it, will prevent the search engines from crawling your web page. Your developer will help you out with this. For this purpose, they will have to go to the “Head Options” for that page and check out if they have applied the following meta tag:



The "Head Options" could be accessed for a particular page from the Design Manager section:


1. Go to Edit > Edit Head:




2. Locate the section titled "Additional HTML Head Markup Layout" to insert the above given meta tag:




Design For A Clear And Conceptual Page Hierarchy

All the websites should correspond with each other in order to maintain a visually similar look across the website. This will provide a clear outlook of your pages to the search engines that would result into efficient crawling and subsequent ranking. In order to apply the same template for multiple pages to apply conceptual similarity in web design, you could follow these steps:


  • Go to Design Manager and select the template that you would like to implement across various pages.
  • In the Actions menu, select “Create Page From Template.”
  • Select from landing page and website.
  • Specify the name for your new page.
  • Hit Create.

In order to know all the pages that are using a specific template, you could select the “Pages Using Template” option from the Actions menu.




Allow Crawling Of JavaScript and CSS

To help Google fully understand your site's contents, allow all site assets that would significantly affect page rendering to be crawled, including CSS and JavaScript files that affect the understanding of the pages. It is vital to use the public URL that is generated in HubSpot CMS. Once the JS or CSS is authored, you will need to get the URL to reference in your script tag to load your file.

You could copy the public URL for both JS and CSS file:







The tips given above will help in optimizing the page for optimal Google crawling. However, at time if you have correctly implemented all of the above steps and still not able to get the page crawled, you could use the following webmasters URL to submit the page manually. The time-frame is not decided, but Google will consider it for crawling if all the technical aspects of the page are good.



Hubspot COS/CMS


Dev Shah

Dev Shah

Dev is the Head of Marketing at HubMonks, a HubSpot content management system (CMS) partner for major inbound marketing agencies and brands. Dev utilizes and swears by the power of the HubSpot inbound marketing methodology. He loves to share valuable insights on HubSpot’s marketing software, content management system (CMS) and the best practices to set up successful inbound marketing campaigns.

Leave a Reply


Want to know how to optimize your website to get more traffic and leads? Ask our certified inbound marketing specialist for a free one time assessment.

Recent Posts