How to Set Your Site's Crawl Rate in Webmaster Tools
In a previous article, Using a robots.txt File, we touched briefly on setting a crawl-delay through a directive in the robots.txt file. While some search crawlers do recognize that directive, Google's search crawler (Googlebot) does not. Adjusting the Googlebot crawl rate is handled within Google's Webmaster Tools.
Googlebot crawling your site is normally considered a good thing. Issues can arise if crawls become excessive and the server your account is negatively affected. Fortunately, by reducing the crawl rate in Google Webmaster Tools, you can reduce the impact of Googlebot on the server.
NOTE: There are three important points to keep in mind when setting the crawl rate in Google Webmaster Tools:
- Setting the crawl rate in Google Webmaster Tools sets the speed of Googlebot requests during each crawl process. It does not affect how often Googlebot comes back to crawl your site or how deeply your site structure is crawled.
- You can only set the crawl rate for the main domain or a subdomain (yourdomain.com or subdomain.yourdomain.com). You cannot set a separate crawl rate for a subfolder (yourdomain.com/subfolder).
- When you set a new crawl rate in Google Webmaster Tools, it will only be valid for 90 days. To keep your reduced crawl rate, log back into Google Webmaster Tools and adjust your crawl rate if it has been more than 90 days since you last adjusted it.
Changing Your Crawl Rate in Google Webmaster Tools
- Begin by logging into your Webmaster Tools account.
- In the home page, select your site. If you have more than one site, select the site you want to change the crawl rate for.
- On the left side of the next page, under Site configuration, click on Settings.
- Under Crawl rate, select "Set custom crawl rate and adjust the slider to a slower crawl rate.
- Click the Save button to save your new crawl rate. As you can see in the image below, we reduced our crawl rate by 33% in this example.
If you have any questions or need any assistance regarding reducing your GOOGLE crawl rate, please feel free to submit a question in our Support Center.
Did you find this article helpful?
Tweet
Category
Google Tools
Comments
59 Points
|
2021-08-05 1:52 am
Hi, thanks for sharing the information post, please respond when i check my URL in the google URL inspection tool having https://stainely.com/how-to-bleed-off-an-overcharged-ac-easy-steps/ inspection tools show URL is not on Google, however is index in google,
|
Help Center Login
Related Questions
Here are a few questions related to this article that our customers have asked:
Ooops! It looks like there are no questions about this page.
Would you like to ask a question about this page? If so, click the button below!
We value your feedback!
There is a step or detail missing from the instructions.
The information is incorrect or out-of-date.
It does not resolve the question/problem I have.
new! - Enter your name and email address above and we will post your feedback in the comments on this page!