In our Using a robots.txt File we went over how to minimize "bots" crawling your entire site and/or directories within your site. In this article we'll go over how to discourage "bot" traffic from an individual file (instead of an entire directory).
The following is an example of the "no follow, no index" code you can use in an individual page:
<html>
<head>
<title>...</title>
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
</head>
When using the <META> tag, keep the following in mind:
Just like any other <META> tag, the <META NAME="ROBOTS"> tag should appear between the opening and closing <HEAD> tags in the HTML code of your page.
Let's look at the example tag again:
<META NAME="ROBOTS" CONTENT="NOINDEX, FOLLOW">
While you can use multiple comma separated values for the "CONTENT" attribute, only certain combinations make sense and should actually be used. For instance, if no <META NAME="ROBOTS"> tag appears, the default behavior is "INDEX, FOLLOW" so it is not necessary to explicitly state that particular combination.
That leaves us with the following combinations:
<META NAME="ROBOTS" CONTENT="NOINDEX, FOLLOW">
<META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW">
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
Keep in mind that this tag is best suited if you only have one or a few pages you want to discourage "bots" from crawling as the tag will need to be inserted on every page you want it to apply to. If you have many files or entire directories you want to control "bot" traffic on, the robots.txt file is a better option.
| Email: | support@WebHostingHub.com | Ticket: | Submit a Support Ticket |
|---|---|---|---|
| Call: | 757-416-6627 | Chat: | Click To Chat Now |
We value your feedback!
There is a step or detail missing from the instructions.
The information is incorrect or out-of-date.
It does not resolve the question/problem I have.
new! - Enter your name and email address above and we will post your feedback in the comments on this page!