Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > HTML > Search Engine Tag?

Reply
Thread Tools

Search Engine Tag?

 
 
Scott
Guest
Posts: n/a
 
      11-20-2005
Is there a tag that I can put on a page that will prevent search
engines from indexing the page? The page is not behind a password.

Thanks!
Scott
 
Reply With Quote
 
 
 
 
Luigi Donatello Asero
Guest
Posts: n/a
 
      11-20-2005

"Scott" <(E-Mail Removed)> skrev i meddelandet
news:(E-Mail Removed)...
> Is there a tag that I can put on a page that will prevent search
> engines from indexing the page? The page is not behind a password.
>
> Thanks!
> Scott



As far as I know you could insert the adress of the page into a file
called robots.txt and
indicate which search engine you do not want to index it.
But I tried to use this method to stop Google indexing some images and they
are still there, so I do not know whether it works, actually.

--
Luigi Donatello Asero
https://www.scaiecat-spa-gigi.com/de...rre-kueche.php



 
Reply With Quote
 
 
 
 
Beauregard T. Shagnasty
Guest
Posts: n/a
 
      11-20-2005
Scott wrote:

> Is there a tag that I can put on a page that will prevent search
> engines from indexing the page? The page is not behind a password.


<head>
<meta name="robots" content="noindex,nofollow">
....

--
-bts
-Warning: I brake for lawn deer
 
Reply With Quote
 
bernhard
Guest
Posts: n/a
 
      11-20-2005

Scott schrieb:

> Is there a tag that I can put on a page that will prevent search
> engines from indexing the page? The page is not behind a password.


You'll find the anserw here: http://www.robotstxt.org/wc/exclusion.html

--

bernhard

 
Reply With Quote
 
Steve Pugh
Guest
Posts: n/a
 
      11-21-2005
Luigi Donatello Asero wrote:
> "Scott" <(E-Mail Removed)> skrev i meddelandet
> news:(E-Mail Removed)...
> > Is there a tag that I can put on a page that will prevent search
> > engines from indexing the page? The page is not behind a password.

>
> As far as I know you could insert the adress of the page into a file
> called robots.txt and
> indicate which search engine you do not want to index it.
> But I tried to use this method to stop Google indexing some images


You should have used robots.txt before Google index the page not
afterwards.

"Prevention is better than cure" and all that.

> and they are still there, so I do not know whether it works, actually.


Wait a few months for Google to get around to reindexing your pages and
if you've used robots.txt properly the images should be removed. Or use
the URL removal form on Google to remove them now.

Steve

 
Reply With Quote
 
Scott
Guest
Posts: n/a
 
      11-21-2005


Steve Pugh wrote:
>
> Luigi Donatello Asero wrote:
> > "Scott" <(E-Mail Removed)> skrev i meddelandet
> > news:(E-Mail Removed)...
> > > Is there a tag that I can put on a page that will prevent search
> > > engines from indexing the page? The page is not behind a password.

> >
> > As far as I know you could insert the adress of the page into a file
> > called robots.txt and
> > indicate which search engine you do not want to index it.
> > But I tried to use this method to stop Google indexing some images

>
> You should have used robots.txt before Google index the page not
> afterwards.
>
> "Prevention is better than cure" and all that.
>
> > and they are still there, so I do not know whether it works, actually.

>
> Wait a few months for Google to get around to reindexing your pages and
> if you've used robots.txt properly the images should be removed. Or use
> the URL removal form on Google to remove them now.
>
> Steve


Steve,

OK, I figured out what to write in robots.txt. What I'm wondering is exactly
where to place that file on the host server.

Scott
 
Reply With Quote
 
Steve Pugh
Guest
Posts: n/a
 
      11-22-2005
Scott wrote:
> > Luigi Donatello Asero wrote:
> > > "Scott" <(E-Mail Removed)> skrev i meddelandet
> > > news:(E-Mail Removed)...
> > > >
> > > > Is there a tag that I can put on a page that will prevent search
> > > > engines from indexing the page?
> > >
> > > As far as I know you could insert the adress of the page into a file
> > > called robots.txt and indicate which search engine you do not want to
> > > index it.

>
> OK, I figured out what to write in robots.txt. What I'm wondering is exactly
> where to place that file on the host server.


At the root of your site.

If a spider wants to visit http://www.example.com/foo/bar/page.html
then it will look for http://www.example.com/foo/bar/robots.txt,
http://www.example.com/foo/robots.txt and
http://www.example.com/robots.txt and apply all the rules it finds.
>From your point of view having a single robots.txt in your root folder

makes for easy maintenance.

Steve

 
Reply With Quote
 
=?iso-8859-1?Q?Kim_Andr=E9_Aker=F8?=
Guest
Posts: n/a
 
      11-22-2005
Steve Pugh wrote:

> Scott wrote:
> > > Luigi Donatello Asero wrote:
> > > > "Scott" <(E-Mail Removed)> skrev i meddelandet
> > > > news:(E-Mail Removed)...
> > > > >
> > > > > Is there a tag that I can put on a page that will prevent
> > > > > search engines from indexing the page?
> > > >
> > > > As far as I know you could insert the adress of the page into
> > > > a file called robots.txt and indicate which search engine you
> > > > do not want to index it.

> >
> > OK, I figured out what to write in robots.txt. What I'm wondering
> > is exactly where to place that file on the host server.

>
> At the root of your site.
>
> If a spider wants to visit http://www.example.com/foo/bar/page.html
> then it will look for http://www.example.com/foo/bar/robots.txt,
> http://www.example.com/foo/robots.txt and
> http://www.example.com/robots.txt and apply all the rules it finds.
> > From your point of view having a single robots.txt in your root
> > folder

> makes for easy maintenance.


Where did you get that idea?
http://www.robotstxt.org/wc/exclusion-admin.html

<quote>
Note that there can only be a single "/robots.txt" on a site.
Specifically, you should not put "robots.txt" files in user
directories, because a robot will never look at them. If you want your
users to be able to create their own "robots.txt", you will need to
merge them all into a single "/robots.txt".
</quote>

--
Kim André Akerĝ
- http://www.velocityreviews.com/forums/(E-Mail Removed)
(remove NOSPAM to contact me directly)
 
Reply With Quote
 
Steve Pugh
Guest
Posts: n/a
 
      11-22-2005
Kim André Akerĝ wrote:
> Steve Pugh wrote:
>
> > If a spider wants to visit http://www.example.com/foo/bar/page.html
> > then it will look for http://www.example.com/foo/bar/robots.txt,
> > http://www.example.com/foo/robots.txt and
> > http://www.example.com/robots.txt and apply all the rules it finds.
> > > From your point of view having a single robots.txt in your root
> > > folder

> > makes for easy maintenance.

>
> Where did you get that idea?


Empirical evidence. Maybe out of date. Maybe robots now follow the
standard, they certainly didn't always. It's been a long time since I
maintained a site that didn't have access to the server root so I
haven't had any direct experience of this part of robots behaviour for
over several years.

> http://www.robotstxt.org/wc/exclusion-admin.html
>
> <quote>
> Note that there can only be a single "/robots.txt" on a site.
> Specifically, you should not put "robots.txt" files in user
> directories, because a robot will never look at them. If you want your
> users to be able to create their own "robots.txt", you will need to
> merge them all into a single "/robots.txt".
> </quote>


Learn something new every day.

Steve

 
Reply With Quote
 
Scott
Guest
Posts: n/a
 
      11-22-2005


Steve Pugh wrote:
>
> Scott wrote:
> > > Luigi Donatello Asero wrote:
> > > > "Scott" <(E-Mail Removed)> skrev i meddelandet
> > > > news:(E-Mail Removed)...
> > > > >
> > > > > Is there a tag that I can put on a page that will prevent search
> > > > > engines from indexing the page?
> > > >
> > > > As far as I know you could insert the adress of the page into a file
> > > > called robots.txt and indicate which search engine you do not want to
> > > > index it.

> >
> > OK, I figured out what to write in robots.txt. What I'm wondering is exactly
> > where to place that file on the host server.

>
> At the root of your site.
>
> If a spider wants to visit http://www.example.com/foo/bar/page.html
> then it will look for http://www.example.com/foo/bar/robots.txt,
> http://www.example.com/foo/robots.txt and
> http://www.example.com/robots.txt and apply all the rules it finds.
> >From your point of view having a single robots.txt in your root folder

> makes for easy maintenance.
>
> Steve


Steve,

So, you're saying I can just upload the robots.txt file to the same place I
upload all my website files? In my case, my web account on the server is
"public_html". And I should configure robots.txt to exclude the one
particular url that I wish not to be indexed?

Thanks!
Scott
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Getting search details from web search engine pandi Java 5 12-14-2009 04:45 AM
Search jobs at world's largest online job search engine. Findresources for all types of solomanjo@gmail.com Computer Support 0 03-13-2008 12:00 PM
Search jobs at world's largest online job search engine. Findresources for all types of solomanjo@gmail.com Digital Photography 0 03-13-2008 11:49 AM
.Net Search Engine - Has anyone used dtSearch .Net Engine? Sasha ASP .Net 3 05-22-2007 04:20 PM



Advertisments