JazzJackrabbit Community Forums - A question regarding spiders

JazzJackrabbit Community Forums (http://www.jazz2online.com/jcf/index.php)

- Site and Forum Rules, Questions & Feedback (http://www.jazz2online.com/jcf/forumdisplay.php?f=14)

- - A question regarding spiders (http://www.jazz2online.com/jcf/showthread.php?t=11238)

cooba

Sep 10, 2004 11:37 AM

A question regarding spiders

Yesterday, two spiders from MSN came and searched for JJ3 stuff. Today, there are a lot of Google spiders wandering around randomly. Weren't they disabled on JCF?

Tubz	Sep 10, 2004 12:21 PM

What the hell is the difference between a Google Spider and a normal Google search bot?

Bobby aka Dizzy

Sep 10, 2004 12:29 PM

I believe the MSN spider chooses to ignore the request to not cache our pages.

Link	Sep 10, 2004 12:48 PM

Quote:

Originally Posted by Bobby aka Dizzy

I believe the MSN spider chooses to ignore the request to not cache our pages.

Actually, the robots.txt file was lost in the server crash.

Bobby aka Dizzy

Sep 10, 2004 04:12 PM

Well I believe certain versions of the msnbot still choose to ignore it but that's true too.

KRSplat

Sep 11, 2004 06:43 AM

Quote:

Originally Posted by JJ Tublear

What the hell is the difference between a Google Spider and a normal Google search bot?

Spiders search websites and cache every single page that there is a link to. I don't really know what I'm talking about, though.

All times are GMT -8. The time now is 10:14 AM.