spiders

[taylrl]

I was just looking at the home page, and i noticed that most of the people on this website are spiders. i know what a spider is, but that is about as far as my knowledge goes. Are they deployed automatically? Do they move? if so how do they decide where to stop, and when to go? A bit more info would be appreciated.

[5614]

There has been a few threads on spiders, specifically google spiders, on this forum. Try using the search button... and google. Ask back if you have a specific question.

[Nezumi]

Back in the day those things rarely looked at a website. Ever since google came in, it seems like there is a militia of them.

[RyanJ]

I was just looking at the home page, and i noticed that most of the people on this website are spiders. i know what a spider is, but that is about as far as my knowledge goes. Are they deployed automatically? Do they move? if so how do they decide where to stop, and when to go? A bit more info would be appreciated.

 

Its actually quite interesting!

 

Depending on their job these can be called trawlers too.

 

The way most work is simple (The example below is how Google's bots work)

 

1) Start on a page scan the page's source for links. These are then added to a temp table for later use. (Google also downloads caches images and some file types for easy loading).

 

2) Grab the links to any media on the page and add them to a record archive. If these already exist then the count is incrimented.

 

3) Next load the temporary table and then repeat the process for each of the found links.

 

I left out about googles rating system because it makes things far more complex.

 

Do they move as I explained above they follow links and by doing so move out in an ever widening grid.

Do they stop this is a bit harder. If they originbate from a small company then they are limited by how far they can spread the net because this is quite memory intensive, for something like google the net size is all but limitless. So they only stop by memory limitations

 

Hope I helped explain it a bit better if you have any questions I'll do my best to explain what I wrote above... hope it makes sence

 

Cheers,

 

Ryan Jones