http://en.wikipedia.org/wiki/Web_crawler
RBSE [3] was the first published web crawler. It was based on two programs: the first program, "spider" maintains a queue in a relational database, and the second program "mite", is a modified www ASCII browser that downloads the pages from the Web.
1 comment:
Check with online crawler to find out how many links a crawler finds in your website...
http://www.mallutalk.com/bmallu/bcrawler/
Post a Comment