A Web Bot or Web Robot is a bit of software that indexes websites for search engines such as Google or Bing. They read a page indexing it’s content. They then follow links form that page and index any other pages they can find.
Once a page has been “crawled” it is then made available on the search engine.
Also known as Web crawlers or Web spiders.