Crawler

Also referred to as a robot or spider, it is the part of the search engine that searches (or "crawls") the Web by following links to web pages and storing and indexing those URLs

More info on Search Engine Optimization