They discover Web sites by “crawling” from URL to URL, and since they shift by webpages, the bots scrutinize the primary text and supplementary areas of the site. Search engines like yahoo are fairly secretive with regards to the systems they use and consistently change them to enhance search results. https://thestargateproject.io