Why does Steve appear in my server logs?

Shareaholic powers content distribution, discovery and sharing of web content for over 450 million people each month. Part of how this works involves the temporary display of images or details related to the web content, such as the title and excerpt for the page. Our system retrieves this information for pages included in the Shareaholic content index.

Steve is Shareaholic's web crawling bot (sometimes also called a "spider"). Crawling is the process by which Steve discovers new and updated pages to be added to the Shareaholic content index. Steve's crawl process begins with a list of URLs, generated from previous crawl processes and augmented with Sitemap data provided by webmasters. New pages, changes to existing pages, and dead links are noted and used to update the Shareaholic content index.

How does Steve identify itself?

Mozilla/5.0 (compatible; Shareaholic/1.0; +http://www.shareaholic.com/steve)

How does Steve access sites?

We do very aggressive caching so you shouldn't see Steve in your logs more than once every few weeks for any one URL. Our goal is to crawl as many pages from your site as we can on each visit without overwhelming your server's bandwidth. We don't like waste either, and are clever at not wasting bandwidth – yours or ours. However, if you experience technical issues, please contact us at support@shareaholic.com. We recommend attaching any relevant server logs that may help.

Can I change Steve's crawl rate?

Shareaholic has sophisticated algorithms that determine how much to crawl each site. If Steve is crawling your site too often and slowing down your server you can change the crawl rate. Use the "crawl-delay" directive in your robots.txt file to control the crawl rate. The use of this setting is not always needed nor is it generally recommended, but it is available for use should the need arise.