eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
Automated Web systems, or bots, conducting a variety of reconnaissance and attack activities accounted for 29 percent of all Web traffic in 2014, according to a report published on Dec. 18 by Web security firm Incapsula.
The greatest proportion of bad bot traffic, about 22 percent, were automated impersonators, which gather information from Web sites while posing as legitimate Web crawlers from companies such as Google. Hacking tools scanning for exploitable vulnerabilities made up more than 3 percent and programs attempting to scrape data from targeted sites made up another 3 percent, according to the report.
While smaller sites tend to see a much larger proportion of bot traffic and fewer human visitors, all sites garner roughly the same amount of attention from bad bots, Marc Gaffan, CEO and co-founder of Incapsula, told eWEEK.
Good bots account for half of all traffic to smaller sites, but a little more than a quarter of traffic to the largest sites. Bad bots account for 31 percent of traffic to the smaller sites and a similar amount—27 percent—to the largest sites, according to Incapsula.
“Any Web site out there on the Internet gets a certain amount of bad bot traffic,” he said. “The same guys are out there scanning every single Web site trying to find ways to get in and to steal information.”
Over the past year, bot traffic has subsided, dropping by about 10 percent, according to the report, which is based on data from more than 15 billion visits to approximately 20,000 websites monitored by Incapsula. Automated Web programs accounted for about 56 percent of traffic to Web sites in 2014, down from more than 60 percent of traffic in 2013.
The decline of the RSS reader may account for much of the drop, according to Incapsula. RSS bot traffic dropped to 27 percent in 2014, from 31 percent of all good bot traffic, according to Incapsula. Google shut down its Google Reader service in July 2013.
Most of the bad bot traffic attempts to impersonate good bots as a way of circumventing site security. Impersonators include distributed denial-of-service attacks that are camouflaged as browsers and Web site scanners that use proxies to hide their location. Most of the Web site probes and attacks impersonate Google’s page-ranking bots because hardly any Web site would block Google, according to Gaffan.
“Your worst nightmare is when Google stops searching for your data,” he said. “The last bot you will ever block from coming to your Web site is Google, so from a bad guy perspective, that is the bot that everyone wants to impersonate.”
While overall bot traffic has declined over the past year, impersonator bot traffic has risen by about 10 percent, according to Incapsula