The story reads like something out of the Marvel universe, only with more pocket protectors and less leather. Google has a special super secret1 group of computer fraud hunters that work out of London, include a Russian engineer and is headed by the awesomely named Douglas de Jager. de Jager not only has a name that sounds like it was made for comic books, he has a backstory that sounds like it came right from Stan Lee2.
Following the creation and sale of BytePlay, a content scraper and something that he openly admits could be used for nefarious purposes, the South African created Spider.io, a fraud fighting company.
After selling BytePlay, Mr. de Jager decided to found Spider.io to fight those dark uses. “I wanted to try to prevent anyone from ever using technologies like the technologies I had built previously to do evil things,” he said.
So, what exactly are the 100 or so members of the team doing? Mostly looking for botnets. The online advertising world offers incredible opportunities for the less than honest to make money due to its immense size and the amount of money involved. In September 2013 alone, Ghostly determined that Google served 316 billion ad impressions, and the numbers have probably risen significantly since then. Trying to find fraudulent activity in that much data is like trying to find a needle in a needle stack.
The system works like this (if you’re a bad guy): A ton of money can be made by hosting ads and getting both clicks and views. You can greatly boost the amount of money by having bots appear to surf these sites and click on links. Obviously, having actual humans do this would be incredibly expensive and time consuming, which is why computers do this.
However, the cheapest computer and the one that can’t be traced is a computer that isn’t yours. Using malware to infect people’s computers, botnets, which are frequently referred to as “zombie armies”, are created. This gives bad actors3 the tools they need to make it appear that actual human beings using computers to surf web sites and click on ads.
When a computer is infected, several browsers can operate in the background and surf sites without the computer owner being aware of it, even when they are currently using the computer. These views and clicks in turn create huge money for the websites that are paying for these fraudulent services.
If you’re wondering why Google wouldn’t simply take the payments for the clicks and impressions and call it a day, the reason is that there are a lot of other ad networks around, and it’s crucial that advertisers feel they’re displaying their ads on a safe network and not wasting their money. This is why Google is spending so much time and effort to do something that may cut into some of their own profits.
One of the main ways that the team looks for botnets is looking for mistakes made by the creators of the bots. One of the few examples that the team were allowed to share was that ZeroAccess, which has since been killed, reset a browser’s cookie field to a space instead of 0.
There are also behavior patterns that may show up that are not normal to human activity but are to bots, even those that are programmed with a variety of possible actions. In the image below, we can see that ZeroAccess bots tended to click on links on the outside of pages, which deviates from the normal behaviors of actual random human actions.
Thanks to the awesome computing power provided by Google, in the form of Powerdrill, a computing system that is able to in under five seconds compute half trillion cells of data, these patterns can be searched for throughout the Google ad network. It is essential that they have rock solid evidence of wrongdoing because once Google has identified a botnet, the website that displayed the ads receives no money and the advertisers are not charged for the clicks or views.
In spite of the fact that Google is making inroads in finding fraudulent activity and has said that they will be sharing their findings and methods with other companies, the reality is that a ton of this kind of activity is probably never detected.