McAfee SiteAdvisor Blog

When we started SiteAdvisor last spring, we thought that our job would be relatively straightforward: sign up for stuff and download stuff and tell you the results. We knew it would be hard to do in practice, but at least it was a relatively predictable problem to tackle. Along the way towards implementation, we realized that the nature of the Web’s sketchy and suspicious practices was more complex, less transparent and more dangerous than any of us first thought.

Michael Kearns is a computer science professor at the University of Pennsylania. Before he joined UPenn, he spent a decade doing artificial intelligence and machine learning research at AT&T Labs and Bell Labs. He’s one of a handful of true pioneers in these fields.

Now, one of the byproducts of the millions of tests our Web bots conduct is an enormous data set we’ve built, not just of adware bundles or spam factories, but of relationships between Web sites. Michael and his grad student Jenn Wortman helped us approach this data in a novel way. Take a look at Screensaver.com for a second.

We initially rated screensaver.com ‘Green’ – safe to use for browsing, signing-up and downloading. Yet after downloading screen savers from here, our PC started popping up contextual ads.

Here’s what’s really happening:

From my user perspective, I’m on a site called screensaver.com, downloading a piece of software from them. From a technical perspective, however, my PC is actually calling a host computer run by freeze.com. Not only don’t I notice this, but even if I do, it won’t help. As an average user, I don’t know anything about freeze.com.

But our database does. What Michael and Jenn helped us realize is that we could use the data from our Web crawl to help users understand where they really are on the Web. This guidance will in turn help users make better, more informed decisions about whom and what they can trust online.

Defining Links
Enter Matt Gattis, a young developer who joined us from MIT. “What defines a ‘bad’ link?" Matt asked. He developed an algorithm for measuring the degree of association between two sites by looking at their linking relationships. And because machines running Matt’s code can’t be fooled by link obfuscation and other social engineering tricks, SiteAdvisor is able to see patterns and relationships that were effectively invisible to the human eye. What we’ve done with link analysis is make the Web more transparent. In fact, we think we’ve created something kind of cool.

The Weakest Links
Here’s how SiteAdvisor’s link analysis works in practice. Take a look at our link diagram for Screensaver.com:

Among many other things, our link analysis shows some basic relationships between sites. For example, the short arrow to freeze.com documents that the biggest ‘target’ for screensaver’s out-bound links is freeze. (In fact, freeze bought screenscaver.com in 2003 from risoftsystems, another red flagged friend. According to a freeze.com press release the sale included a five year “sponsorship contract highlighting RISS products.")

Improving the Odds
In an ideal world, users get full disclosure. Web sites not only tell the user what’s being installed, they disclose where the install is coming from in a way that’s meaningful to the non-technical user. I for one am not holding my breath. As a practical matter, without our link data, users are effectively browsing while blind. Clicking through to an unknown site is like betting it all on black. Heaven forbid if the marble lands on red. I’m not here to argue against aimless browsing; I love the serendipitous Web discovery. The problem with surfing blindly is that within three or four clicks, you can find yourself in places where all safety bets are off.

With SiteAdvisor, I know if the site I’m on engages in link practices that can land me in hot water. Browsing with our link analysis data is like going to a party where the only person you know happens to be the most social person in the room. He can tell you who’s friends with whom, who’s hooking up and who has trouble holding their liquor. Good person to know.