Home Antivirus Product Testing is Changing, Whether Vendors Like it or Not

Antivirus Product Testing is Changing, Whether Vendors Like it or Not

This week NSS Labs released their Q2 2010 Corporate Endpoint Protection Products report. NSS has only publicly announced the two products it specifically recommends against: Panda‘s Internet Security 2010 (Enterprise) and AVG‘s Internet Security Business Edition 9. However, it takes only a quick look at Trend Micro’s web site to guess how NSS rated Office Scan (hint: very well). Some vendors have protested NSS’s ratings in the past, but like it or not NSS is changing the way security testing is conducted.

As security threats evolve, e-mail has been displaced by the Web as the primary delivery mechanism for malicious code. The old model of virus definition based antivirus software has been increasingly called into question. In 2007, Australia’s Computer Emergency Response Teamclaimed that leading products missed 80% of new viruses. To compensate, companies like Trend Micro and Kaspersky are developing cloud based “reputation services” to evaluate URLs and code.

NSS president, and former VP of marketing at antivirus vendor ESET, Rick Moy explained in a phone interview: cybercriminals now typically use social engineering to trick users into downloading malware from web sites and run it voluntarily. Malware creators run “campaigns” on Twitter and other social media sites baiting users with anything from pornography to free iPads.

Even the most savvy of users can occasionally be tricked by social engineering – we posted our own list of tech savvy Twitter users who fell for phishing scam last year. For an explanation of how such savvy users get fooled, read Cory Doctorow’s recent Lotus Magazine piece explaining why he fell for a phishing scam.

Trend Micro recently published an independent report claiming the IT industry is being lulled into a false sense of security by vendors. The report cites an NSS survey which found half of respondents thought their antivirus solutions would protect them from threats 100% of the time, and that another 10% of respondents thought their solutions would protect them 99% of the time.

Moy says there’s also a perception in the enterprise that anti-malware products are essentially interchangeable, but that’s turning out not be the case: NSS’s testing found wild disparities between the efficacy of different products, and found that a company’s previous track record is no indication of how well it will perform.

NSS tests differ from most other testers, such as AV-Comparatives for example, in that NSS’s test computers that are actually connected to the Internet – something the company calls “live testing.” NSS tested PCs, running up-to-date copies of Windows 7 and using Internet Explorer 8 with SmartScreen disabled, by visiting known malicious sites to verify whether each product tested could successfully blocks malware from being downloaded and/or executed.

According to Moy, typical testing involves using malware sets from Wildlist or Antivirus Bulletin – both of which use samples provided by the antivirus protection industry, some of which might be quite old. The results of AV-Comparatives’ most recent report, its “Retrospective/Proactive Test,” are radically different from NSS’s. Trend Micro did fairly poorly in this test, while Panda did quite well. AVG out performed Trend Micro. Peter Stelzhammer of AV-Comparatives confirmed via e-mail that the NSS and AV-Comparatives tests are not comparable, and that AV-Comparatives is working on a test similar to NSS’s.

A consensus is forming in the security industry that there’s a need for new prevention techniques – and new testing methodologies to evaluate those techniques. The need for new testing methodologies was the theme of the “Measuring The Actual Security Anti-Virus Products Provide Customers” panel at SOURCE 2010 Boston in May, which included Stelzhammer, NSS CEO Vik Phatak, and representatives from CheckVir Labs, Dennis Technology Labs, PC Security Labs and West Coast Labs. Stelzhammer detailed the difficulties involved in doing live, Internet-connected testing and explained the methodologies for AV-Comparatives’ future NSS-like tests.

AV-Test has released a test based on methodologies similar to NSS’s, with comparable results. According to the report published by Trend Micro: “ICSA and others such as Virus Bulletin state they will be evolving their certification practices in coming months to include real-time testing and/or testing against today’s threats.”

NSS has a recent history of raising eyebrows in the security industry.

In March of 2009 NSS published the results of a Microsoft sponsored test that found Internet Explorer 8 was more effective in blocking malicious web sites than Firefox, Safarai, Chrome, and Opera. This lead to many skeptical articles and accusations of bias. However, when looking at what was actually being tested, the results aren’t particularly radical: NSS found that Microsoft’s blacklist blocked more sites than the blacklists used by other browsers. NSS was not tested for other browser vulnerabilities.

In September 2009, Network World reported that NSS was shifting its focus towards conducting self-funded tests instead of vendor sponsored tests. NSS would sell the reports and consult vendors, but would not take money for testing.

One of NSS’s first self-funded tests found 3com TippingPoint 10 firewall to be deficient. Moy told TechWorld he thought that Tipping Point must not have been investing enough in improving its products. Some commenters were, shall we say, skeptical about NSS’s rating.

That same month, NSS released its first Corporate Endpoint Protection Products test, unfunded and using its new live testing methodology. It ranked AVG, Panda and Moy’s former employers ESET at the bottom of the heap. It gave top marks to Trend Micro.

In March of this year NSS released a free report detailing the failure of many commercial products to defeat variants of the infamous Aurora virus that infected Google’s computers. McAfee was the only product NSS tested that successfully blocked variants of the virus, and NSS found AVG was the only product tested that didn’t block the original exploit.

In an entry on the company blog, AVG protested the results of the Aurora test and called a few things into question. It noted that the report initially indicated that NSS had tested AVG 8 instead of the newer AVG 9, and that NSS later claimed this was a typo and that it had tested version 9. AVG also claimed NSS gave them different information about the results of the testing before the publication of the report and that NSS was slow providing methodology before publication. AVG also provided a screenshot showing AVG blocking the Aurora virus.

ESET also fired back at NSS, claiming that NSS’s report didn’t comply with two of the Anti-Malware Testing Standards Organization’s Fundamental Principles of Testing. ESET also complained that NSS did not provide access to samples used in its testing.

Since NSS has moved ESET out of “caution” category in the newest report, we asked ESET if it had made any changes to its product to improve its ratings with NSS. “ESET was unable to get any useful feedback on its performance in the previous test without paying a substantial sum to NSS for ‘consultancy,'” replied David Harley, ESET Research Fellow & Director of Malware Intelligence via e-mail. “Even when ESET offered to pay the agreed-upon sum, the fee kept escalating. To this day, information on the test samples has never been supplied, so we are unable to assess the competence and validity of the test, let alone make any changes that would impact our performance in the more recent test.”

Moy answered critics on the NSS blog. First, he addressed AVG’s objections. Moy disputed AVG’s claims that NSS withheld information and points out that the methodology and steps for reproducing its Aurora testing was available in the published report. He provided a video showing AVG 9 failing to protect against the Aurora exploit, and pointed out that the screenshot AVG provided depicted Firefox, even though Aurora was an Internet Explorer virus.

In a post seemingly in response to ESET, Moy wrote:

Some vendors used the anti-malware testing standards organization (AMTSO) to try to discredit the test. One of their objections was that we recommend against buying products that scored on the bottom third of our test. Sorry, we unabashedly believe malware protection should indeed be the key purchasing criteria for an AV product. And for vendors who claim their anti-spam on the corporate desktop will improve their protection against socially-engineered malware hosted on web sites, that’s just stretching it.

Moy told us that all the NSS test methodology is available for free on the company’s web site and that ESET was mostly upset NSS didn’t release its malware sample set.

Although NSS has received money from vendors for past tests, and does do security consulting, Phatak told us NSS did not receive money from Trend Micro or McAfee before the tests were conducted.


Science is about repeatable, verifiable results. The only way to glean a better understanding of the efficacy of various endpoint protection products is for more labs to employ cutting edge testing methodologies. The more tests released in the future, the more information enterprises will have in making informed decisions.

About ReadWrite’s Editorial Process

The ReadWrite Editorial policy involves closely monitoring the tech industry for major developments, new product launches, AI breakthroughs, video game releases and other newsworthy events. Editors assign relevant stories to staff writers or freelance contributors with expertise in each particular topic area. Before publication, articles go through a rigorous round of editing for accuracy, clarity, and to ensure adherence to ReadWrite's style guidelines.

Get the biggest tech headlines of the day delivered to your inbox

    By signing up, you agree to our Terms and Privacy Policy. Unsubscribe anytime.

    Tech News

    Explore the latest in tech with our Tech News. We cut through the noise for concise, relevant updates, keeping you informed about the rapidly evolving tech landscape with curated content that separates signal from noise.

    In-Depth Tech Stories

    Explore tech impact in In-Depth Stories. Narrative data journalism offers comprehensive analyses, revealing stories behind data. Understand industry trends for a deeper perspective on tech's intricate relationships with society.

    Expert Reviews

    Empower decisions with Expert Reviews, merging industry expertise and insightful analysis. Delve into tech intricacies, get the best deals, and stay ahead with our trustworthy guide to navigating the ever-changing tech market.