How to track the success or failure of your anti-virus tools? More importantly, what metrics can enterprises use to determine whether their malware defenses are working better today than, say, a week ago? Identifying anti-malware characteristics that are worth measuring is harder than you might think.

A Case for Security Metrics

Without measuring the effectiveness of your controls, how can you know whether you are spending the right amount on your information security program, or that your funds are being spent on the right tools or processes? Tracking metrics related to security controls gives CISOs and business executives the ability to steer the security program in the right direction. Without metrics, the security program exists as an art project, rather than an engineering or business discipline.

In the book Security Metrics, Andrew Jaquith highlights the following characteristics of a good metric, stating that it needs to be:

• "Consistently measures, without subjective criteria
• Cheap to gather, preferably in an automated way
• Expressed as a cardinal number or percentage, not with qualitative labels like 'high,' 'medium,' and 'low'
• Expressed using at least one unit of measure, such as 'defects,' 'hours,' or 'dollars'
• Contextually specific—relevant enough to decision-makers so that they can take action"

Tracking the Number of Caught Infections Isn’t Very Useful

When asked how to track the effectiveness of an enterprise anti-malware program, people often propose tracking the number of infections caught by anti-virus software. However, this turns out to be not very helpful. A change in this number in either direction doesn't provide us with many insights:

• If the number of caught infections goes up, this might be because we’re using improved detection capabilities. Or it might be because we're being sent more malware, so there's more of it to detect.
• If the number of caught infections goes down, it might be because our detection capabilities have worsened. Or it might be because we're being sent less malware, so there's less of it to detect.

The number of caught infections can be driven by factors external to the enterprise. It doesn't tell us whether our anti-virus tool is doing a great job or whether we need to swap it out. It doesn't provide actionable feedback, but merely serves as a reminder that malware is out there and that anti-virus tools can catch some of it.

Useful Metrics for Measuring Enterprise Malware Defenses

Perhaps a more effective way of tracking the status of anti-virus tools is to look at the following metrics across the enterprise:

• What percentage of systems have anti-virus tools installed and enabled on them?
• What percentage of systems have up-to-date anti-malware signatures?
• What percentage of systems have specific anti-malware tool or feature installed and active (e.g., anti-spyware, browser protection, etc.)

An enterprise can also measure its ability to respond to malware infections by using the following metrics, tracking the average time across incidents in a given time period:

• How quickly do system administrators respond to an anti-virus alert?
• How long does it take to return to business after a system needs to be reimaged due to a malware infection?
• How long does it take to recover from a bad anti-virus signature that adversely affects systems in the enterprise?
• How long does it take to fix anti-virus software installation on a system where the software isn't functioning properly?
• How quickly does the anti-virus vendor issue a custom signature in response to a submitted sample?
• How long does it take to disconnect an infected system from the network after deciding to do so?

An enterprise can also track the following composite metrics related to its malware defenses. These are more complex, and I need to give them more thought:

• What is the ratio of malware incidents identified using anti-virus tools vs. other means? If the enterprise seeks to decrease reliance on anti-virus tools, this metric might be helpful, though it is weakened by the external attack factors I mentioned in the beginning of this post.
• What percentage of known infected systems had a high-severity vulnerability that was unpatched? This metric might help the enterprise track the severity of infections: the incident was probably more sever if it occurred on a highly-vulnerable system. The goal might be to lower the figure.
• On what percentage of known infected systems did the user have local administrator privileges? This metric might also help measure the severity of incidents: malware on a system where the user has admin rights is likely to do more damage. The goal might be to lower this figure.

What do you think of this? Can you recommend other metrics for measuring enterprise malware defenses?

Update: I followed up this post with a note that examined More Metrics for Measuring Enterprise Malware Defenses.