My initial post on measuring the effectiveness of enterprise malware defenses generated very helpful feedback, which I’d like to share in this note. Good metrics provide an objective way of understanding the extent to which the measured security controls are working. I proposed a number of metrics that would help the organization to keep an eye on its anti-malware efforts.
The Scope of Collected Metrics
Jennifer Bayuk pointed out the importance of carefully deciding the scope of data that will be collected and the actions that will be taken as the result. For instance, consider the following metric I proposed earlier:
On what percentage of known infected systems did the user have local administrator privileges?
If the enterprise is already certain that local administrator privileges increase the severity of infection, then it might broaden the scope of this metric to include all systems, rather than measuring only at the infected ones.
I prefer tracking administrative privileges only on known infected systems because it’s often impractical to strip away admin rights everywhere. And maybe that’s OK, if in a particular organization the users who have admin rights don’t get infected anyway. I’d rather focus on those situations where the user gets infected while possessing administrative privileges.
Infection Characteristics on Workstations
Another friend, who prefers to stay anonymous, recommended tracking additional characteristics of infections on workstations:
- Where on the local system was the malware present? Was it in the Temporary Internet Files folder? Java cache folder? Email temporary attachment folder? Tracking the number of infections that involved these locations can help you understand which of your malware defenses are failing (browser, Java, email, etc.)
- What is the ratio between the number of malware samples detected in real time vs. scheduled scans? Most organizations prefer to catch malware in real time, as the user attempts to save or execute the malicious program. Detecting the presence of malware later during a scheduled scan indicated that the system is already infected; removing the malware might not fully clean the host.
Additional Malware Metrics
Phil Waterbury recommended additional metrics for measuring enterprise malware defenses, which included:
- What percentage of systems are reinfected within 3 days of the initial infection?
- How long does it take to deploy a custom anti-virus signature across the enterprise?
Phil pointed out that to collect and track such metrics, the enterprise needs to standardize the process its help desk follows when responding to, classifying and tracking malware-related incidents. He also emphasized the importance of standardizing on the approach for handling malware incidents from the perspective of tools and techniques—otherwise the collected metrics won’t be consistent across incidents.
Thanks for everyone’s feedback on the topic of malware metrics! If you’re interested in learning more on the topic, take a look at the paper Security Metrics: An Overview by Clare E. Nelson (PDF). Oh, and did I mention that I teach a course on combating malware in the enterprise?