When analyzing malware discovered during a security incident, the investigator often formulates indicators of compromise (IOCs): the signs of infection that can help the enterprise determine what systems may may been compromised. The incident responder might create a signature for the malware sample he or she examined. How can the organization look for this malicious file across the file systems in its environment without waiting for its antivirus vendor to generate the signature?

Unfortunately, no traditional antivirus tools that I’ve encountered allow its users to use custom signatures. That’s a pity, since the enterprise could have used the AV engine already deployed across its IT infrastructure to scan the file system for IOCs. Fortunately, I’ve come across 3 free tools that an organization can use to scan files using a custom signature: ClamAV, YARA and Vscan.

ClamAV for Custom Malware Signatures

ClamAV is a free antivirus engine. Its Unix version allows the user to create custom signatures for files based not only on their cryptographic hash, but also to fingerprint file sections, match specific byte sequences, use wildcards, and combine signatures according to Boolean rules.

ClamAV seems well-suited for scanning file systems for signs of identified malware samples if you can run the scan from a Unix host. (In this use-case, you’d ignore the signatures that ClamAV comes with.) Maintainers of the ClamAV project created a manual to document the process of creating signatures for ClamAV.

YARA for Custom Malware Signatures

YARA is a free tool for “helping malware researchers to identify and classify malware samples.” Like ClamAV, it can scan files using custom signatures, looking for byte sequences and strings; its signature syntax also supports regular expressions and conditionals.

YARA can runs on most operating systems, and is also available as an extensible Python library. You can use it to create custom malware signatures. The tool's website provides several sets of signatures that could be used as starting point to learn about creating your own.

YARA is a very popular approach to creating custom malware signatures. A project for exchanging YARA signatures is Yara Exchange Group.

Vscan for Custom Malware Signatures

Vscan is a free toolkit for “making fast but crude measurements of the prevalence of named textual features in algorithmically selected samples of large corpora.” In other words, it can scan files to identify those that match user-specified patterns. It’s designed to run on Unix systems.

Vscan is shipped with a custom signature file for identifying local web pages that match common malware signatures; this file can be a starting point for understanding the tool’s signature-creating syntax, along side the documentation that is available on the tool’s website. In addition to being able to identify the files that match custom signatures, the tool includes components that generate reports that can scale across a large number of findings.

Update: Vscan hasn’t been updated for sometime since its initial release and is included here primarily to illustrate the underlying concept and perhaps to inspire others to implement similar techniques or improve upon existing tools that implement them.

Conclusion

Perhaps some day traditional antivirus vendors will allow the administrators to deploy custom signatures using the engines already installed on most systems in the enterprise. In the mean time, ClamAV, YARA and Vscan are free tools for identifying the files that match IOCs relevant to a particular security incident. These tools are an excellent addition to an incident responder’s toolkit.

If this topic is interesting to you, take a look at the following resources:

• I discussed the role that these tools play in defining context-specific signatures for incident response on the SANS forensics blog.
• A related effort in the context of Indicators of Compromise is OpenIOC, which includes IOC Editor and IOC Finder.
• To learn how to analyze malware so that you can create custom signatures, see my Reverse-Engineering Malware course at SANS Institute.