How You Can Start Learning Malware Analysis

Malware analysis sits at the intersection of incident response, forensics, system and network administration, security monitoring, and software engineering. You can get into this field by building upon your existing skills in any of these disciplines. As someone who's helped thousands of security professionals learn how to analyze malware at SANS Institute, I have a few tips for how you can get started.

Understand Where You Currently Fit Into the Malware Analysis Process

There are several ways to describe the skills you to analyze malicious software. I like grouping them in 4 categories, which I detailed in the post Mastering 4 Stages of Malware Analysis. I'm outlining them below, so you can begin thinking about your current capabilities in these areas:

The analysis steps flow from the bottom of the pyramid (easiest) to the top (most challenging), though people vary the order depending on their skills and other factors, and often revisit some of the steps as they uncover new details about the specimen:

  1. Fully-Automated Analysis: Run ("detonate") the suspicious file in an automated analysis environment ("sandbox") to get a report on its activities, such as its interaction with the file system and network.
  2. Static Properties Analysis: Examine metadata and other details embedded in the file (e.g., strings) without running it, so you can spot the areas you might want to examine more deeply in subsequent steps.
  3. Interactive Behavior Analysis: Run the file in an isolated laboratory environment, which you fully control, tweaking the lab's configuration in a series of iterative experiments to study the specimen's behavior.
  4. Manual Code Reversing: Examine the code that comprises the file, often with the help of a disassembler and a debugger, to understand its key capabilities and fill in the gaps left from the earlier analysis steps.

Memory, file system, and network forensics efforts (when applicable) also contribute to the understanding.

Ask yourself, "What skills do I have today, and where do they fit into the malware analysis process?" Start experimenting with the area where you feel most comfortable, and progress from there. The following resources can help you move forward.

Review and Learn From Others' Analysis and Findings

If you're just getting into malware analysis, you can start by examining the reports published by the more experienced analysts and automated sandboxes. As you review these details, note which aspects of the analysis make sense to you, and which areas require further study. This is one way for you to generate a learning plan.

If you're wondering where to find malware analysis reports, here's one starting point: Look at the sources of the posts I make on the LearnREM page I maintain on Facebook. There, you'll see website and blogs that I like to read to keep up with the industry. (You can view it even without signing into Facebook.)

When reading these reports, you'll come across some malware sample that you'd like to examine more deeply. Make a note of the hash and other attributes of that file, then look for that file's report in public malware analysis sandboxes.

You can access several malware analysis sandboxes for free. Search them for the malware you wish to explore; chances are good that they've already analyzed that file. Examine the reports, paying attention to the flagged behaviors that indicate that the file might be malicious. Look for Indicators of Compromise (IOCs) that would help you spot that malware in the wild.

Start Experimenting With Malware in Your Lab

Set up a lab that allows you to experiment with malware in an isolated, controlled environment. I shared my recommendations for doing this in the blog post 5 Steps to Building a Malware Analysis Toolkit Using Free Tools. You can use virtualization software to set up Windows and Linux systems to facilitate your analysis.

Next, consider watching my Practical Malware Analysis Essentials talk, which will offer practical tips for analyzing Windows malware in your lab:

In addition to watching the video above, consider reviewing the corresponding slides. Use these materials to repeat in your own lab the analysis I demonstrated. If you'd like a copy of the malware sample I used in this talk, reach out to me. (Be sure to exercise caution when experimenting with real-world malware like this.)

In addition to the Windows-based approach to examining malware, which the video above demonstrates, consider bringing Linux into your environment. The REMnux toolkit is designed for this very purpose, and is available as a virtual machine and a Docker container. You can see REMnux in action in my video Analyzing Windows Malware on Linux: Getting Started Tips and Examples.

In addition to watching the video above, consider reviewing the corresponding slides. Use these materials to repeat in your own lab the analysis I demonstrated. If you'd like a copy of the malware sample I used in this talk, reach out to me. (Be sure to exercise caution when experimenting with real-world malware like this.)

REMnux includes a variety of malware analysis tools. You can learn about the types of activities you may need to perform by reviewing the categorized listing of the tools on REMnux. For example, you'll find a variety of tools for static properties analysis, including FLOSS, which helps identify risky API calls in Windows malware, as documented in a Malware Triage post by Xavier Mertens.

Dive Deeper Into Code-Level Analysis

Examining malicious software at the code level is often the most challenging and time-consuming part of the process, especially when you're disassembling compiled programs. If you try to learn assembly on its own, it might feel too dry and discouraging. For this reason, I recommend getting to know assembly in the context of a debugger.

The following Introduction to Malware Analysis video might feel a bit dated, but is worth your attention. It offers another set of steps (and a malware sample) you can recreate in your lab. Moreover, it provides a glimpse at how you can analyze code dynamically by running the specimen in a debugger. In this demo, I used the now-outdated debugger OllyDbg, but this approach applies to the modern x64dbg alternative:

As this video shows, you can start dynamic code analysis of a Windows executable by setting breakpoints on risky API calls inside a debugger. This can bring you to the code worth examining, at which point you can begin figuring out the logic of the malicious assembly instructions. I listed some API calls worth considering in the Reverse-Engineering Malicious Code cheat sheet. The tool FLOSS, mentioned earlier, can help also help with spotting them.

Some malware you'll encounter will be designed to evade detection and analysis. You can learn about such techniques, and how to examine them, my video Evasion Tactics in Malware from the Inside Out and review the corresponding slides. One of the tools I use in this demo is x64dbg, so the video will also help you get started with this powerful debugger:

To deepen your understanding of code-level analysis, start experimenting with reviewing malicious code statically, without running it in a debugger. You can do this using Ghidra, which includes a disassembler and decompiler. For an introduction to this powerful tool, see the Code Analysis With Ghidra video by Anuj Soni, and review his accompanying blog post:

Learn to Examine Malicious Scripts and Document Files

In addition to being able to analyze compiled binaries, you should learn how to examine malicious scripts and documents. Such forms of malware are often used to deliver other malicious payloads and can directly implement the attacker's logic.

For an overview of the steps you can take to examine malicious scripts, such as JavaScript, watch Evan Dygert's presentation Shortcuts for Understanding Malicious Scripts and review the accompanying slides and malware samples:

For insights into examining malicious PowerShell artifacts, watch Mari DeGrazia's talk Finding and Decoding Malicious Powershell Scripts:

To learn about analyzing malicious Microsoft Office documents, you can start with this in-depth video by Didier Stevens, titled Analyzing Malicious Office Documents:

You can learn more about analyzing malicious documents and practice in your own lab by following Ryan Chapman's workshop Understanding and Analyzing Carrier Files.

You'll also find my Analyzing Malicious Documents cheat sheet useful.

Continue Reading, Experimenting, and Learning

Reading and watching the resources mentioned above will help you learn about malware analysis approaches, but you'll need to find time for focused, deliberate practice to learn how to apply them. That's why the tips I mentioned offer pointers to several ways in which you can start practicing.

Once you're ready to go deeper into the field, consider taking the Reverse-Engineering Malware course, which I've co-authored and teach along with other experienced security practitioners.

There's a plethora of articles, blog posts, and videos that can help you get into malware analysis, beyond what I've shared with you. You'll find them with a few online searches, but here are a few additional pointers:

Updated

About the Author

I transform ideas into successful outcomes, building on my 25 years of experience in cybersecurity. As the CISO at Axonius, I lead the security program to earn customers' trust. I'm also a Faculty Fellow at SANS Institute, where I author and deliver training for incident responders. The diversity of cybersecurity roles I've held over the years and the accumulated expertise, allow me to create practical solutions that drive business growth.

Learn more