Get The Latest Insights

By Jessica Ellis | November 22, 2016

Frustrating, isn’t it?

You put all that effort into designing a security awareness training program… 


But is it helping keep your organization safe? Or is it just satisfying your compliance requirements?

The truth is you have no idea. After all, how can you measure return on investment (ROI) for something intangible like security awareness training?The Foundation of ROI

There’s one factor above all others that crushes your ability to measure ROI: A lack of objectivity.

Without an objective way of measuring success, there is simply no way to accurately measure ROI. So before you do anything else, ask yourself this question: How do we win?

Whatever process or system you’re designing, it’s essential to know in advance how you’ll measure results. Can you measure how many dollars are made? Or how many new clients are signed? Or how much time is saved?

If not, you’ll never have an objective way to measure ROI.

And if all you care about is satisfying compliance requirements, that’s fine. Checking the boxes is simple enough.

But if what you really want is to improve security awareness (and behaviors) throughout your organization, measuring the results of your program is vital. You need to know precisely how the program is performing compared to your expectations, so you can adjust the content and delivery accordingly.

Susceptibility: A Strong Contender

There are dozens of ways to measure success. For security awareness training, you could track time spent remediating avoidable security events, or the number of potential phishing emails reported by users.

But if you really want to measure ROI accurately, we suggest an alternative approach. Instead of measuring your users’ response to ‘real’ security events, simply create your own.

Start by taking steps to understand the current likelihood that, given an email-borne attack, a cybercriminal will gain access or cause damage to your network. In other words, conduct a baseline assessment to know how many malicious emails will be clicked by your employees. 

To calculate susceptibility, simply divide your total failures by the total number of tests performed. For example, if you send 100 phishing emails to your users and that results in 30 click-throughs, the calculation would look like this: 30 ÷ 100 = 30%

It makes sense that if you’re trying to minimize incidents of users clicking on phishing links, phish them. Create your own customized phishing and spear phishing campaigns, send them to your users, and track the susceptibility rate. If your program is working, this rate will fall over time.

By taking this approach, you can safely gain an objective insight into the impact of your training program. After all, the best time to identify weaknesses in your program isn’t when an unauthorized stranger walks into your offices and sits down at an unlocked terminal. The best time to identify those weaknesses is when the ‘stranger’ is actually a member of your security team, who can track the fall in poor security behaviors (susceptibility) over time. 

The better trained your employees are in recognizing spear phishing attempts, the lower your PSR (Phishing Susceptibility Rate) will be and the lower the costs of phishing attacks.

TCO.pngAnd as your users’ susceptibility to different forms of attack falls, you’ll have concrete figures to evidence the ROI of your program.

Sweat the $$$

We know, we know. Statistics are all well and good, but what your executive board really wants to know is whether your program is saving the organization money.

We’ve got you covered.

One of the strongest arguments for measuring susceptibility is that when combined with existing breach statistics, it becomes a simple task to estimate savings.

Take phishing susceptibility. Last week we published a post demonstrating a simple method for calculating the cost of phishing to your organization. Using the very same method (and model) you can quickly and easily measure the effectiveness of your program at preparing users to identify phishing emails.

An organization with 5,000 employees, for example, will spend an average of $225,844 per year on phishing incident response at a susceptibility rate of 30 percent. If your program is able to reduce users’ susceptibility to 5 percent, though, that figure plummets to $37,641, resulting in a saving of $188,203.

To calculate your ROI, then, simply plug these same figures into the basic ROI formula: 

 ROI = (Gain from investment-Cost of investment) / Cost of investment

For the sake of argument, let’s say your security awareness training program costs $100,000 per year. In that case: 

  ROI = ($188,203-$100,000) / $100,000 = 88%

And this doesn’t just work for phishing. So long as you can estimate the cost of a single security incident and measure your susceptibility (failure) rate, the costs, savings, and ROI calculations are a simple task.

Bringing Calculations Into the Real World

Now of course, it’s not always going to be this easy.

The phishing susceptibility model used to inform the above example took us some time to produce, and makes use of the latest research findings from a number of separate primary sources. If you wanted to construct something similar for a different attack vector, it wouldn’t be a five minute job.

But all hope is not lost. With a little investigation, you’ll have all the information you need to estimate the average cost of a security incident in terms of incident response. By our calculations, incident response costs a 5,000-employee organization around $100 per hour, with the average incident taking approximately 5 hours to resolve. That’s $500 per incident.

According to Verizon, an average of 3.5 percent of security incidents result in a breach, and the average breach costs $3.5 per employee. That’s $17,500 for our 5,000-employee organization.

Knowing all this, then, the final stage of the calculation becomes simple. Let’s imagine you’re currently experiencing 100 security incidents per year as a result of unsafe browsing behaviors.

Based on the cost estimates above, you’re currently spending $50,000 on incident response and $61,250 on breaches as a result of these unsafe browsing behaviors.

IR cost = $500 x 100 annual incidents = $50,000

Breach cost = 3.5 annual incidents x $17,500 = $61,250

Total = $50,000 + $61,250 = $111,250

You might develop a testing program that periodically presents users with simulated pop-ups to click on. By tracking click rates, you’ll see over time how much your users’ behaviors improve, and you can use those figures to evidence the ROI of your program.

If, then, you can reduce users’ susceptibility by 50 percent, you can half your annual costs in this area: a $55,625 saving. If the program costs $30,000 annually to enact, your ROI will look like this:

   ROI = ($55,625-$30,000) / $30,000 = 85%

Start Strong, Finish Strong

It’s notoriously difficult to evidence the need for investment in security awareness training.

And ultimately, while the above calculations are simple, they rely completely on your ability to objectively measure success. If you have a concrete (or at least evidence based) way to track susceptibility, measuring ROI is simple.

But if you don’t have this, your ROI calculations will be nothing more than a ‘best guess’.

For this reason, it’s imperative that you start your security awareness training program knowing exactly how you’ll measure success. Whether it’s click rates or something else, having this metric in place from the start will make it far easier to evidence the need for further investment. 

More on Quantifying Costs to Your Organization

For an individual organization, quantifying that potential impact of a spear phishing attack beforehand can be difficult. This can often lead to underinvestment in crucial security initiatives, leaving the organization exposed.

The PhishLabs Cost of Phishing Susceptibility model is a resource that helps security leaders easily quantify the cost of phishing attacks while showing how that cost can be substantially lowered by reducing the organization’s vulnerability to phishing.  We will demonstrate the model during the presentation. 

Watch the on-demand webinar to learn about:

  • How to quantify the potential cost of phishing attacks to your organization. 
  • Rate of compromises due to phishing attacks.
  • Costs to respond to incidents. 
  • Data breach costs to your organization.


Additional Resources: