A Framework for Evaluating and Patching the Human Factor in Cybersecurity

The presentation discusses a study on measuring security awareness of users in mitigating social engineering attacks. The study uses an expert-based procedure to rank criteria for different types of attacks and develops three complementary solutions to measure those criteria continuously and objectively. The endpoint solution, network-based solution, and attack simulation framework are used to derive a profile of the user and evaluate their success rate in mitigating attacks. The study concludes that self-reported behavior of subjects may differ significantly from their actual behavior, and security awareness scores derived from objective measures are highly correlated with the user's success in mitigating social engineering attacks.

• An expert-based procedure is used to rank criteria for different types of attacks
• Three complementary solutions are developed to measure those criteria continuously and objectively
• The endpoint solution, network-based solution, and attack simulation framework are used to derive a profile of the user and evaluate their success rate in mitigating attacks
• The study concludes that self-reported behavior of subjects may differ significantly from their actual behavior, and security awareness scores derived from objective measures are highly correlated with the user's success in mitigating social engineering attacks

The study conducted an experiment involving 160 participants that installed the framework for a duration of seven weeks. They simulated four different attacks on the users: phishing, spam, permission abuse, and certificate manipulation. The results show that users classified as with high awareness using the endpoint and network-based solution were more likely to mitigate the attack in comparison to users classified as with low awareness.

Social engineering (SE) attacks have dramatically changed in recent years: They are no longer limited to PCs and they goes way beyond phishing. Despite those changes, current methods for evaluating the resilience of users to SE attacks are still mainly focus on phishing attacks and do not distinguish between different platforms.Furthermore, current methods depend, to a large extent, on the subjects’ response to surveys, which tend to be subjective, biased and require the subjects’ active involvement and collaboration; thereby are less accurate and consuming significant human resources. Other solutions are based on measuring the momentary behaviour of subjects while facing a simulated phishing attack. These methods however, tend to be sensitive to environmental factors and cannot be used for evaluating users’ behaviour continuously.We present a methodology and an automated, scalable and objective framework for continuously evaluating the resilience of users to specific types of social engineering attacks. The methodology includes a set of measurable criteria for a security aware user; and an expert-based procedure for deriving security awareness models for different attacks classes (each class is an aggregation of SE attacks that exploit a similar set of human vulnerabilities). The framework utilizes data collected and analyzed from different data sources to measure the set of criteria:Android agent, which measures the users' actual behaviourwhile operating with their smartphones.Chrome extension, which measures the users' actual behaviourwhile operating with their PCsNetwork traffic monitor, which analyzes the network traffic transmitted-to/received-from the devices.Attack simulator, which implement multiple type of SE attacks on the users. In order to evaluate the proposed framework, we conducted an empirical experiment involving 162 users for a duration of seven to eight weeks. The results show that (1) the skills required from a user to mitigate an attack are different for different attack classes; (2) the self-reported behaviour of users differs significantly from their actual behaviour and (3) the security awareness level derived from the actual behaviour of users is highly correlated with their ability to mitigate SE attacks.