Cybersecurity metrics matter

Eric Goldstein's call for using data highlights industry challenges. It raises crucial questions about legitimacy and the need for cyber security.

Eric Goldstein’s quote on the need to use data indicates that the cybersecurity industry is now at a crossroads. The community is finally starting to generate useful data, but the question that remains is: what do they do with it?

Holy Cow, we actually have really good data now, let’s use it for something.

Eric Goldstein, the Cyber Security and Infrastructure Security Agency (CISA) Executive Assistant Director, at a Center for Strategic and International Studies (CSIS) event on October 23, 2023.  

As explored in a recent article in the Journal of Cyber Policy, cyber security constitutes a realm of research. It is a matter of national security that frequently operates with limited evidence or data. The community possesses the scant capability to comprehend the scope at which cyber security functions. It is primarily due to the industry’s limited capacity to generate, retain, and analyse the extensive volumes of data necessary for comprehending cyber security as a domain.

The challenge of cyber security data and metrics

What is signified by being a discipline that operates devoid of data? Moreover, can cyber security genuinely be considered a legitimate discipline when macro evidence is absent from its operation? Under these circumstances, how is it possible to establish a field, and what ramifications does this entail?

These are the questions that keep someone like me awake at night. They should also concern you. Unlike in the field of nuclear weapons, the capacity to grasp the extent of activities conducted by states and criminal actors remains limited in the realm of cyber security.

Counting missiles and malware

In nuclear security, there is typically a general understanding of a state’s nuclear weapon inventory. More crucially, information is frequently available regarding materials, testing, and launch facilities. In the context of cyber security, however, there is no analogous situation.

Another example will help elucidate this point. We can ascertain the daily count of cruise missiles launched by Russia against Ukraine. Additionally, we possess data on the total number of tanks lost during the invasion, and certain groups, such as Oryx furnish visual confirmation for each data point. However, in the realm of cyberspace, how many instances of Russia’s attacks on Ukraine have been recorded remains unclear.

Credit. Midjourneay

Academics engaged in cyber research require more detailed data. Merely stating that there have been thousands or millions of attacks lacks meaning, as such figures lack context and scale. Frequently, when the industry discusses cyber-attacks, it refers to cyber operations—deliberate campaigns with well-defined objectives. Nevertheless, the data presented to substantiate assertions of an escalating cyberwar often comprises individual instances that fail to convey the overall scale or impact.

Transitioning from speculation to the domain of statistics and metrics in the field of cyber security is deemed imperative. A report authored by my research team from Seton Hall University and CSIS, reveals that Russia undertook 47 distinct cyber operations during the initial six months of the conflict.

Interestingly, civilians, rather than the military, constituted the primary targets of the majority of these operations. The significance lies in the fact that anyone can make assertions of an extensive cyber offensive by Russia against Ukraine. However, in the absence of corroborating evidence, such claims of a dramatic cyber war remain unsubstantiated.

Barriers and roadblocks

The primary obstacle to data and analysis in the realm of cyber security is the presumption of secrecy. There exists a firmly entrenched notion that quantifying cyber security attack data is an insurmountable task due to the covert nature of these operations, rendering them uncountable. Unlike a bullet or an artillery shell, a piece of malware is not easily quantifiable.

This perspective is confronted by the characteristics of cyber operations and, secondly, by the conduct of covert operations. Contrary to claims made, a cyber-attack constitutes a public event, frequently unfolding openly. Secrecy within the domain of cyber security often misinterprets the essence of covert operations. Covert operations may remain undisclosed, yet this does not imply their resistance to measurement.

Some posit the notion that cyber operations are a component of the intelligence process. However, this assertion falls short because intelligence inherently involves the analysis of information. It remains uncertain whether major organizations, including CISA, possess the analytical capacity to scrutinize the extensive volumes of data they generate or gather. In the absence of analysis, metrics, and quantification, everything within the domain of cyber security descends into speculation devoid of concrete measurement.

Most scholars seem to believe that measurement and metrics in cyber security are impossible or overly difficult, the reality is that establishing metrics is not only possible but essential to the management of the threat that cyber warfare poses to society.

Brandon Valeriano

Fixing the situation

Regrettably, in contrast to the history of strategic studies, which frequently concentrates on measuring military campaigns through the framework of operational research, this approach is largely absent in most investigations of emerging technologies. The field must contemplate ways to augment analysis while simultaneously offering top-down guidance on the influence of data analytics and metrics on performance.

Enhancing data quality necessitates collaboration and a sense of humility. The creation of improved datasets for communities of interest is an ongoing process, demanding both time and exertion. Subsequently, the acquired data must undergo analysis and processing.

Data investigations in cyber security could be facilitated by the establishment of a new federal organization, although the challenge lies in the fact that most individuals are inclined to diminish the federal government’s capacity to engage in cyberspace activities, citing concerns related to freedom of speech, liability, and regulation.

The truth is that data solutions necessitate collaboration between the government and the private sector. The community must foster collaboration between the public and private sectors to assemble a comprehensive understanding of the data landscape in cyber security.

Although legislation exists to facilitate the collection of cyber security incidents, there is no established formal procedure for collecting and analysing this data. Frequently, the objective of collection lacks specific guidance and lacks a structured analysis process. Until these issues are rectified, the community operates without clear direction while confronting one of the most formidable security threats of the modern era.


Journal Reference

Valeriano, B. (2022). The need for cybersecurity data and metrics: empirically assessing cyberthreat. Journal of Cyber Policy7(2), 140-154. https://doi.org/10.1080/13501763.2023.2172060

Brandon Valeriano (PhD) recently moved to Seton Hall University’s School of Diplomacy after serving as the Bren Chair of Military Innovation at the Marine Corps University. He is a Visiting Fellow at the Royal Danish Defense College and also serves as a Distinguished Senior Fellow at the Marine Corps University, Senior Advisor to Gray Space Strategies, and Senior Advisor to the Cyberspace Solarium Commission 2.0. He has published six books and over one hundred articles on cybersecurity and international security.