Quantifying the Effects of Operational Technology or Industrial Control System based Cybersecurity Controls via CVSS Scoring

— This paper has examined the application of the Common Vulnerability Scoring System applied to operational technology or industrial control system-based cybersecurity controls and demonstrated that the unique considerations and aspects of these environments are more accurately captured when compared against a traditional IT based evaluation. Multiple business drivers are compelling consumer goods manufacturers to augment and connect their manufacturing systems bringing with it increases in potential for experiencing a cybersecurity incident [1]. While other business verticals are able to utilize cybersecurity standards and control documents tailored for their industry, manufacturers do not have a set of materials that directly correlate to the operational technology environments in which their systems reside [2]. Cybersecurity practitioners face additional challenges in developing an understanding of the severity of the risks within these environments due to the lack of current quantifiable methods of evaluating the risks. The findings from this research provide cybersecurity practitioners with a repeatable and extensible method to derive the operational risk present to an organization due to the technologies and business strategies employed in the pursuit of business objectives.


I. INTRODUCTION
Quantifying compliance against a set of controls or standards provides a means of organizations comparing their own adherence to other organizations or can be used when comparing two locations to each other. The NIST Cybersecurity Framework provides a hierarchical system for compliance with whole number scoring values of 1 through 4 based on level of adherence [3]. When evaluating the specific risk which are reviewed through the controls, the common vulnerability scoring systems (CVSS) provides a method of understanding of the severity of the risk as it applies to the organization based on changes in values of specific metrics such as Access Complexity or Target Distribution [4].
To gain or maintain competitive advantage over other businesses within their marketspace, consumer goods manufactures must continually identify methods of producing products at a lower cost, more quickly, and in alignment with the changing expectations of consumers [5]. Many of these initiatives align with recent paradigms within the space such as Industry 4.0, and Smart Manufacturing, both of which leverage technology for advanced data analytics and generating insights into business operations. These technologies introduce changes to the operational technology environments and both the technologies involved and the methods in which they were deployed generate additional risks for the business [6]. Unless identified and incorporated into a business' risk plans, these risks are left unaddressed and create a situation in which vulnerable systems can be exploited and generate a multitude of losses for the organization.

A. Common Vulnerability Scoring System
The use of CVSS has been an approach for scoring and quantifying risk since it was first introduced in 2005 [7]. Through its multiple iterations based on public feedback, CVSS has been updated to include additional considerations for risk, more accurate metrics for scoring risk, and industry specific variation. Applying the CVSS method of generating a score to identify areas of higher risk has been applied to other security models and the creation of a vulnerability management center was explored in the research performed by Walkowski et al. [8]. Applying CVSS to non-traditional areas of technology such as operational technology would follow the same methodology in which it has been applied to the fastgrowing areas of IIoT in which alternative communication protocols and software packages introduce new risks into the environment [9].
The use of calculated metrics for risk within operational technology has been in use within the critical infrastructure sector and examples such as the research conducted by Venkataramanan et al. [10] demonstrate how the variables and factors related to OT equipment can be utilized in quantifying the risks. Research into the area of autonomous construction equipment, which has large overlap with the technology and risk factors which are present within manufacturing environments has been explored through the research by Sonkor et al. [11]. Though there has been CVSS research applied to facets of operational technology, there exists gaps in applying this approach specifically to consumer goods manufacturing.

B. Quantitative Modeling of Cybersecurity Risks
Deconstructing the Common Vulnerability Scoring System for application to specific assets and vulnerabilities associated with them resulted in a threat model that better captured the current risk profile of the assets and enabled asset owners to prioritize remediation actions [12]. This research did not specifically focus on the application of CVSS to a set of cybersecurity controls but demonstrated the validity of utilizing the concept of CVSS as a means of quantifying the severity of risk associated with a specific object. The quantification of other facets related to cybersecurity risk severity was demonstrated in the research by Hughes and Cybenko [13] in which system susceptibility, threat accessibility, and threat capability are explored. The researchers explored aspects of cybersecurity risk in a similar method to the evaluation of business strategy and technical solutions as a function of cybersecurity risk severity which is part of the proposed methodology of this research.
The use of quantitative methods as a means of performing assessments has yielded positive results for multiple business verticals. Industry specific cybersecurity controls such as NIST 800-30 were used in conjunction with the OCTAVE Allegro framework to perform risk assessment and risk scenarios within the medical device field resulting in the identification of data and devices which required additional safeguards to meet acceptable risk targets [14].
This research focused on the application of cybersecurity a cybersecurity framework to a set of industry-specific cybersecurity controls, but the quantitative measurements only utilized the exploitability sub score of CVSS instead of attempting to utilize all relevant parameters as is the objective with this research. Within the financial information systems realm, research has been conducted in the space of breaking down the different layers at which technology or business strategies interact and calculating the severity of risk associated with each layer [15].
The resulting analysis provided insight on the type and potential for cybersecurity risks to be present within systems but did not attempt to represent that as a single number or grading. Similar analysis of cybersecurity risks within business information systems demonstrated methods of quantification based upon monetary costs associated with remediation or mitigation [16].

A. Problem Statement
The industrial and automated control systems utilized by the manufacturers of consumer goods do not have a standardized, comprehensive, and quantifiable cybersecurity framework to adequately assess their inherit risks due to current documents either presenting a qualitative approach to evaluating compliance or providing no guidance on how to evaluate and determine compliance against the standards or controls [17]. There are no requirements to adhere to any cybersecurity standard for producers of consumer goods such as children's toys, air fresheners, or power tools. The acceleration of adoption of new technologies such as cloud-based data analytics, Industrial Internet of Things (IIoT), and lights-out manufacturing are not reflected in the standards and controls currently in existence [18].

B. Hypothesis Statement
H0: When evaluated against the set of controls, the new approach is not necessary to assess the inherent risk severity more comprehensively within an OT environment and the resulting CVSS framework assessment would not present a differing result to the assessors. H1: When evaluated against the set of controls, the new approach is necessary to assess the inherent risk severity more comprehensively within an OT environment and the resulting CVSS framework assessment would present a differing result to the assessors.

C. Research Question
Would additional considerations applied to cybersecurity controls and standards, utilized by consumer goods manufacturing companies, result in any difference in the risk severity assessment scoring when measured with the common vulnerability scoring system?

A. Method
The research method selected for this study is quantitative design science as the objective is to identify a change in risk assessing as a means of providing a possible solution for security practitioners to perform their work more holistically [19]. The quantitative approach of analyzing cybersecurity materials, specifically risks and risk severity assessment methodologies, has been successfully leveraged in many research projects including those of Aksu et al. [12] and Algarni et al. [16]. As each identified gap is independent of any other gaps within the materials under examination, there is no basis for conducting a weighting or comparison between each gap nor is it possible to calculate the significance of the gap as each organization may have their own unique circumstances related to the calculation of cybersecurity risk severity [6]. The CVSS vector string as a means of quantifying the severity of cybersecurity risk has been employed in similar cybersecurity research and is an appropriate fit for the proposed methodology.
Data collection was conducted for each control. The data being collected is publicly available and no human interaction such as interviews or surveys were leveraged in the collection activities. The data collection was conducted by: 1) Identifying the relevant operational technology focused cybersecurity standards and controls documents which are targeted towards consumer goods manufacturers. 2) Reviewing the industry specific business strategies and technical solutions which may prompt additional consideration for the impacts they may have on controls when compared to non-manufacturing operational technology environments.
The CVSS formulas serve to demonstrate the severity of the newly identified risks and provide a means of demonstrating the significance of the controls in the process of assessing all present risks within the environment. The controls were evaluated to adequately address the risks identified during the data analysis phase of the research [20]. The artifact construction was conducted in the following sequence: 1) For each control, construct an appropriate CVSS formula which calculates the severity of the risk identified within the control; 2) For each control parameter, evaluate if the technical solutions or business strategies employed by the organization are relevant and aligned to the CVSS variables; 3) For relevant parameters within the control, determine how the technical solution or business strategy impacts the value assigned to each variable; 4) Calculate the new CVSS value for each control based upon the changes in variable values; 5) Compare the original and new values to determine if the severity of the cybersecurity threat has increased, decreased, or remained the same.

B. Population and Sample
In total there are 262 items across all three documents (NIST Cybersecurity Framework, NIST 800-82, and ISA / IEC 62443) which comprise the first population set. Many of the unique items from each document can be cross-referenced or aligned with items from the other documents resulting in a reduced number of unique cybersecurity controls. The sampling involved generating five representative use cases from the five categories of the NIST Cybersecurity Framework, Identify, Protect, Detect, Respond, and Recover [3], which are also represented within NIST 800-82 and ISA / IEC 62443.
The second population set consists of the set of CVSS values generated through application of the proposed methodology against the original and re-evaluated version of controls found by usage of the CVSS formulas. There is a collection of CVSS Base, Temporal, and Environmental scores calculated for each control all of which can range in value between 0 to 10 with the value of 9 or greater signifying the most serious or critical matters to address immediately. A total of five CVSS value sets were generated, and each set serves as a means of comparing the risk evaluations between the two methods of evaluating of the control.

C. Definitions and Formula
Definition 1: All variables which exist within the CVSS 3.1 formula are utilized in the construction of Set C. This set is referred to as "The Set of CVSS score variables" and is shown in (1). Each element of the set is associated with the variables used in the generation of one of the three CVSS scores.
Definition 2: A control related to cybersecurity, such as those within the focus of this research, are denoted as a given Set P which contains CVSS score variables ! , " , … , # to be assigned values V1, V2, …, Vn, where $ ⊆ , and n is an integer. Equation (2) shows the representation of a control represented in set form: Definition 3: Each technical solution or business strategy which consumer goods manufacturing companies are utilizing to gain or maintain competitive advantage that alters the severity of cybersecurity risk carried by the organization are represented as a set S which is a subset of set C with the specific CVSS score variables which are relevant to it and is shown in (3). A cybersecurity control may be impacted by one or more technical solutions or business strategies.
Definition 4: When applying a technical solution or business strategy set SX to a cybersecurity control PCSC, it is important to understand of the solution or strategy is increasing or decreasing the amount or severity of the threats present within the organization. In instances where SX increases threats, the highest severity rating value for each variable is assigned for the resultant set P'CSC as shown in (4). In instances where SX decreases threats, the lowest severity rating value for each variable is assigned. The same rule applies if applying multiple SX sets to a control PCSC as shown in (5).

A. Results
The control ID.AM-3 within the NIST Cybersecurity framework [3] states, "Organizational communication and data flows are mapped" (p. 24). Declaring PID.AM-3 as NIST Cybersecurity Framework control ID.AM-3, the relevant CVSS score variables AV, AC, PR, UI, C, I, A, CR, IR, and AR would be included within the set. The initial values for each variable are shown in (6) as the standard representation of a CVSS 3.1 format.
Many non-Ethernet communication networks such as Data Highway, ControlNet, or Profibus exist within manufacturing facilities and may be initially overlooked by assessment teams who are only familiar with traditional IT devices and systems [21]. The technical solution of utilizing non-Ethernet networks for the communication and control of critical manufacturing operations may be represented as S1. Due to the criticality of the communications and their significance on the operation of the equipment, this is factor which increases threats and the values assigned to the variables are the highest value and as such S1 is defined as: ! = , ()) , ()) , ()) , ()) / If S1 is applied to PID.AM-3 the resulting P'ID.AM-3 will contain elevated values for several variables which impacts the CVSS value calculated for the control. Therefore, based upon (4) the (9) is constructed:  To reduce the costs and resources associated with warehousing both raw materials and finished goods, many manufacturers are utilizing the concepts of Just in Time Manufacturing which requires the careful scheduling of goods and transportation [1]. An organization will define the amount of time a disruption of the supply chain can be tolerated and will plan to have adequate materials and warehouse space to support that business resiliency objective. With a cybersecurity incident which impacts the organization's own logistics capabilities, both upstream and downstream partners will be impacted by the inability to receive or send the expected goods or materials. A cybersecurity incident which impacts the production capabilities of the organization may result in challenges associated with housing received materials and meeting obligations for delivering finished goods. Third party partners who experience these same types of cybersecurity incidents will similarly impact the organization's ability to operate. Set S4 defines the business strategy of just in time manufacturing and is shown in (10).
For non-manufacturing operational technology environments, the dependence on third parties as part of the core operation of the facility is lower [24]. While ancillary systems such as billing may ultimately impact the ability for a process to operate such as with Colonial Pipeline [25], their dependence on third party material logistics is not as strong.
For operational technology controls [3] which evaluate third party agreements or responsibilities such as DE.AE-5 which states "Incident alert thresholds are established" (p. 38) and RS.CO-4 which states "Coordination with stakeholders occurs consistent with response plans" (p. 41), two sets can be created which represent the severity of the threat in non-manufacturing environments. The equations shown in (11) and (12) correlate to control DE.AE-5 with (13) and (14) correlating to RS.CO-4. T Applying S4 to both PDE.AE-5 and PRS.CO-4, shown in (15) and (16)

B. Summary
The main findings from the research indicate that it is feasible to quantitatively evaluate cybersecurity controls via the generation of CVSS 3.1 vector strings against the business strategies or technical solutions utilized in organizations. Furthermore, this methodology for quantitative analysis demonstrated that cybersecurity controls written for specific environments may not provide comprehensive criteria for evaluating compliance against the control resulting in unidentified or unaddressed organizational risks. Though utilizing many of the technical components of other operational technology industry verticals, consumer goods manufacturing companies possess unique factors that require additional considerations by cybersecurity professionals when evaluating an organization's compliance with a set of cybersecurity controls.
The changes in severity rating demonstrated within the controls identify multiple areas such as third-party dependence and network integrity requirements for which organizations may have been underserving the threats present in these areas resulting in additional risks being carried. The method of evaluating existing controls against the unique considerations of a business vertical with the objective of more comprehensively evaluating the present threats can be quantifiably measured using CVSS values as a comparison tool. As it relates to the research hypothesis, the resulting CVSS values are measurably different due to the methodology explored in the research resulting in the null hypothesis being rejected.

VI. CONCLUSION
The quantitative results presented by the methodology demonstrate that cybersecurity practitioners evaluating conformance to a cybersecurity standard without an understanding of the environment they are acting within can result in and overconfidence in the amount of risk which has been successfully mitigated through existing measures. Similarly, organizations may be expending too many or too few resources in addressing cybersecurity risks based on assessments which have not been appropriately conducted due to a lack of organizational factors related to business strategies and technical solutions. To have a more accurate understanding of these threats and their relation to business risks, organization should refine their understanding of how the CVSS parameters align with organizational activities and take a more data-driven approach to risk management based on the values assigned to each of the parameters for a specific organizational initiative.
When examining the differences in CVSS values which arose when comparing the original control against one which had additional considerations applied to it, other business verticals which utilize operational technology may also be addressing organizational risks with decisions based on insufficient or inadequate data. Other business verticals leveraging operational technology such as supply chain logistics companies utilizing warehousing and autonomous systems [24], mining which is increasingly leveraging autonomous machinery [26], and the IIoT-focused initiatives such as Smart Cities [27] are all incorporating technologies which can have drastic effects on the cybersecurity posture of the organization. Without having specific cybersecurity standards written for each business vertical, those adopting materials written for similar but different segments should incorporate a strategy such as the methodology explored within this research to accurately quantify their risks and develop data-driven strategies and appropriately address them.
The findings from this research should provide a cautionary example of not assuming that all cybersecurity threats are adequately evaluated if a seemingly appropriate cybersecurity standards documented is utilized when performing an audit or assessment. Though proving conformance or adherence to a cybersecurity body of work may be necessary as part of regulatory or business requirements, it should not be viewed by management teams as a guarantee that the organization has acknowledged and adequately address all risks present within the organization. Those performing assessments within an industry vertical should become educated on the business and technical drivers present within the industry and understand how they may impact the cybersecurity posture of the organizations they are working with. As demonstrated within the research, a lack of understanding of how an organization operates results in an incomplete understanding of the environment resulting in assessments which do not reflect the real risks currently held by the organization.
Findings from the research demonstrated that it was possible to perform quantitative analysis of cybersecurity controls through the perspectives of the control as originally drafted and again with additional organizational considerations applied to its evaluation. The use of the Common Vulnerability Scoring System version 3.1 as the basis for the quantitative evaluation provided a predefined set of parameters and variables against which the controls and business considerations could be mapped. By applying the methodology described in Section III against the five use cases explored within Section IV, the results were that business considerations could cause the CVSS values to increase, decrease, or remain the same. This methodology is effective in both evaluating a single control against a single business consideration or when evaluating multiple controls or business considerations in conjunction with one another.