Technology control testing is a comprehensive evaluation of an enterprise’s technological infrastructure, systems and processes. It is a critical aspect of corporate governance designed to ensure that technology resources are being utilized effectively, efficiently and in line with established policies and regulatory requirements. Control testing demonstrates that controls are mitigating risk as intended, and it encompasses a wide range of activities, including validating system functionality, assessing network security, verifying data integrity and evaluating disaster recovery procedures. It can also involve testing specific controls designed to prevent or detect unauthorized access, data breaches and system failures. These tests may be carried out internally by the IT department or externally by independent auditors or consultants.
Enterprises perform technology control testing for several reasons. First, it helps identify potential vulnerabilities in their IT systems that could be exploited by malicious actors, thereby mitigating the risk of cyberattacks and data breaches. This is particularly important given the increasing prevalence and sophistication of cyberthreats.
Second, technology control testing ensures that an enterprise’s IT systems are operating as intended and supporting business processes effectively. This is crucial for maintaining operational efficiency and productivity. For instance, a flaw in the inventory management system could lead to stockouts or excess inventory, both of which can have significant financial implications.
Third, this testing verifies compliance with applicable laws, regulations and industry standards related to information security and data protection. Noncompliance can result in substantial fines, legal sanctions and reputational damage. As such, technology control testing is a key element of enterprise compliance programs. Many enterprises adopt industry standard risk frameworks, such as those developed by the US National Institute of Standards and Technology (NIST), to design and test controls.1
Technology control testing is a vital business practice in the contemporary corporate landscape. It provides a mechanism for safeguarding valuable technology assets, ensuring the continuity and efficiency of business operations and maintaining compliance with regulatory requirements. Thus, it plays a fundamental role in risk management and corporate governance and should be adopted by all organizations focused on maturing their risk and control processes.
The Factory Model
Technology control testing is conducted by many enterprises in disparate industries, including financial services, healthcare and retail. Regardless of the industry, control testing typically involves four basic steps: gather evidence, analyze the evidence, assess effectiveness and substantiate the results. The first three steps can vary in time and complexity; well-designed and automated controls can be tested quickly, while more complex and manual controls may require more time and resources to thoroughly evaluate. The risk framework also dictates how often controls are tested—usually annually, but sometimes quarterly. Typically, enterprises use a binary effective or ineffective rating, although some may have a three-or five-scale option.2
Efficient factory operations are a complex, multifaceted function that involves strategic planning, rigorous process control and continuous improvement.
Testing is normally accomplished by individuals trained in risk principles, ideally using an automated risk management system to govern and manage the process. A key concern is how to create and maintain an efficient control testing team, particularly in light of ongoing financial challenges and the need to manage resources effectively while delivering quality service. To accomplish this, techniques perfected by the manufacturing industry can be adopted. If controls are viewed as factory widgets that must be processed in a timely, efficient manner, the factory model offers significant advantages.
Efficient factory operations are a cornerstone of successful manufacturing enterprises, and these operations are characterized by myriad factors, including:
- Streamlined processes—Efficient factories have well-defined, streamlined processes that minimize waste, reduce redundancy and enhance productivity. Standard operating procedures (SOPs) are meticulously developed and implemented, ensuring consistency and accuracy in production activities. Well-established governance for control testing processes minimizes administrative overhead and ensures greater organizational efficiency.
- Optimal use of resources—Efficient factory operations are characterized by optimal resource utilization. This includes judicious use of raw materials, energy and human resources to maximize output while minimizing costs. Highly efficient factories often employ lean manufacturing strategies to eliminate waste and improve productivity. For control testing, optimizing resources ensures sufficient capacity for unexpected control testing needs or priority requests, such as reacting to regulator supervisory actions.
- Technological integration—The advent of Industry 4.0 has resulted in the greater integration of advanced technologies into factory operations.3 From automation and robotics to data analytics and artificial intelligence, technology plays a crucial role in enhancing efficiency, reducing error rates and increasing production speed. Automated technology solutions can be used to develop and manage the control testing process. Examples include Archer GRC or ServiceNow Information Risk Management applications.
- Effective workforce management—A well-trained, motivated workforce is key to efficient operations. Effective workforce management involves providing necessary training, fostering a positive work environment, ensuring worker safety and implementing performance incentive schemes. In addition, proper training (i.e., supporting professional certifications such as Certified Information Systems Security Professional [CISSP], Certified in Risk and Information Systems Control [CRISC] and Certified Information Systems Auditor [CISA]) establishes a strong foundation of expertise. When integrated with state-of-the-art technology solutions and proper training, organizations become more desirable places to work for information security professionals.
- Adherence to quality standards—High efficiency is often synonymous with high-quality outputs. Efficient factories strictly adhere to quality standards and regulations. Quality control and assurance processes are integrated into every stage of production to minimize defects and ensure product reliability. Control testing results, especially in highly regulated industries such as banking or insurance, are often reviewed by external regulatory agencies. This scrutiny requires a robust quality assurance process to ensure the results and associated evidence withstand such oversight.
- Robust supply chain management—Efficient factory operations are closely linked with robust supply chain management. An effective supply chain ensures the timely availability of raw materials, minimizes inventory holding costs and ensures the quick distribution of finished goods. Adopting a supply chain view for control design and testing is logical: the control is a product produced by the factory and has associated key performance indicators such as time to market (i.e., design to implemented control), test cycle time (throughput from start to finish of testing) and burndown processing (throughput of controls through the entire factory process).
A comprehensive governance process is the bedrock of any control testing effort.
Efficient factory operations are a complex, multifaceted function that involves strategic planning, rigorous process control and continuous improvement. They are pivotal in improving the bottom line, ensuring stakeholder satisfaction and maintaining a competitive advantage in the marketplace.
Key Components of the Factory Model Approach
Leveraging a factory model approach for technology control testing means adopting three key principles and associated tasks: establishing a governance process, creating appropriate operational procedures and establishing automated reporting.
Establish a Governance Process
A comprehensive governance process is the bedrock of any control testing effort. This process typically involves defining clear roles and responsibilities for managers, testers, reviewers and certifiers. The delineation of roles and responsibilities is crucial to guarantee that all individuals involved in the control testing process understand their duties and obligations. In the factory model approach, roles can be divided into three categories—governance, execution and review—each of which plays an essential part in the overall process.
Another critical aspect of the governance process is identifying the supply chain for controls. This starts with defining how, when and why new controls are designed and how the existing inventory is changed and managed.
To establish the governance process, organizations should implement a control testing program charter. A program charter for a technology control testing organization is a formal document that outlines the objectives, scope, stakeholders and responsibilities of the organization. The charter serves as a blueprint for the organization’s operations and provides direction for all staff members.
The first element of a program charter is the definition of the objectives, which provide the overall direction for the organization. These objectives should be specific, measurable, attainable, relevant and time-bound (SMART). For a technology control testing organization, the objectives might include ensuring the reliability and security of technology systems, improving the efficiency of control processes and reducing the risk of technology failures.
The scope of the program is another essential element of the charter that should be clearly defined. It outlines the most critical areas of focus for the organization, such as specific technology systems, control processes and risk management activities.
The charter should also identify the key stakeholders of the testing organization. These might include internal stakeholders such as IT staff, management and employees and external stakeholders such as regulators. The charter should outline the stakeholders’ roles and responsibilities and how they will be involved in the control testing organization’s activities.
Finally, the charter should define the responsibilities of the control testing organization. This includes the tasks that the organization will undertake, the standards it will adhere to and the outcomes it is expected to achieve. The responsibilities should be aligned with the organization’s objectives and scope to ensure that it can effectively fulfill its mission.
A program charter for a technology control testing organization should provide a clear and comprehensive roadmap for the organization’s operations. By defining the objectives, scope, stakeholders and responsibilities, the charter can help to guide the organization toward its goals and ensure that it delivers value to its stakeholders.
Create Appropriate Operational Procedures
Suitable operational procedures that guide the control testing process must be developed. These procedures dictate the sequence of tasks, the specific methods of executing each task and the criteria for evaluating the outcomes. They should be flexible enough to accommodate changes in technology and regulatory requirements, yet rigid enough to ensure consistency and reliability in control testing.
Operational procedures should align with the organization’s overall risk management framework and may consist of:
- Control design procedures—This includes the rationale for creating new controls, ownership of the process and identification of key stakeholders (i.e., first-line risk partners, second-line risk partners and control owners responsible for executing the process associated with the controls). In the supply chain management view, the control design procedures govern the processes for delivering a solid control to the loading dock of the control testing factory.
- Control testing methodology—This defines the key elements of control testing, including frequency of testing (i.e., annual or semiannual, typically based on whether the control is key or not), type of design (i.e., automated, preventative, manual or detective) and pass/fail criteria. The methodology should also identify the required evidence to support evaluation of the pass/fail criteria.
Establish a System of Record and Automated Reporting
A mature control testing organization should implement an industry-standard technology solution to manage the control testing process. These systems help organizations manage their processes and become the system of record for control testing results.
Beyond being the system of record, implementation of live operational dashboards provides the ability to actively measure and manage the factory throughput. Ideally, the testing system used to manage the control inventory and execute testing will be designed to create detailed, customized dashboards accessible by stakeholders, managers and the testers themselves.
Automated dashboard reporting provides real-time updates on the status of testing operations, which is pivotal in the dynamic environment of control testing. Quick access to data enables business leaders to make informed decisions promptly, thus reducing the time lag involved in operational changes. This real-time data availability is paramount for detecting anomalies in the processes and enabling immediate remedial action to prevent potential losses.
Automated dashboard reporting also reduces the risk of human error, thereby improving the accuracy of the reports generated. Traditional manual reporting methods are susceptible to errors due to the complex nature of the data involved. However, automation eliminates these potential inaccuracies, ensuring the reliability of the information on which decisions are based.
Moreover, automated dashboards facilitate the consolidation and visualization of data, thereby aiding the comprehension and interpretation of complex data sets. The visual representation of data makes it easier to understand and analyze trends, identify issues and benchmark performance against predefined targets. This feature is particularly beneficial in control testing operations that involve large volumes of data.
In addition, automated dashboard reporting promotes transparency and accountability. Easy access to data ensures that all stakeholders are aware of the enterprise’s progress, facilitating open communication and collaboration. This feature fosters a culture of responsibility and accountability, thereby improving the overall productivity and effectiveness of operations. A sample dashboard for control testing is shown in Figure 1.
The live dashboard data can be accessed by all stakeholders for individual reporting needs. The data can also be benchmarked against established key performance indicators that measure the throughput of the factory and help managers understand the efficiency (or inefficiency) of the end-to-end process. Supply versus demand, control tester availability, control test cycle time and control results by domain (i.e., access management, capacity management and vulnerability management) can easily be monitored and managed.
Conclusion
The factory model approach offers a structured and efficient method for technology control testing. By establishing a governance process, creating appropriate operational procedures and implementing automated reporting, enterprises can bolster the efficiency and effectiveness of their control testing efforts, ensuring that they are equipped to navigate the complex technological and regulatory landscape.
Endnotes
1 US National Institute of Standards and
Technology (NIST), NIST Special Publication
800-53 Security and Privacy Controls for Federal
Information Systems and Organizations, Revision 5,
USA, September 2020, http://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final
2 Powers, M.; “Technology Control Automation:
Improving Efficiency, Reducing Risk and
Strengthening Effectiveness,” ISACA Now,
19 January 2022, http://4crt.rf518.com/resources/news-and-trends/isaca-now-blog/2022/technology-control-automation-improving-efficiency-reducing-risk-and-strengthening-effectiveness
3 Uzialko, A.; “Industry 4.0: How Technology Is
Revolutionizing the Manufacturing Industry,”
Business News Daily, 21 February 2023, http://www.businessnewsdaily.com/10156-industry-manufacturing-iot.html
MICHAEL POWERS | PH.D., CRISC
Is an IT risk director at a midwestern US regional banking institution, where he is responsible for managing a Technology Risk Center of Excellence. He is also an adjunct professor of quantitative statistics, project management and cybersecurity at three universities. He can be reached at mpowersphd@gmail.com.