DNV GL shares what AI systems need in order to be deemed trustworthy
Risk management and quality assurance company DNV GL released a position paper giving perspectives on what needs to be considered in developing verification processes and the assurance of artificial intelligence (AI) systems in industrial contexts. Authors Dr. Asuncion T. Lera St. Clair and Dr. Øyvind Smogeli shared their perspectives.
According to the company, the trustworthiness of AI systems is not very different from that of a leader or an expert to whom, or an organization to which, we delegate our authority to make decisions or provide recommendations to reach a particular goal. AI systems should be subjected to the same quality assurance methods and principles used for any other technology.
The rigor with which we evaluate an expert’s recommendation depends on the importance of her recommendation and its context, reports the company. This means that the rigor and efforts required to build trust in the deployment of a specific AI system will depend on the severity and probability of potential consequences.
The company goes on to say that the deployment of AI systems in society introduces complexity and creates digital risks. While complexity in traditional mechanical systems is naturally limited by physical constraints and the laws of nature, complexity in integrated, software-driven systems—which do not necessarily follow well-established engineering principles—seems to easily exceed human comprehension. This increased complexity, driven by digitalization, is deepened by the integration of AI technologies introducing new risks and opening substantive trust gaps.
As a contribution to the global debate on trust in AI, the company has put forward characterizations of trustworthy industrial AI systems, with a focus on the integration of AI into existing cyber-physical systems and other digital assets. Further, it discusses how AI-enabled digital assets require assurance of development and deployment processes, as well as product assurance of the digital asset itself.
As recommended by the European Commission, AI is defined as “systems that display intelligent behavior by analyzing their environment and taking actions—with some degree of autonomy—to achieve specific goals.”
The company defines trustworthy AI systems as those that display the following characteristics:
- Ability to perform and capacity to verify delegated tasks
- Appropriate human-machine interdependency
- Clearly defined purpose
- Transparent impact on relevant stakeholders
Legitimacy: First and foremost, the company says that an AI system should be legitimate. Its legitimacy depends on issues related to algorithm and model training, data governance, the suitability of the chosen AI algorithm for the problem to be solved, and the context of this problem. It is essential to establish that the AI system's residual risk is acceptable to all stakeholders, regardless of the system's benefits, such as cost-efficiency. Ultimately, the legitimacy of deploying AI methods and tools will depend both on the system being fit-for-purpose and on risk management being placed at the core.
Ability to perform and capacity to verify delegated tasks: Similar to leaders, experts or organizations, and following established quality assurance and performance principles, it is necessary to establish that AI systems are competent and have the ability to do the work delegated to them. This entails ensuring that their design, deployment, and operational performance are of sufficient quality and robustness. Even though many AI algorithms are of a black box nature, transparency can be improved through “explainability.” Last, the combination of all these criteria needs to generate appropriate evidence for the eventual verification of trustworthiness.
Appropriate human-machine interdependency: Human-to-machine and machine-to-machine interactions and interdependencies deserve close scrutiny. A rapidly increasing number of functions in many cyber-physical systems (from cars, ships, and airplanes to infrastructure such as energy systems and pipelines) are already being elevated to higher levels of autonomy. It is critical to map and understand the agents and roles involved in the development, deployment, use, and maintenance of an AI system, as well as the external stakeholders affected by the AI system in operation. Transparent and understandable communication between all these types of agents and stakeholders, including machine-to-machine interaction, is key for ensuring the trustworthiness of AI.
Clearly defined purpose: The motive and purpose of deploying an industrial AI system need to be disclosed in order to ensure trustworthiness. This disclosure includes revealing the potential benefits and risks for all stakeholders. The motive and purpose of the AI system also need to be assessed in relation to corporate accountability processes.
Transparent impact on relevant stakeholders: The trustworthiness of AI systems must also be judged by looking at the impact they may have. A large share of the ethical considerations relates to the possible impact of AI systems on people’s rights to privacy, non-discrimination, unbiased decision making, etc. In industrial safety-critical application contexts, the deployment of AI systems could be subject to the same impact assessment methods that are common for other technologies. Establishing the impact of a particular AI system presupposes ascribing responsibility to different agents and distinguishing between intentional and unintentional actions. Last, the impact of an AI system would have to be monitored continuously and throughout the system's lifecycle.
DNV GL proposes these characteristics of trustworthy industrial AI systems as best practices emerging from ongoing work on the assurance of digital assets. The company is taking steps to provide assurance of digital assets, including those that incorporate AI systems.
For more information, visit www.dnvgl.com.