Lucas Garcia, Principal Product Manager for Deep Learning, MathWorks, talks to IoT Insider about how recent White House legislation mandating that AI developers share their safety test results and measurements means engineers need to adopt new methods to meet the new rules. Verification and validation (V&V) is what Garcia is advocating for such AI monitoring – but what is it? And how will it work within systems in IoT?
Can you explain the end-to-end V&V process for AI systems, particularly in the context of IoT?
First, it’s important to distinguish between V&V. Verification ascertains whether a component has been designed and developed in line with the specified requirements, whereas validation checks if the product meets the client’s needs and expectations.
V&V processes ensure not only that the system is constructed correctly but also fulfil its intended purpose and can be utilised in real-world scenarios. Traditional V&V processes guide engineers through software development and testing across various phases.
At the initial stage, an engineer would need to break down the requirements to understand how the components of the system connect and the overall anticipated use.
The subsequent stage encompasses progressions to implementation, where engineers dedicate time to coding the components. The final stage involves integrating and rigorously testing the components to validate that the system meets the initial requirements and performs as anticipated. In IoT systems, continuous testing is essential as it can detect early flaws in the system.
However, this workflow needs adapting to incorporate AI, due to the necessity of providing certainty about the emerging properties from AI models. Additionally, examining how the AI system is trained is a crucial element in understanding data treatment, ensuring the performance measured on the desktop generalises well when deployed to the Edge. This process involves iterative refinement, necessary to ensure all errors are identified and to ultimately have confidence in deploying effective IoT systems with AI components.
The ultimate goal is to systematically ensure any errors in the data-driven learning process are identified and rectified, providing confidence that the AI components within IoT systems will perform reliably in their operational environments.
What role does analysing decision-making processes play in the V&V of AI systems within IoT?
V&V analysis processes determine if there are any risk and failure points throughout the process, allowing engineers to implement safeguards to ensure the system behaves correctly.
Throughout the process, a major challenge for engineers is dealing with black box models, referring to systems where models take inputs and produce outputs without revealing the methods that generated the outcomes. Through the use of the Deep Learning Toolbox and its Verification Library, engineers can build, train, and simulate deep neural networks in IoT systems, analyse and identify how decisions were made for the model to arrive at a specific decision, and verify that the AI model performs reliably within its operating context.
This is a critical stage in the process, as it enables engineers to understand and predict behaviours in different scenarios, which is key in sensitive IoT applications where not understanding how the AI makes decisions can have severe consequences.
How can engineers effectively simulate AI-enabled IoT systems and what are the key elements to focus on in these simulations?
Simulation is crucial in developing IoT systems, allowing the creation of virtual environments and digital twins without real-world risk, enabling engineers to test different conditions that wouldn’t be possible in real-world deployment.
To effectively simulate these scenarios, engineers can use platforms like MATLAB and Simulink. Simulink is a block diagram to design systems with multidomain models, simulate before moving to hardware, and deploy without writing code. By simulating IoT-based decision making, engineers can analyse system performance without the additional expense of full-scale deployment. For example, customers in aerospace and automotive have physical systems that are time-intensive and expensive to build and test.
When simulating AI-enabled systems and bringing them to production, it is key to also have real-time monitoring and drift detection systems that constantly check the AI model is behaving within the accurate bounds of what is safe or expected. Once any inaccurate behaviour is detected, corrective measures can be taken, such as using a backup system, shutting down the AI, or creating a request for model re-training if the data has drifted. These monitoring systems can act as a safety net to provide an additional layer of protection and reliability.
Engineers use scenario-based testing, which involves putting the AI model through a range of situations it might encounter when deployed to see how it reacts, ensuring it can handle different scenarios safely and effectively. IoT engineers can also benefit from the use of code generation capabilities that are automatically tailored for IoT hardware deployments.
In summary, a monitoring system, scenario-based testing, and the use of code generation when transitioning to hardware are three key aspects of simulating AI-enabled IoT systems.
How does the V&V process help in identifying and mitigating risks associated with AI in IoT?
In the era of AI-enabled safety-critical systems, V&V procedures are becoming crucial for obtaining industry certifications and complying with legal requirements. Building and maintaining trustworthy systems requires engineers to employ verification techniques that provide explainability and transparency for the AI models running those systems, as well as building trust in model predictions.
When engineers use an AI model to add automation to a system, they must overcome the transparency challenges these systems present. This is especially relevant when the black-box nature of the model prevents the use of other approaches. Through explainability techniques, engineers can identify the regions of an image that contribute most to the final assessment, enabling them to understand the model’s primary focus when making predictions.
When using AI in IoT applications, particularly for those that are safety-critical, engineers are tasked with confirming that the AI performs reliably and safely under all operating conditions. By incorporating V&V into the development lifecycle of AI in IoT applications, engineers can systematically identify and mitigate risks, ensuring the system is not only compliant with requirements but also can be deployed with confidence in its ability to handle real-world complexities and challenges.
There’s plenty of other editorial on our sister site, Electronic Specifier! Or you can always join in the conversation by commenting below or visiting our LinkedIn page.