How do we actually use AI in high-reliability system design?
Clearly, AI has rapidly become an incredible engineering tool. Today’s models can generate (near) production-quality code, review designs, summarize complex documentation, and accelerate many software development tasks. The question is no longer whether AI is useful, but how it should be used when developing high-reliability systems.
In industries such as healthcare, aerospace, and industrial control, software development follows a rigorous engineering process. Requirements flow into the system design, which is implemented and then verified through Verification and Validation (V&V). Every requirement must be traceable to the final implementation, and the behavior of the system must be predictable and verifiable.
This is where AI introduces a fundamental challenge. Generative AI produces probabilistic rather than deterministic outputs. While the results at first glance appear excellent, they cannot be guaranteed to be correct for every situation, and their failure modes are difficult to fully characterize.
Safety-critical systems are engineered to extremely low probabilities of hazardous failure: often on the order of one failure in millions or even billions of operating hours, depending on the application and applicable standards. Modern AI systems are not designed or certified to provide those kinds of guarantees.
The challenge is not just that AI can occasionally produce an incorrect answer. It’s also difficult to systematically analyze all of the ways it might fail. This is particularly important for engineering activities such as Failure Mode and Effects Analysis (FMEA), where engineers identify, analyze, and mitigate every credible failure mode. Whether evaluating an infusion pump, an aircraft controller, or another safety-critical system, understanding how a system can fail is just as important as understanding how it normally operates.
How can you be confident in a product if you don’t fully understand how it can fail?
Fortunately, this does not mean AI has no place in high-reliability engineering. In fact, it can be an extremely powerful development tool.
AI can accelerate architecture exploration, draft code, generate unit tests, review implementations, explain legacy code, and assist with documentation. However, the resulting artifacts must still pass the same engineering process as if they had been written entirely by a human. They require design reviews, verification, testing, traceability, and validation before becoming part of the final product.
The important distinction is between using AI to build a system and allowing AI to operate the system.
For example, an engineer might use AI to help develop the calculation algorithm for an infusion pump. Through testing, review, and verification, that algorithm can be refined into deterministic software that produces the same outputs for the same inputs every time. Once verified, the deployed software no longer depends on AI; it is simply conventional, deterministic code.
This represents a conservative engineering approach and is appropriate for systems where the consequences of failure are severe.
Using AI as part of the runtime decision-making process is a fundamentally different problem. Because the model itself becomes part of the operational system, its probabilistic behavior must also be considered part of the system’s safety case. Existing regulatory frameworks are largely built around deterministic software, traceability, and repeatable verification, making certification of autonomous AI significantly much more challenging.
In lower-risk applications, organizations may decide that the benefits of runtime AI outweigh the associated risks, particularly when a qualified human remains responsible for reviewing or approving important decisions. As the severity of potential failures increases, however, the acceptable level of uncertainty decreases dramatically.
AI will undoubtedly continue to improve, and its role in engineering will continue to expand. However, for high-reliability industries, widespread adoption of AI as a runtime decision-maker is likely to be gradual. In the foreseeable future, AI’s greatest value will be as a tool that helps engineers build better deterministic systems, not as the deterministic system itself.
How are you using AI in your product development? Have you found a balance between accelerating development and maintaining the level of reliability your application requires?