Council Post: Software Too Big to Fail; A Case for Independent Software Verification and Validation

In today's interconnected world, software systems underpin critical functions across virtually every industry. From financial services and healthcare to transportation and communication, these systems are foundational to our daily business operations and lives. When software reaches a level of ubiquity and criticality, it becomes "too big to fail." The failure of such software could lead to significant business and personal disruptions, highlighting the imperative for rigorous, independent verification and validation (IV&V). This methodology, successfully implemented in various market verticals, enhances software reliability and safety, ensuring these indispensable systems remain robust and secure. What is "Too Big to Fail"? The phrase " Too big to fail " describes businesses or sectors so deeply embedded in a financial system or economy that their collapse would be catastrophic. This concept gained prominence during the 2007-2008 financial crisis, when the failure of major banks necessitated substantial government intervention to prevent economic disaster. In response, the United States passed the Dodd-Frank Act in 2010 , imposing stringent regulatory requirements on financial institutions deemed "too big to fail". The Federal Reserve maintains a list of these systemically important entities. The concept of "too big to fail," which originally applied to financial institutions, has now expanded to encompass critical software systems. This shift was starkly illustrated on July 19, 2024, when CrowdStrike , a leading cybersecurity firm, experienced a catastrophic incident due to a faulty software update. The update caused widespread disruptions, affecting approximately 8.5 million Microsoft Windows devices globally, originating from a misconfiguration in CrowdStrike's Falcon sensor software. The impact of this incident was profound and far-reaching, affecting critical sectors including transportation, healthcare, and financial services. It led to grounded flights, delayed medical procedures, and interrupted banking services, demonstrating the interconnectedness and vulnerability of modern technology systems. Despite CrowdStrike's quick response in reverting the faulty update and collaborating with major cloud providers – like Microsoft Azure and Google Cloud Platform – to mitigate the effects, the incident's repercussions were severe, widespread and long-lasting. This global software failure at CrowdStrike serves as a compelling example of how certain software platforms have become "too big to fail" in their own right. The incident's impact on a wide range of industries – from banking and retail to media and healthcare – underscores the critical nature and global ubiquity of these systems. Some organizations continued to struggle with recovery long after the initial incident, highlighting the need for resilient disaster recovery and incident response plans. Whether caused by human error or artificial intelligence (AI) malfunction, such incidents demonstrate the potential for widespread disruption when critical software systems fail. This event has brought into sharp focus the need for enhanced safeguards, stricter oversight, and potentially new regulatory frameworks to manage the risks associated with software that has become integral to global economic operations and society. This realization calls for a regulatory framework akin to the Dodd-Frank Act, but tailored for pervasive, critical software systems. The goal should be to reduce the likelihood of catastrophic failures to six sigma-level events – a standard of six standard deviations from the mean (i.e., a truly rare occurrence). Independent Verification and Validation (IV&V): Ensuring Software Quality and Reliability Verification and Validation (V&V) are essential processes in software development. Verification asks, "Are we making the product right?" while validation asks, "Are we making the right product?" In other words, verification ensures that software functions correctly according to its design specifications, whereas validation confirms that the software meets the broader needs and expectations of its users. The "I" in IV&V stands for Independent, meaning that an external third party performs these functions. This independent entity has separate leadership, software analysis teams, and funding sources distinct from the organization whose software is being evaluated, removing conflicts of interest and allowing for a more robust V&V process. NASA 's Independent Verification and Validation (IV&V) Program exemplifies this rigorous approach in action, particularly in the critical field of space exploration. The IV&V Facility has significantly contributed to the success of various missions, including International Space Station operations, Mars Rover software, and Extravehicular Mobility Unit (spacesuit) hardware-software integration. In these contexts, software reliability is crucial for both mission success and astronaut safe

Council Post: Software Too Big to Fail; A Case for Independent Software Verification and Validation

What is "Too Big to Fail"?

Keep reading. It's free.

Related stories