Real risks often arise where a system has never failed before, according to experts at Grymaxion Plovdiv.
July 2024. CrowdStrike, a leading cybersecurity firm, released a routine update. A single unvalidated configuration change led to a failure. It was not a hacker attack. Not a virus. Not an external threat. A routine update that knocked out 8.5 million devices worldwide. Airlines, banks, hospitals, government services. For the 500 largest US companies alone, direct losses amounted to around $5.4 billion.
Following this, thousands of IT companies stepped up their testing of updates. Logical. But here’s the question: what will cause the next major outage? Will it come from updates? Most likely not. There is a well-known saying: generals always prepare for the last war. The same thing happens in software development – we solve yesterday’s problem, note the experts at Grymaxion.
Applications and service software must always be available to the user. A system failure is always a test of resilience. And anyone with any connection to the IT industry will tell you that it is impossible for a software product to operate 100% trouble-free. But does the client understand this, and do they realise the true cost of ensuring maximum uptime?
GRYMAXION Ltd. believes that the stability and reliability of a service is demonstrated not by the absence of failures, but by how early the warning signs that could lead to them are detected.
Why the ‘next failure’ is not like the last one
Complex systems do not fail at a single point. They fail at the junctions. In risk management, this is described by James Rison’s ‘Swiss Cheese Model’: each layer of defence (testing, code review, monitoring, the release process) is a slice of cheese. There are many layers of protection between a threat and an incident, and each has flaws (‘holes’) which, if they overlap, can lead to a failure. A failure is the moment when the weaknesses of several layers happen to line up.
Every incident is unique; it is a combination of coinciding factors. Fixing or restoring one ‘hole’ does not close the others. Therefore, when we look back and analyse the past, everything always seems ‘predictable’ – this is hindsight bias, not analytics.
Near-misses as a source of alerts
The database is approaching its disk capacity limit. The scaling system barely managed to cope with the load. A configuration error was detected before the application was launched. Monitoring detected an anomaly, but the service continued to operate. From a business perspective, nothing happened.
But for the engineering team, ‘near-miss’ incidents are valuable signals. A deployment rolled back at the last minute. An alert that was ignored. A load that nearly brought the service down. A configuration that ‘accidentally’ didn’t make it into the production environment.
The problem is that a near-miss has no status, no severity level, no ticket and no person in charge, according to the experts at Grymaxion Business Solutions. Formal investigation processes are tailored to actual incidents – to what has already gone wrong. But ‘near-misses’ can quickly be forgotten. Yet it is precisely these situations that can point to future failures. And these are not specific bugs, but risk configurations that have not yet materialised.
Metrics that look back, and metrics that look ahead
Most teams measure reliability using standard indicators. The average time it takes a team to detect a failure is known in the industry as MTTD (Mean Time to Detect). The average time it takes to resolve a fault is known as MTTR (Mean Time to Repair). These are useful, but they share a common problem: experts at Grymaxion Ltd believe that these are all metrics of the past. They measure what has already happened.
What’s more, data from the VOID (Verica Open Incident Database) open incident database shows that even mean recovery time is an unreliable metric. The distribution of incidents by duration is heavily skewed: most are resolved quickly, but a few cases drag on for a disproportionately long time. In such a situation, the average value does not reflect the real picture and creates a false sense of control.
Historical metrics are important, but they are not enough. Leading indicators – those that point to the state of the system prior to a failure – paint a different picture. How many ‘near misses’ were recorded in a month? How many reviews were conducted? What percentage of findings actually turned into specific tasks, rather than remaining on paper? A team that measures only the past sees only the past.
Reliability isn’t about preventing recurrence; it’s about being prepared for the unknown
Near-misses, load tests, architectural reviews and failure experiments – all these tools are designed to resolve issues before they escalate into incidents.
Grymaxion Business Solutions knows how to ask the tough questions before they become urgent. Our team has developed a system for tracking and analysing near-misses – because that is where the clues lie as to what might break tomorrow.
A truly mature engineering culture is not about perfect code or 100% uptime. It is about staying one step ahead and thinking of reliability as a process rather than an outcome, and understanding:
- where the system might reach its limits
- which component will become a bottleneck as the load increases
- which scenarios still need to be tested
- what problems might arise when the architecture changes
- what kind of incident an identified and logged anomaly could lead to
- why the system came close to failure, how this was detected, and what helped prevent the incident