In the aftermath of the Horizon IT scandal, a seismic event in the UK’s legal landscape, the scrutiny extends beyond the legal ramifications to the technical intricacies that precipitated the crisis. Horizon wrongly calculated money discrepancies in post office branches. Hundreds of sub-postmasters and postmistresses were wrongly prosecuted, leading to imprisonment, financial ruin, and personal hardships. The UK government has pledged to overturn convictions and compensate victims, with a budget of £1 billion allocated for compensation. The scandal involves the Post Office and technology firm Fujitsu, as Horizon was developed by Fujitsu and introduced by the Post Office in 1999. A public inquiry is now ongoing in 2024, and compensation schemes have been established for the affected individuals. This article delves into Horizon’s design flaws and explores nuanced software development approaches that could have averted the controversy.
Deciphering Horizon:
Originally conceived as an electronic point-of-sale (Epos) system to usher in a new era of efficiency for post office operations in the UK, Horizon’s implementation faltered under the weight of critical design flaws, leading to tumultuous repercussions for post office operators. This analysis aims to illuminate the technical shortcomings that played a pivotal role in the scandal’s unfolding.
The Envisaged Advantages:
Horizon’s ambitious vision promised to revolutionize post office operations. Replacing paper receipts with an electronic database should have saved post office operators time and effort, allowing them to manage their accounts at the push of a button. At the simplest end, for instance, the Horizon system could collate all the transactions over the course of a month and calculate how much cash was still expected to be in the Post Office’s coffers.
Its failure to live up to expectations underscored the imperative for robust development practices. The desire to digitize transactions and streamline financial management became a cautionary tale.
Identifying the Flaws:
Insights from the development team exposed a litany of issues within the Horizon system, with bug counts reaching the hundreds as early as 2001. The lack of proficiency among team members became a critical factor in the unraveling of the system.
Learning from Mistakes: Software Development Strategies
At Impekable we have worked on a number of very large scale projects including financial services organizations. Some our best practices that could have prevented many of the big challenges experienced with the Horizon project are;
1. Strangler Pattern Implementation:
The Strangler Pattern, also known as Strangler Fig Application, is a software development and migration strategy that allows for the gradual replacement or refactoring of an existing system with a new one. This pattern is particularly useful when dealing with legacy systems that need to be updated, replaced, or migrated to a modern architecture without disrupting the entire system at once. Initiate with a phased approach, focusing on a single transaction, and employ the Strangler pattern to run it parallell with the existing system.
Incremental implementation mitigates risks, facilitating early issue identification and resolution. Prioritize isolating and resolving issues in a controlled environment. In the past when adopting the Strangler patter we have found the following approach useful (make this a digram)
Identifying the Target Functionality:
Begin by identifying specific functionalities or components within the existing system that need to be replaced, upgraded, or refactored. These are the areas where the new system will be introduced.
Building Parallel Functionality:
Develop new components or services that replicate the identified functionalities. These new components should be designed to coexist with the existing system without disrupting its operation.
Incremental Rollout:
Instead of replacing the entire system at once, the new components are gradually introduced in a controlled and incremental manner. This can be done feature by feature, module by module, or through other logical divisions.
Routing Traffic:
Implement mechanisms to route specific transactions or requests to either the existing system or the new components based on the chosen rollout strategy. This could involve load balancers, routing rules, or other routing mechanisms.
Continuous Monitoring and Validation:
Throughout the incremental rollout, closely monitor the behavior and performance of both the existing and new components. Automated tests, user acceptance testing, and monitoring tools play a crucial role in ensuring that the new components meet expectations.
Iterative Process:
The process is iterative, allowing for continuous refinement and improvement of the new components. Feedback from users and system monitoring help identify areas for optimization and enhancement.
Complete Replacement:
Over time, as more functionalities are replaced, and confidence in the new system grows, the balance gradually shifts. Eventually, the entire system is replaced, and the old components are no longer in use.
2. Quality Integration through Test Driven Development (TDD):
Embrace Test Driven Development (TDD) by prioritizing small user stories with explicit acceptance criteria.
Automated testing within a Continuous Integration/Continuous Deployment (CI/CD) process establishes a robust foundation, fostering quicker responses to evolving requirements and reducing technical debt. This approach would not have resulted in the situation of Horizon found itself in with 1000’s of critical bugs.
3. Responsibly Scaling Systems:
Approach scaling with caution, emphasizing a phased and iterative rollout.
Decompose the problem into smaller components, addressing issues incrementally to reduce the risk of systemic failures. Consider incorporating feature toggles to enable or disable functionalities seamlessly.
4. Transparent Communication:
Cultivate open lines of communication between developers, stakeholders, and end-users, ensuring a shared understanding of system capabilities and limitations.
Establish clear channels for reporting and addressing issues promptly, fostering a culture of transparency, accountability, and collaborative problem-solving.
5. Continuous Learning and Adaptation:
Cultivate a culture of continuous improvement, regularly reassessing and adapting development practices based on feedback, industry best practices, and evolving technologies.
Prioritize ongoing training and professional development for the development team to stay abreast of advancements in software engineering. Encourage cross-functional learning to promote a holistic understanding of the system.
6. User-Centric Design and Testing:
Prioritize user-centric design principles, incorporating user feedback throughout the development lifecycle.
Implement user acceptance testing (UAT) as an integral part of the development process, ensuring that the system aligns with user expectations and needs.
The Horizon IT scandal serves as a powerful reminder of the critical importance of robust software development practices. By embracing a comprehensive set of strategies, including phased approaches, quality integration, responsible scaling, transparent communication, continuous learning, and user-centric design, the industry can learn from the intricate mistakes of Horizon. This multifaceted approach not only guards against catastrophic outcomes but also fosters a culture of innovation, resilience, and adaptability in the dynamic landscape of software development.
