Incident Summary
On 4 July 2026, an incident affected our Finance Services between approximately 17:15 and 19:40. During this window, some customers may have experienced errors or degraded performance.
Root Cause
A routine interaction with the platform triggered a previously undetected defect in an application component. This caused the application to enter a processing loop, generating excessive load and destabilising the platform.
Resolution
Once the issue was identified, incident response was initiated. The condition that triggered the defect was identified and corrected, and the affected services were restarted to restore normal operation. Service was confirmed as fully restored by 20:40, and subsequent monitoring confirmed the platform has remained stable.
Timeline (all times BST)
-
17:14 – A routine interaction was made within the platform.
-
17:15 – This triggered a previously undetected defect, causing application services to enter a processing loop and fail.
-
18:25 – The issue was identified and incident response was initiated.
-
18:27 – An initial service recovery was attempted. This was unsuccessful, as services remained blocked by requests pinned from the original failure.
-
18:40 – A full platform restart was initiated, with root cause investigation continuing in parallel.
-
19:30 – Investigation concluded. The triggering configuration was identified and its corrected state verified.
-
19:40 – Platform restart completed and stability confirmed. Service fully restored.
Preventative Actions
To reduce the likelihood and impact of a similar incident in future, we are:
- Deploying a permanent fix for the underlying defect, which has been prioritised in our development roadmap.
- Enhancing our existing monitoring to detect this specific condition more quickly, based on findings from this incident.
- Reviewing our alerting thresholds and processes.