API Resource Usage & Delay to Email Notifications
Resolved
Oct 08 at 10:12am BST
Impact
A core database was unavailable for approximately 1 hour 41 minutes
- Daily office summary notifications were not delivered
- Incomplete Shift Notifications we not delivered
- Applications experienced connectivity errors during this period
Note: Daily notifications affected by this incident will not be resent as the information may now be outdated. Normal notification delivery will resume tomorrow.
What Happened
At 06:22 UTC, a core database service was terminated when resource usage exceeded available system capacity. The service required manual intervention to restart at 08:03 UTC.
Resolution
We have implemented the following preventative measures:
- Added emergency resource buffer to handle usage spikes
- Reduced database resource allocation to provide better headroom
- Enabled automatic service restart on this specific error to eliminate manual intervention requirements
- Enhanced resource monitoring for early warning of potential issues
All services are now operating normally with improved resilience.
Current Status
✅ Resolved - All services operating normally.
Affected services