On November 30, 2023, the Engineering team began to review customer reports of extensions errors for authorization requests impacting customers. During the impact period, users would have received 503 ‘Service Temporarily Unavailable’ errors in logs as well as an ‘UnauthorizedError: Authorization extension’ error. Following triage by the Engineering team, the issue was resolved by scaling up necessary legacy resourcing to serve traffic which had been previously scaled down as part of planned maintenance activities. We sincerely apologize for any impact this had on you and your users.
The root cause of this issue was due to a misconfiguration discovered by the Engineering team whereby active services in use were pointed to legacy infrastructure which was deliberately scaled down as part of previously planned routine maintenance activities. Once the Engineering team discovered this misconfiguration, the necessary resourcing, including the appropriate auto-scaling groups, were scaled back up to provide underlying hosts which could service traffic as needed.
To avoid similar incidents from happening in the future, the following actions were taken:
● Repair the misconfiguration which was the root cause of this issue for legacy infrastructure calls following these maintenance activities.
● Update theplaybooks for these maintenance activities to include additional checks before infrastructure scaling.
● Review the viability of an optimized revert script for these maintenance activities in the future.