User Authentication Issue
Incident Report for Measuremen
Postmortem

On February 14, 2024, our users experienced a service disruption when attempting to log in. The impacted users would have experienced intermittent logins and received 5xx error messages between 17:38 and 18:38 UTC. We sincerely apologize for the impact this has caused you. On February 14th, at 17:30 UTC, Engineers made back-end updates to the infrastructure’s extensibility repository, introducing a regression. This implementation inadvertently applied previous DNS record values that targeted our the wrong infrastructure. These environments had since been scaled down, resulting in the 100% requests failing with HTTP 503 errors. DNS records were manually updated to proxy via the network edge provider to mitigate the impact. Furthermore, records were synchronised accordingly to avoid similar issues in the future.

Posted Feb 22, 2024 - 19:45 CET

Resolved
This incident has been resolved.
Posted Feb 14, 2024 - 22:24 CET
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Feb 14, 2024 - 19:38 CET
Investigating
We are currently investigating this issue.
Posted Feb 14, 2024 - 18:38 CET
This incident affected: Measuremen Authentication (User Authentication) and Measuremen Portal.