This incident began approximately on Monday 15/02/2021 13:50 and ended approximately on Monday 15/02/2021 21:00 (all times are GMT).

Description of the impact

The new Developer Experience had a slow query impacting customers migrating on such a platform. As more customers were being migrated more workloads were generated and customers also not on the new developer experience started experiencing delays on review processing.

Timeline and Root Cause

• Starting from 15/02/2021 11:45 GMT the CPU Utilisation p99 went up to 99% therefore the cluster has been scaled up for the first time.
• Starting from 13:50 GMT also the CPU Utilisation p50 went up to 99% starting the actual incident. At 14:00 GMT the product engineering team started to investigate the issue as the first deployed a new detailed monitoring configuration to gain more visibility on the database cluster.
• At 16:30 GMT we were able to identify a query that was causing the congestion on the database and therefore delays on the review processing queue.
• At 18:19 GMT we deployed the patch and also scaled up the cluster to speed up recovery time.
• At 21:16 GMT incident has been resolved since review processing times were back to normal levels consistently

Detection, Remediation and Prevention

• In the short term, we are focusing on additional database monitoring improvements and query optimisations.
• In the long term, our engineering team is planning to introduce deeper performance testing early in the development phase.