We fully recovered. Code reviewer workers are now steadily back to normal.
Posted Jul 08, 2023 - 07:19 BST
Update
We have been closely monitoring the situation and have observed an highly improved performance since the rollout of the second patch. However, we want to ensure transparency and will continue to monitor the situation closely keeping the incident open for more 8 hours.
Posted Jul 07, 2023 - 23:20 BST
Update
We have been rolling out a second patch and we are monitoring code reviewer workers recovery.
Posted Jul 07, 2023 - 22:22 BST
Update
During today monitoring, our team have identified an additional factor contributing to the system's slowness and are currently working on developing a patch to address it.
Posted Jul 07, 2023 - 18:02 BST
Update
We are continue monitoring the situation that is slowly recovering.
Posted Jul 07, 2023 - 11:18 BST
Monitoring
We are rolling out a fix meanwhile handling the backlog of pending requests generated by the unexpected performance issue.
Posted Jul 07, 2023 - 06:55 BST
Update
We are testing the fix to get ready for its rollout as soon as possible.
Posted Jul 06, 2023 - 22:54 BST
Identified
We identified a clear source for performance degradation on worker filesystem usage and we are working on its fix.
Posted Jul 06, 2023 - 20:43 BST
Update
We are continuing to investigate this issue.
Posted Jul 06, 2023 - 17:16 BST
Investigating
We are currently experiencing degraded performance affecting our code review workers.
Posted Jul 06, 2023 - 17:16 BST
This incident affected: Clayton (Code Review Workers).