tag:status.clayton.io,2005:/historyClayton Status - Incident History2024-03-29T07:51:27+00:00Claytontag:status.clayton.io,2005:Incident/198751082024-01-31T14:41:05+00:002024-01-31T14:41:05+00:00Website not reachable<p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>14:41</var> GMT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>13:21</var> GMT</small><br><strong>Monitoring</strong> - The team is monitoring the traffic load on the website and REST APIs.</p><p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>12:06</var> GMT</small><br><strong>Identified</strong> - The problem has been identified in third parties overloading our backend infrastructure of API requests, the team has mitigated the issue and monitoring the situation.</p><p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>11:46</var> GMT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>11:46</var> GMT</small><br><strong>Investigating</strong> - The website is not currently reachable or shows only the loading indicator. The engineering team is investigating the underlying root cause.</p>tag:status.clayton.io,2005:Incident/197648992024-01-19T20:07:12+00:002024-01-19T20:07:12+00:00"Sharing Violation" dismissed issues are erroneously reopened<p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>20:07</var> GMT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>17:05</var> GMT</small><br><strong>Monitoring</strong> - The engineering team is monitoring the progress on recovering scans.</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>14:11</var> GMT</small><br><strong>Identified</strong> - Our team has successfully identified the root cause of the problem. Contrary to our initial understanding, the issue did not involve the reopening of dismissed issues. Instead, it was traced back to a failure in the logic responsible for identifying test classes, leading to the erroneous detection of issues within these classes.<br /><br />Our engineering team has addressed this underlying problem, and the necessary fixes have been implemented. The service is now restored to its intended functionality, we will now begin the recovery process for scans that were affected by this problem.</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>12:26</var> GMT</small><br><strong>Investigating</strong> - Following the recent release of our service, we have identified a regression that has led to the detection of certain issues previously dismissed. At present, these issues appear to be associated with the "Sharing Violation" rule. Our team is actively investigating the root cause of this problem.</p>tag:status.clayton.io,2005:Incident/177607652023-07-17T00:00:17+01:002023-07-17T00:00:18+01:00Scheduled infrastructure maintenance<p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>00:00</var> BST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'>15</var>, <var data-var='time'>08:00</var> BST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jul <var data-var='date'> 4</var>, <var data-var='time'>12:06</var> BST</small><br><strong>Scheduled</strong> - We will operate an infrastructure update.</p>tag:status.clayton.io,2005:Incident/177816412023-07-08T07:19:59+01:002023-07-08T07:19:59+01:00Degraded performance<p><small>Jul <var data-var='date'> 8</var>, <var data-var='time'>07:19</var> BST</small><br><strong>Resolved</strong> - We fully recovered. Code reviewer workers are now steadily back to normal.</p><p><small>Jul <var data-var='date'> 7</var>, <var data-var='time'>23:20</var> BST</small><br><strong>Update</strong> - We have been closely monitoring the situation and have observed an highly improved performance since the rollout of the second patch. However, we want to ensure transparency and will continue to monitor the situation closely keeping the incident open for more 8 hours.</p><p><small>Jul <var data-var='date'> 7</var>, <var data-var='time'>22:22</var> BST</small><br><strong>Update</strong> - We have been rolling out a second patch and we are monitoring code reviewer workers recovery.</p><p><small>Jul <var data-var='date'> 7</var>, <var data-var='time'>18:02</var> BST</small><br><strong>Update</strong> - During today monitoring, our team have identified an additional factor contributing to the system's slowness and are currently working on developing a patch to address it.</p><p><small>Jul <var data-var='date'> 7</var>, <var data-var='time'>11:18</var> BST</small><br><strong>Update</strong> - We are continue monitoring the situation that is slowly recovering.</p><p><small>Jul <var data-var='date'> 7</var>, <var data-var='time'>06:55</var> BST</small><br><strong>Monitoring</strong> - We are rolling out a fix meanwhile handling the backlog of pending requests generated by the unexpected performance issue.</p><p><small>Jul <var data-var='date'> 6</var>, <var data-var='time'>22:54</var> BST</small><br><strong>Update</strong> - We are testing the fix to get ready for its rollout as soon as possible.</p><p><small>Jul <var data-var='date'> 6</var>, <var data-var='time'>20:43</var> BST</small><br><strong>Identified</strong> - We identified a clear source for performance degradation on worker filesystem usage and we are working on its fix.</p><p><small>Jul <var data-var='date'> 6</var>, <var data-var='time'>17:16</var> BST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jul <var data-var='date'> 6</var>, <var data-var='time'>17:16</var> BST</small><br><strong>Investigating</strong> - We are currently experiencing degraded performance affecting our code review workers.</p>tag:status.clayton.io,2005:Incident/169291822023-04-22T08:58:07+01:002023-04-22T08:58:07+01:00Degraded performance<p><small>Apr <var data-var='date'>22</var>, <var data-var='time'>08:58</var> BST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>22</var>, <var data-var='time'>08:57</var> BST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Apr <var data-var='date'>21</var>, <var data-var='time'>11:01</var> BST</small><br><strong>Investigating</strong> - We are currently experiencing degraded performance affecting our web app.</p>tag:status.clayton.io,2005:Incident/168916492023-04-17T17:09:35+01:002023-04-17T17:09:35+01:00Degraded performance<p><small>Apr <var data-var='date'>17</var>, <var data-var='time'>17:09</var> BST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>17</var>, <var data-var='time'>16:44</var> BST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Apr <var data-var='date'>17</var>, <var data-var='time'>15:59</var> BST</small><br><strong>Investigating</strong> - We are currently experiencing degraded performance affecting our web app and code reviews.</p>tag:status.clayton.io,2005:Incident/111811452022-10-17T09:09:50+01:002022-10-17T09:09:50+01:00Scheduled release<p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>09:09</var> BST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>08:31</var> BST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'>20</var>, <var data-var='time'>09:30</var> BST</small><br><strong>Scheduled</strong> - Scheduled maintenance as part of our periodic release cycle.</p>tag:status.clayton.io,2005:Incident/107831792022-08-08T18:01:22+01:002022-08-08T18:01:22+01:00Code review performance degradation<p><small>Aug <var data-var='date'> 8</var>, <var data-var='time'>18:01</var> BST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'> 8</var>, <var data-var='time'>16:57</var> BST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Aug <var data-var='date'> 8</var>, <var data-var='time'>15:29</var> BST</small><br><strong>Investigating</strong> - We are currently investigating an issue affecting the performance of code reviews.</p>tag:status.clayton.io,2005:Incident/107528382022-08-03T22:01:32+01:002022-08-03T22:01:32+01:00Code review workers performance degradation<p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>22:01</var> BST</small><br><strong>Resolved</strong> - The performance of code review workers is back to normal.</p><p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>18:06</var> BST</small><br><strong>Monitoring</strong> - We allocated more workers to recover and the situation is getting back to normal.</p><p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>15:11</var> BST</small><br><strong>Identified</strong> - Due to a new functionality the workload on code review workers is higher than expected we are working to mitigate the issue.</p>tag:status.clayton.io,2005:Incident/100817962022-05-25T08:40:00+01:002022-05-26T18:41:07+01:00DNS issue<p><small>May <var data-var='date'>25</var>, <var data-var='time'>08:40</var> BST</small><br><strong>Resolved</strong> - The web application is not reachable due to a DNS configuration issue</p>tag:status.clayton.io,2005:Incident/97562282022-04-11T08:34:28+01:002022-04-11T09:09:25+01:00DNS outage - clayton.io unreachable<p><small>Apr <var data-var='date'>11</var>, <var data-var='time'>08:34</var> BST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>11</var>, <var data-var='time'>08:21</var> BST</small><br><strong>Update</strong> - Our team was able to issue a fix for the DNS outage we experienced this morning. We are seeing error rates dropping significantly, however, like with anything DNS-related, it may take some time for the fix to propagate. Please bear with us.</p><p><small>Apr <var data-var='date'>11</var>, <var data-var='time'>08:10</var> BST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Apr <var data-var='date'>11</var>, <var data-var='time'>07:57</var> BST</small><br><strong>Identified</strong> - We are currently experiencing a DNS issue that is making our web application unreachable.</p><p><small>Apr <var data-var='date'>11</var>, <var data-var='time'>07:51</var> BST</small><br><strong>Investigating</strong> - We are currently investigating a DNS issue that is preventing our web application from being reached.</p>tag:status.clayton.io,2005:Incident/97396032022-04-09T15:18:24+01:002022-04-09T15:18:24+01:00Weekend maintainance<p><small>Apr <var data-var='date'> 9</var>, <var data-var='time'>15:18</var> BST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Apr <var data-var='date'> 9</var>, <var data-var='time'>15:15</var> BST</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Apr <var data-var='date'> 9</var>, <var data-var='time'>15:00</var> BST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'> 8</var>, <var data-var='time'>15:37</var> BST</small><br><strong>Scheduled</strong> - We are performing some maintenance tasks. During these tasks the system might become unavailable for some time.</p>tag:status.clayton.io,2005:Incident/88001652021-12-10T10:35:55+00:002021-12-10T10:35:55+00:00Code review performance degradation<p><small>Dec <var data-var='date'>10</var>, <var data-var='time'>10:35</var> GMT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>10</var>, <var data-var='time'>06:17</var> GMT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Dec <var data-var='date'> 9</var>, <var data-var='time'>10:50</var> GMT</small><br><strong>Investigating</strong> - We are currently investigating an issue affecting the performance of code reviews.</p>tag:status.clayton.io,2005:Incident/75862162021-07-23T18:01:38+01:002021-07-23T18:01:38+01:00Code review processing delays<p><small>Jul <var data-var='date'>23</var>, <var data-var='time'>18:01</var> BST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>23</var>, <var data-var='time'>17:35</var> BST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jul <var data-var='date'>23</var>, <var data-var='time'>16:45</var> BST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.clayton.io,2005:Incident/73684892021-06-30T15:30:24+01:002021-06-30T15:30:24+01:00High review failure rate and processing delays<p><small>Jun <var data-var='date'>30</var>, <var data-var='time'>15:30</var> BST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>30</var>, <var data-var='time'>11:48</var> BST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jun <var data-var='date'>30</var>, <var data-var='time'>09:38</var> BST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p>tag:status.clayton.io,2005:Incident/63024152021-02-17T20:15:36+00:002021-02-17T20:15:36+00:00Review Processing Delays<p><small>Feb <var data-var='date'>17</var>, <var data-var='time'>20:15</var> GMT</small><br><strong>Resolved</strong> - Our processing backlog is now clear and we are processing requests as normal.</p><p><small>Feb <var data-var='date'>17</var>, <var data-var='time'>16:47</var> GMT</small><br><strong>Investigating</strong> - Our code review processing infrastructure is running behind which is causing latency in the reporting and slower-than-normal response times.</p>tag:status.clayton.io,2005:Incident/62743662021-02-15T21:16:05+00:002021-02-16T14:18:12+00:00Review Processing Delays<p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>21:16</var> GMT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>21:00</var> GMT</small><br><strong>Update</strong> - Processing is gradually going back to normal levels. We keep monitoring the situation.</p><p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>19:53</var> GMT</small><br><strong>Update</strong> - We have released a fix and continue monitoring our instances as we are slowly clearing up our backlog of requests</p><p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>18:19</var> GMT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>17:13</var> GMT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p><p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>16:59</var> GMT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>16:30</var> GMT</small><br><strong>Monitoring</strong> - We are making some unplanned maintenance in order to address today's issues - we'll continue to post updates on the situation</p><p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>14:00</var> GMT</small><br><strong>Investigating</strong> - Our code review processing infrastructure is running behind which is causing latency in the reporting and slower-than-normal response times.</p>tag:status.clayton.io,2005:Incident/61736062021-02-02T18:15:26+00:002021-02-02T18:15:26+00:00High review failure rate<p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>18:15</var> GMT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>16:42</var> GMT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>14:59</var> GMT</small><br><strong>Investigating</strong> - We are currently investigating a high failure rate in our code review tasks.</p>tag:status.clayton.io,2005:Incident/55042232020-11-05T18:30:00+00:002020-11-06T15:27:18+00:00Code review tasks remain queued for a very long time<p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>18:30</var> GMT</small><br><strong>Resolved</strong> - A problem is causing code reviews jobs to remain queued for an excessively long time before being processed.</p>tag:status.clayton.io,2005:Incident/51580112020-09-23T19:34:23+01:002020-09-23T19:34:23+01:00Degraded performance<p><small>Sep <var data-var='date'>23</var>, <var data-var='time'>19:34</var> BST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>10:30</var> BST</small><br><strong>Investigating</strong> - We are currently experiencing long waiting times for code reviews.</p>tag:status.clayton.io,2005:Incident/43980872020-06-18T18:24:54+01:002020-06-18T18:24:54+01:00High review failure rate<p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>18:24</var> BST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>17:05</var> BST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>15:54</var> BST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>11:56</var> BST</small><br><strong>Investigating</strong> - We are currently investigating an high failure rate in our code reviews.</p>tag:status.clayton.io,2005:Incident/40266582020-05-04T14:27:22+01:002020-05-04T14:27:22+01:00Code review workers issues<p><small>May <var data-var='date'> 4</var>, <var data-var='time'>14:27</var> BST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'> 4</var>, <var data-var='time'>11:46</var> BST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>May <var data-var='date'> 4</var>, <var data-var='time'>08:20</var> BST</small><br><strong>Investigating</strong> - We are currently investigating issues affecting the code review workers.</p>tag:status.clayton.io,2005:Incident/34636512020-01-24T13:26:35+00:002020-01-24T13:26:35+00:00High Review Failure Rate<p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>13:26</var> GMT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>09:44</var> GMT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>09:22</var> GMT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>20:37</var> GMT</small><br><strong>Investigating</strong> - We are currently investigating an issue that is causing a high failure rate from our code review workers.</p>tag:status.clayton.io,2005:Incident/32614632019-11-27T19:16:57+00:002019-11-28T11:14:48+00:00Incorrect review locks<p><small>Nov <var data-var='date'>27</var>, <var data-var='time'>19:16</var> GMT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>27</var>, <var data-var='time'>17:57</var> GMT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Nov <var data-var='date'>27</var>, <var data-var='time'>15:37</var> GMT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Nov <var data-var='date'>27</var>, <var data-var='time'>14:45</var> GMT</small><br><strong>Investigating</strong> - We are currently investigating a malfunctioning of our web app that could cause some of the reviews to be incorrectly locked, and developers to be wrongly reported as unlicensed in some circumstances.<br /><br />Please contact us immediately if this is affecting you so we can provide further assistance.</p>tag:status.clayton.io,2005:Incident/29010812019-08-23T09:46:09+01:002019-08-23T09:46:09+01:00Performance issues<p><small>Aug <var data-var='date'>23</var>, <var data-var='time'>09:46</var> BST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>22</var>, <var data-var='time'>15:43</var> BST</small><br><strong>Investigating</strong> - We are currently investigating a performance degradation incident.</p>