Increased error rate
March 08, Sun
Affected Service:
Main Website
-
19:26, Mar 07 CESTIssue IdentifiedWe are seeing an elevated 5xx error rate on the main site. We are aware of the root cause of this incident due to this happening in the past. Our team will be online soon to mitigate the issue.
-
23:17, Mar 07 CESTOn HoldCurrently awaiting response from the provider in order to identify what's causing the service to reach its inode limit.
-
23:46, Mar 07 CESTWorkingWe're attempting to fix the issue while we wait for a response.
-
23:48, Mar 07 CESTIssue IdentifiedUPDATE: We just received a response and are working to resolve the issue in a timely manner.
-
00:13, Mar 08 CESTOn HoldWe have reached out to our provider again as this issue has reached a point where it's beyond our control.
The incident has been caused by the main email account on cPanel being flooded with emails. Ideally, this should not cause much of an issue, but all messages are stored on disk and each of them counts as a single inode, attachments counting as another inode each, contributing to the overall very high usage. According to the provider, we're very close to the limit (498727/500000) and a huge part of that is caused by the emails folder. Because there are over 400000 of them, we can't do anything to remove them as everything we've tried has resulted in a timeout or a server error.
The website is currently stable because we were able to reduce the amount of used inodes, but this is by no means a permanent solution and we will be waiting for a proper solution from the provider. -
01:46, Mar 08 CESTOn HoldThe site appears to be unstable again, we're still awaiting a response from the provider.
-
03:08, Mar 08 CESTOn HoldWe managed to get the website back to working condition temporarily. The provider did respond but we were unable to perform the action they told us on our end, so we're waiting for them to do it.
-
03:16, Mar 08 CESTMonitoringWe have applied a fix and are monitoring the results.
There was no need to contact the provider as we were able to do it in the end. -
15:53, Mar 08 CESTResolvedSite is stable and we do not expect this incident to happen again anytime soon. We have also cancelled the scheduled maintenance tasks as the issue has been fully resolved.