We're seeing good performance over the weekend and into today's "morning rush".
Ticket Imports Imports are still disabled at this time but we will be looking to enable that very soon. If you've started an import, don't worry, it will be queued up and will process once the feature is enabled.
Posted Apr 12, 2021 - 08:49 CDT
Update
Monitoring will continue over the weekend. We now also have automatic cycling of pods for background jobs. If queues start to back up, this cycling will ensure it doesn’t reach a level that impacts the cloud help desk itself. It will also ensure email queues stay at manageable levels.
Posted Apr 09, 2021 - 14:43 CDT
Update
Outbound email flow is now stabilized and caught up.
Posted Apr 09, 2021 - 12:03 CDT
Update
There is currently a 20 minute delay on outgoing emails. Inbound email is unaffected. Dev are monitoring this to help identify the cause while the issue is occurring.
Posted Apr 09, 2021 - 11:21 CDT
Monitoring
The cloud help desk has been stabilized and mail is now flowing normally. Ticket imports are still disabled as they can contribute to the problems we've been seeing. Our dev and ops teams are fully focused on resolving these issues completely.
Unfortunately we are again seeing mail delivery issues related to the ongoing problems with the cloud help desk. While many changes have already been made to mitigate this, we're seeing the same end results (mail processing issues) for different reasons. Our dev and ops teams will continue to work to get this under control.
The performance issues have subsided and ticket creation (via email) is working as intended. We will continue to work to mitigate the symptoms as we continue to address the core issue.
We will keep the Status Page updated with progress but if you're looking for additional insights, we are monitoring and updating this topic in the Spiceworks Community as well:
There are continuing issues this morning with the queue of jobs backing up. This is impacting the speed of mail delivery into the system. Our dev and ops team are working to permanently fix this, sorry for the continued hassle.
Posted Apr 08, 2021 - 07:11 CDT
Update
We're continuing to monitor for further issues.
Posted Apr 07, 2021 - 15:00 CDT
Update
Seeing a few minutes delay on emails, the dev team is working quickly to resolve this.
Posted Apr 07, 2021 - 14:28 CDT
Monitoring
We're continuing to see good performance at this stage but will still continue to monitor the cloud help desk services closely.
Posted Apr 07, 2021 - 12:37 CDT
Update
We just took the Cloud Help Desk out of maintenance mode after we saw the background-jobs level out. This will alleviate the symptoms for a short while as we further investigate what is going on. We've also pulled in our Infrastructure development squad to help assist.
Posted Apr 07, 2021 - 09:18 CDT
Identified
The cloud help desk has been temporarily put into maintenance mode to allow us to fix the issue. You will not be able to access the cloud help desk at this time.
Posted Apr 07, 2021 - 08:47 CDT
Investigating
We've noticed further issues this morning with the queue of jobs starting to back up. This is impacting the speed of mail delivery into the system.
Posted Apr 07, 2021 - 07:01 CDT
Monitoring
We have resolved the issue and the backlog of jobs is now being processed. This includes email, notifications, reports and other tasks. Due to the large quantity of accumulated jobs it may take some time for this to complete.
Posted Apr 06, 2021 - 12:20 CDT
Identified
The issue has been identified and a fix is being implemented. You may also be seeing delays on reports or reports failing to run at this time.
Emails are being queued up and are not lost.
Posted Apr 06, 2021 - 09:52 CDT
Investigating
We're seeing multiple reports of emails not making it to the cloud help desk to create tickets. Our dev team has been advised of this and are looking into the issue.