Command processing is back to normal now and should no longer see delays. We're continuing to monitor.
Posted Apr 17, 2024 - 08:43 UTC
Update
Specific commands are continuing to process slowly. We're monitoring and anticipate the lag to subside in a few hours.
Posted Apr 17, 2024 - 06:03 UTC
Monitoring
We've resolved the issue by rolling out a fix and verified that new commands are being processed by the devices correctly. Commands that were queued during this incident will be processed in a few hours.
More fixes are being rollout out immediately to speed up message processing. We'll continue to monitor the services for next few hours before closing the incident.
Posted Apr 16, 2024 - 23:05 UTC
Identified
The issue has been identified and we're working on a fix. Commands continue to process slowly and stay in Queued state till the fix is deployed.
Posted Apr 16, 2024 - 20:34 UTC
Investigating
Our team has identified a possible cause of the partial service outage which impacts device provisioning and commands. We’re working to resolve it.
Which services are affected?
Device Provisioning: Yes. Limited set of devices. Commands: Yes. Commands are not reaching a limted set of devices.
API: All APIs are working correctly. No impact Console: Console is accessible to login and other operations. No impact. Customer Devices: Devices are working fine