We no longer see delayed processing of messages. We're continuing to monitor.
Posted Mar 03, 2024 - 09:54 UTC
Update
We have rolled out the fix for most of the customers and the message processing lag is steadily going down. Some customers will continue to see delays in Last Seen, Online/Offline Status, and Command processing till the lag comes to 0. We've also scaled up the infrastructure to speed up the process.
Posted Mar 02, 2024 - 22:26 UTC
Update
We've identified the fix and working on deploying this for all customers. The problems was with one of our message processing systems which had slowed down and caused the lag.
Posted Mar 02, 2024 - 17:39 UTC
Identified
Our team has identified an issue causing degraded performance in message processing from devices. These messages include device status update, telemetry and command updates. We’re working to resolve it.
Which services are affected?
API: No Console: Yes (delayed last seen, incorrect online/offline status, delayed graphs and alerts). All other console operations are working as expected. Devices: No (device operations continue to work as expected)