Around 07/13/2023 17:41:00 UTC, Users of Zuper hosted in US-WEST-1C region started receiving 401 HTTP errors while logging in and accessing pages in Web & Mobile app.

We investigated the issue and traced it to memory issues on our in-memory data store responsible for handling the session tokens. This was also flagged by our internal alerting systems immediately.

Our team identified the root cause and started working on the event by updating the eviction policy of the in-memory datastore and purging the cache. All services were fully restored by 07/13/2023 18:06:00 UTC.

We have also set in place further alerting mechanism to proactive identify & address such future incidents.