Missing events for customers with 30 day retention
Incident Report for Lucky Orange
An ongoing issue with one of our vendors lead to misconfiguration in the database that stores events (sessions, pages, etc.) for customers with 30 day data retention. This caused our data collection system to incorrectly mark events destined for that database as ignorable and the events were silently ignored without being processed and stored. Because these events were silently ignored, no monitoring alarms were triggered as the system believed everything was working properly.

Therefore, for a period of time yesterday, no events are available for customers with 30 day retention. We sincerely apologize for the missing data and have since added alerts to catch this scenario if it were to happen again. Additionally, we are taking steps internally to alter our logic for which events should be ignored to ensure that no data is lost in the future.

Thank you for your support and understanding.
Posted Jun 08, 2022 - 15:30 CDT