We've completed data validation for all affected customers and are satisfied that everything should be as expected. If you see any unusual sending behavior from 2/26-2/28 related to users in segmentations that use the signupDate, it could have been part of this incident. Please let us know if you have any questions: support@iterable.com
Posted Mar 04, 2019 - 13:25 PST
Monitoring
The restore of signup date and signup source values has been completed for all affected customers. The data is now being validated and we will post an additional update once this process has been completed.
Posted Mar 02, 2019 - 00:41 PST
Update
Data restore to our production environment has commenced and we have completed the process for 15 impacted customers thus far. Work will continue over the weekend, and we will update again once it is complete.
Posted Mar 01, 2019 - 18:07 PST
Update
Efforts to restore data are ongoing. The engineering team has completed numerous test runs and are preparing to begin restoration of data into the production environment. Some customers will take longer to restore than others. Next update by 6pm Pacific Time.
Posted Mar 01, 2019 - 14:14 PST
Update
The data ingestion lag is very close to being eliminated - delays are at less than five minutes.
The restore process for the overwritten fields is ongoing, and will take some time. Unlike a normal restore process, we can't just use a full backup - we need to pull specific data from the backup and apply it to the affected customer projects. It is critical that this is done correctly, so we are being very methodical.
We should have a better idea of when the restore will be complete later today, at which point we will update the incident again. For the time being, frequent updates will not be able to provide any additional info. If you have any questions, feel free to connect with us at support@iterable.com
Next update by 2pm Pacific Time.
Posted Mar 01, 2019 - 09:29 PST
Update
The engineering team worked through the night on restoring correct sign-up date and source data for the affected customers. These efforts are ongoing and progress is being made. As a result of the recovery process, we've had to temporarily switch to an alternate data ingestion path, which has had a knock-on effect of causing lag in data ingestion. The current lag time is about two hours, but we don't anticipate it deteriorating further.
In parallel with the data restoration efforts, we are working to move customers back over to the optimal ingestion path, hopefully within the next hour. This should then alleviate the lag.
Thank you for your patience. We'll provide another update by 9:30am Pacific Time.
Posted Mar 01, 2019 - 08:14 PST
Update
We are continuing to work and make progress on restoring any overwritten signupDate fields. It's worth noting that signupSource fields are also affected. We will provide the next update at 9:30 AM PDT
Posted Mar 01, 2019 - 01:52 PST
Update
Our engineers are continuing work on restoring any overwritten signupDate fields. It's worth noting that signupSource fields are also affected. Our current estimate is that the restore process will take about six hours, so rather than numerous updates during that process, we will provide the next update after the restore is complete.
Posted Feb 28, 2019 - 17:40 PST
Identified
We have identified a situation in which user profiles that have been updated at any point since Tuesday will have the SignupDate field overwritten with the update date instead. This may have had a knock-on effect for workflows that trigger off this field. This is impacting only a subset of customers, typically those with the largest data sets. The cause of the issue has been identified and fixed, and the engineering team is now working on restoring the correct data. Next update by 5:30pm Pacific Time.