tag:status.iterable.com,2005:/historyIterable Status - Incident History2024-03-28T11:40:27-07:00Iterabletag:status.iterable.com,2005:Incident/203513502024-03-25T08:10:06-07:002024-03-25T08:10:06-07:00Errors and slowness across the app for some customers<p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>08:10</var> PDT</small><br><strong>Resolved</strong> - We have fully recovered and customers should be seeing standard interactions and speed in the UI. If customers are seeing errors on templates or campaigns created during the impact window (6:15am PT to 7:30am PT), please recreate them as there may be some required information missing due to this issue.</p><p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>07:33</var> PDT</small><br><strong>Monitoring</strong> - We were seeing slowness across the app for some customers from 6:15am PT to 7:30am PT. These were just errors in the UI and there is no impact to sends or other core functionality. We have recovered and are actively monitoring.</p>tag:status.iterable.com,2005:Incident/203318352024-03-22T14:36:12-07:002024-03-22T14:36:12-07:00Ingestion on c11 stopped<p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>14:36</var> PDT</small><br><strong>Resolved</strong> - c11 has fully recovered and the queue has cleared.</p><p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>14:00</var> PDT</small><br><strong>Update</strong> - We have determined the impact was an increase in failed requests to user update API and only impacted a small subset of customers. No sends or triggering of campaigns were dropped, although some may have been delayed. Customers should try any failed requests again. We will continue monitoring the cluster health before closing out this incident.</p><p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>13:58</var> PDT</small><br><strong>Update</strong> - We have determined the impact was an increase in failed requests to user update API and only impacted a small subset of customers. No sends or triggering of campaigns were dropped, although some may have been delayed. Customers should try any failed requests again. We will continue monitoring the cluster health before closing out this incident.</p><p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>13:17</var> PDT</small><br><strong>Investigating</strong> - Starting at 12:15PT, C11 is experiencing performance issue. Customer may experience API requests fail such as triggering campaigns and user profile updates. Scheduled messages are still queued to send, but may be delayed. We are showing improvement. Please retry any failed API requests. Next update by 2PT.</p>tag:status.iterable.com,2005:Incident/199714832024-02-26T23:30:57-08:002024-02-26T23:30:58-08:00Notice of Upcoming Scheduled Maintenance<p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>23:30</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>22:00</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>13:40</var> PST</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>18:10</var> PST</small><br><strong>Scheduled</strong> - Hello,<br /><br />Following a customer wide email that was sent out on Monday, January 29th, we are writing to give you advance notice of a scheduled maintenance window, which is expected to last up to 90 minutes. Please find the maintenance schedule below, including the date and start time in different time zones:<br /><br />Pacific Standard Time (PST): Monday, February 26th, beginning at 10:00 PM<br />Eastern Standard Time (EST): Tuesday, February 27th, beginning at 1:00 AM<br />Greenwich Mean Time (GMT): Tuesday, February 27th, beginning at 6:00 AM<br /><br />During this maintenance window, you may experience:<br />Brief periods of 500 internal server errors in both the Iterable Application and the API.<br />Short periods in which message sends, journeys and webhooks are paused or delayed. Should this occur, they will resume and function as expected once the maintenance is complete.<br />We recommend that users do not initiate any new journeys or make changes to existing journeys during this window.<br />Links will be unaffected and will operate normally<br /><br />This scheduled maintenance is necessary to ensure the optimal performance and security of our systems with the latest AWS Postgres database upgrade, which is of utmost importance to continue to provide you with secure and reliable service. This maintenance window was selected in order to provide minimal impact to customer operations.<br /><br />If you have any questions about this scheduled maintenance, please reach out to your Customer Success Manager.</p>tag:status.iterable.com,2005:Incident/198812942024-02-01T02:28:31-08:002024-02-01T18:33:47-08:00SMS send failures<p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>02:28</var> PST</small><br><strong>Resolved</strong> - Engineering has completed resending some of the transactional SMS messages. Engineering is still looking into possibilities to resend the remaining transactional SMS messages. All failed marketing messages during this time will not be automatically resent. Please reach out to support if more assistance is needed.</p><p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>02:15</var> PST</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>02:13</var> PST</small><br><strong>Update</strong> - Engineering is continuing to resend messages and are monitoring the situation</p><p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>00:33</var> PST</small><br><strong>Update</strong> - Engineering is continuing to resend SMS messages. The next update will be at or before 2:00 AM PST.</p><p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>21:42</var> PST</small><br><strong>Update</strong> - Engineering is continuing to work on resending the failed SMS messages. We are also working to identify the affected SMS campaigns. The next update will be at or before 11:00 PM PST.</p><p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>19:36</var> PST</small><br><strong>Monitoring</strong> - We detected a high volume of SMS send failures via Twilio from 2:55 PM PST to 6 PM PST. The issue has been addressed, but messages sent during that time may have failed. Engineering is working to resend the failed messages. Next update will be at or before 8:30 PM PST</p>tag:status.iterable.com,2005:Incident/198431612024-01-26T22:39:31-08:002024-01-26T22:39:37-08:00API ingestion failure<p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>22:39</var> PST</small><br><strong>Resolved</strong> - The issue has been resolved by rotating the expired certificates, and the previously observed ingestion lag has subsided.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>22:12</var> PST</small><br><strong>Identified</strong> - The ingestion delay is still ongoing, and the team is in the process of updating the expired certificates.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>21:55</var> PST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>20:47</var> PST</small><br><strong>Investigating</strong> - Beginning at approximately 7:44 PM PST customers may have experienced API ingestion delay across multiple endpoints. As of approximately 8:41 pm PST here we have seen SSL handshake failure and team is investigating to resolve the issue. If you have any questions or believe you were impacted during this incident window please reach out to support@iterable.com or your customer success manager</p>tag:status.iterable.com,2005:Incident/198408762024-01-26T16:52:24-08:002024-01-26T16:52:29-08:00Elevated Api errors and Performance issues of Campaign page<p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>16:52</var> PST</small><br><strong>Resolved</strong> - Eng confirms the Iterable API is fully operational. For questions or concerns, please contact support@iterable.com or your CSM.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>14:11</var> PST</small><br><strong>Update</strong> - Beginning at approximately 11:30 AM PST customers may have experienced elevated API error rates across multiple endpoints. During this time any 5xx errors experienced would need to be retried. Due to the API failures customers may have seen some Journey trigger failures which will need to also be retried. During this period the Iterable webApp was also experiencing performance issues for customers trying to access different pages. As of approximately 2:05 pm PST here we have seen no further API errors and the webApp is fully operational. If you have any questions or believe you were impacted during this incident window please reach out to support@iterable.com or your customer success manager</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>12:45</var> PST</small><br><strong>Investigating</strong> - Iterable is experiencing performance errors on the web app as well various API endpoints. The team is currently conducting a more in-depth investigation into these issues. Next update at 1:45 pm PT or sooner.</p>tag:status.iterable.com,2005:Incident/197293412024-01-16T13:18:59-08:002024-01-16T13:18:59-08:00Webapp degraded performance<p><small>Jan <var data-var='date'>16</var>, <var data-var='time'>13:18</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>16</var>, <var data-var='time'>13:13</var> PST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jan <var data-var='date'>16</var>, <var data-var='time'>12:59</var> PST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jan <var data-var='date'>16</var>, <var data-var='time'>12:58</var> PST</small><br><strong>Investigating</strong> - We're currently investigating reports of Iterable web app degraded performance and log in failures starting around 12 pm PT. Next update at 1:30 pm PT or sooner.</p>tag:status.iterable.com,2005:Incident/196699582024-01-11T11:28:44-08:002024-01-11T11:28:45-08:00Degraded performance on C16<p><small>Jan <var data-var='date'>11</var>, <var data-var='time'>11:28</var> PST</small><br><strong>Resolved</strong> - Our engineering team has verified Cluster 16 has recovered. The issue is now resolved. Should you experience issues, please contact support at support@iterable.com.</p><p><small>Jan <var data-var='date'>11</var>, <var data-var='time'>09:34</var> PST</small><br><strong>Monitoring</strong> - The engineering team has been able to return Cluster 16 to a healthy state. All operations on the cluster should be fully recovered at this point. The team will continue to monitor. Next update at 11am PT or sooner</p><p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>19:07</var> PST</small><br><strong>Update</strong> - The engineering team was able to get C16 to a healthier state, and the cluster health continues to improve. Customers may still experience some occasional slowness in viewing lists and campaign insight metrics in the app and slower campaign and journey processing. The team will continue to investigate. Next update at 1/11 9:30am PT or sooner.</p><p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>18:01</var> PST</small><br><strong>Update</strong> - The engineering team is continuing to investigate the root cause of the C16 degradation and is looking for paths to remediation. Next update at 7pm PT or sooner.</p><p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>17:11</var> PST</small><br><strong>Update</strong> - The engineering team is continuing to investigate the root cause of the C16 degradation and is looking for paths to remediation. Next update at 6pm PT or sooner.</p><p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>14:58</var> PST</small><br><strong>Update</strong> - The engineering team is continuing to investigate the root cause of the C16 degradation and is looking for paths to remediation. Next update at 4pm PT or sooner.</p><p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>14:09</var> PST</small><br><strong>Update</strong> - The engineering team is continuing to investigate the root cause of the C16 degradation and is looking for paths to remediation. Next update at 3pm PT or sooner.</p><p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>13:00</var> PST</small><br><strong>Update</strong> - The engineering team is continuing to investigate the root cause of the C16 degradation and is looking for paths to remediation. Next update at 2pm PT or sooner.</p><p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>12:01</var> PST</small><br><strong>Update</strong> - The engineering team is continuing to investigate the root cause of the C16 degradation and is looking for paths to remediation. Next update at 1pm PT or sooner.</p><p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>11:02</var> PST</small><br><strong>Update</strong> - The engineering team is continuing to investigate the root cause of the C16 degradation and is looking for paths to remediation. Next update at 12pm PT or sooner.</p><p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>09:57</var> PST</small><br><strong>Investigating</strong> - The engineering team identified degraded performance on C16 starting January 9th, at around 11pm PT. We expect customers have experienced slowness in viewing lists and campaign insight metrics in the app and slower campaign and journey processing from that time. The engineering team is continuing to investigate the root cause of this degradation. Next update at 11am PT or sooner.</p>tag:status.iterable.com,2005:Incident/196488582024-01-08T10:58:57-08:002024-01-08T10:58:58-08:00Web App Intermittently Not Available<p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>10:58</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>10:21</var> PST</small><br><strong>Monitoring</strong> - Between 9:35 AM and 10:10 PST, some users may not have been able to access the Web App. The engineering team was able to fix the issue, and the login is operating as expected now. Sends and the API were not impacted by this issue.</p>tag:status.iterable.com,2005:Incident/196248322024-01-05T19:55:39-08:002024-01-05T19:55:39-08:00API Ingestions, Campaign Sends, and Proof Sends are experiencing degraded performance<p><small>Jan <var data-var='date'> 5</var>, <var data-var='time'>19:55</var> PST</small><br><strong>Resolved</strong> - All systems have returned to Normal Operations. If you have any questions, please don't hesitate to contact the Iterable Support Team.</p><p><small>Jan <var data-var='date'> 5</var>, <var data-var='time'>18:02</var> PST</small><br><strong>Monitoring</strong> - The affected components recovered, and we will continue to monitor and verify that the recovery has been sustained. The next update is at 8 PM PST.</p><p><small>Jan <var data-var='date'> 5</var>, <var data-var='time'>16:55</var> PST</small><br><strong>Update</strong> - The Engineering team is actively working on this incident and is seeing improvements. The next update is at 6 PM PT.</p><p><small>Jan <var data-var='date'> 5</var>, <var data-var='time'>15:05</var> PST</small><br><strong>Update</strong> - The engineering team is investigating possible contributors to the components not working as expected. The next update will be at 4 PM PST.</p><p><small>Jan <var data-var='date'> 5</var>, <var data-var='time'>14:19</var> PST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jan <var data-var='date'> 5</var>, <var data-var='time'>14:04</var> PST</small><br><strong>Investigating</strong> - API Ingestions, Campaign Sends, and Proof Sends are experiencing degraded performance. The engineering team is investigating. The next update will be at 2:30 PM PST.</p>tag:status.iterable.com,2005:Incident/195952052024-01-02T12:00:34-08:002024-01-02T12:00:35-08:00Investigating Web App Logins<p><small>Jan <var data-var='date'> 2</var>, <var data-var='time'>12:00</var> PST</small><br><strong>Resolved</strong> - The engineering team has resolved access issues to the Iterable web platform (web app). Sends and the API were not impacted during this incident. If you have any more questions, feel free to contact the Iterable support team.</p><p><small>Jan <var data-var='date'> 2</var>, <var data-var='time'>11:27</var> PST</small><br><strong>Update</strong> - Users are unable to access the Iterable web platform (web app). The engineering team is working to bring the web app back online. Engineering reports sends and the Iterable API is not impacted at this time. The next update will be at 12:30 PM PST.</p><p><small>Jan <var data-var='date'> 2</var>, <var data-var='time'>11:05</var> PST</small><br><strong>Investigating</strong> - We are currently investigating reports of login issues and slow performance that started to occur at 10:35 AM PST. Update to follow at 11:30 AM PST.</p>tag:status.iterable.com,2005:Incident/193706062023-12-07T16:32:12-08:002023-12-07T16:32:32-08:00Ingestion lag on C22<p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>16:32</var> PST</small><br><strong>Resolved</strong> - Our engineering team has verified c22 has recovered. The issue is now resolved. Should you experience issues, please contact support at support@iterable.com.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>15:30</var> PST</small><br><strong>Identified</strong> - Identified - Engineering has made some configuration changes and are currently monitoring the backlogs. We are seeing ingestion begin to recover and catch up. Next update at 4:30 PM PT or sooner.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>13:34</var> PST</small><br><strong>Investigating</strong> - At approximately 12:30PM PT Engineering has received alerts for ingestion lag on cluster C22. Engineering is currently investigating the ingestion lag increase. During this time customers may experience delays on user events and updates. Scheduled and triggered sends are not impacted. Next update at 3:30 PM PT or sooner.</p>tag:status.iterable.com,2005:Incident/192667742023-12-01T14:49:18-08:002023-12-01T14:49:18-08:00C21 Journey Delays<p><small>Dec <var data-var='date'> 1</var>, <var data-var='time'>14:49</var> PST</small><br><strong>Resolved</strong> - The original journey performance issues that triggered this incident have now been resolved. If you experience further journey performance issues, please contact our Support team.</p><p><small>Dec <var data-var='date'> 1</var>, <var data-var='time'>14:04</var> PST</small><br><strong>Identified</strong> - A problematic Journey was identified and suspended. Performance is recovering, and our team will continue to monitor. Next update at 3:00PM PST.</p><p><small>Dec <var data-var='date'> 1</var>, <var data-var='time'>13:20</var> PST</small><br><strong>Investigating</strong> - Customers on cluster 21 may be experiencing delays with journey performance. Our engineering team is working to identify the cause. Emails are still being processed. Next update at 2:00PM PST.</p>tag:status.iterable.com,2005:Incident/191959322023-11-22T09:29:20-08:002023-11-22T09:29:21-08:00Issues connecting to app.iterable.com<p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>09:29</var> PST</small><br><strong>Resolved</strong> - Issues connecting to app.iterable.com are now resolved. Please reach out to your Customer Success Manager (CSM) or support if you have any questions or need further assistance.</p><p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>08:07</var> PST</small><br><strong>Investigating</strong> - "Between 7:34 am - 7:43 am pst app.iterable.com had connection issues where customers were not able to access app.iterable.com. This is now working as expected. The engineering team is investigating the issue that caused this. Next update will be provided around 9 am pst."</p>tag:status.iterable.com,2005:Incident/189934562023-11-15T09:50:37-08:002023-11-15T09:50:38-08:00Stuck users across multiple journeys<p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>09:50</var> PST</small><br><strong>Resolved</strong> - All remaining stuck users have been auto-removed from journeys yesterday Tuesday, November 14 and this incident is now resolved. If you need any further assistance related to this incident, please reach out to support for assistance.</p><p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>09:47</var> PST</small><br><strong>Update</strong> - All remaining stuck users have been auto-removed from journeys yesterday Tuesday, November 14 and this incident is now resolved. If you need any further assistance related to this incident, please reach out to support for assistance.</p><p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>17:16</var> PST</small><br><strong>Update</strong> - Our engineering team will continue to process triggered journeys until tomorrow Friday November 10, 12PM (PST). If you need users to be cleared or proceeded, please continue to reach out to your Customer Success Manager (CSM) or support for assistance. Any stuck users remaining will be auto-removed from journeys on Tuesday, November 14.</p><p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>16:02</var> PST</small><br><strong>Update</strong> - Our engineering team has completed proceeding the majority of journeys that required action. We plan to continue working on the remaining affected journeys tomorrow. An update will be scheduled tomorrow 11/09. If you need users to be cleared in the meantime, please reach out to your Customer Success Manager (CSM) or support for assistance.</p><p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>17:09</var> PST</small><br><strong>Update</strong> - Our engineering team worked on proceeding non-scheduled journeys with event triggers today and have completed proceeding most of the journeys of that nature. The next update will be scheduled on 11/08. If you need users to be cleared in the meantime, please reach out to your Customer Success Manager (CSM) or support for assistance.</p><p><small>Nov <var data-var='date'> 6</var>, <var data-var='time'>19:30</var> PST</small><br><strong>Update</strong> - Our engineering team restored the transactional data from triggered journeys. The next step is to proceed with most of the stuck users in Non-Scheduled Journeys, starting tomorrow. The next update is scheduled for Tuesday, 11/07. If you continue to experience issues with stuck users on scheduled journeys, please reach out to your Customer Success Manager (CSM) or support for assistance in clearing or proceeding with your users.</p><p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>11:30</var> PDT</small><br><strong>Update</strong> - Our engineering team has undertaken the following actions to remediate stuck users. <br /><br />Stuck Users on Exit Tiles:<br />These users, who completed their journeys but remained stuck on the last tile, have been cleared from the Journey by engineering. <br /><br />Users Stuck in Scheduled Journeys:<br />A majority of stuck users have been cleared from scheduled journeys. If you still are experiencing stuck users, please contact your CSM or support to have your users cleared or proceeded. <br /><br />Users Stuck in Non-Scheduled Journeys (e.g., custom events, API-triggered, etc.):<br />Our engineering team is focused on restoring transactional data from triggered journeys. Next update will be provided on Monday, 11/06. If you need users to be cleared or proceeded in the meantime, please contact your CSM or the support team.</p><p><small>Nov <var data-var='date'> 2</var>, <var data-var='time'>12:40</var> PDT</small><br><strong>Update</strong> - To address stuck user issues, our engineering team has undertaken the following action items:<br /><br />Stuck Users on Exit Tiles:<br />These users, who have completed their journeys but remained stuck on the last tile, have been processed by our engineering team. This action will complete by end of day today 11/02/23<br /><br />Users Stuck in Scheduled Journeys:<br />Our engineering team will clear or proceed with stuck users at 3 PM (PST). Please contact your CSM if you wish to have users cleared or moved forward from their journeys by 2:30 PM (PST). Requests submitted after 2:30 PM (PST) will be addressed next week.<br /><br />Users Stuck in Non-Scheduled Journeys (e.g., custom events, API-triggered, etc.):<br />Our engineering team will resume efforts on Monday, 11/06, to assess the possibility of retaining transactional data from these journeys. If you need users to be cleared or to proceed in the meantime, please get in touch with your CSM or the support team.</p><p><small>Nov <var data-var='date'> 1</var>, <var data-var='time'>13:59</var> PDT</small><br><strong>Monitoring</strong> - Status: Users became stuck in some tiles from 3:42pm to 8:31pm PST on Oct 30th. The issue is now in monitoring status. Visitors after that impact window have been proceeding through journeys as expected. Many of the stuck visitors have completed the journey and falsely appear stuck. CS and Eng have been working through remediation steps. Stuck visitors in scheduled journeys are being resumed/proceeded. Falsely stuck visitors in custom event triggered journeys are also being corrected. <br /><br />Transactional data cannot be referenced within Journeys with the following trigger types. <br />- Journey was trigged by custom event or workflow API trigger.</p>tag:status.iterable.com,2005:Incident/189719482023-10-30T13:30:41-07:002023-10-30T13:30:41-07:00app.iterable.com currently unreachable<p><small>Oct <var data-var='date'>30</var>, <var data-var='time'>13:30</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>30</var>, <var data-var='time'>13:26</var> PDT</small><br><strong>Investigating</strong> - app.iterable.com is reachable again. Thank you for your patience.</p>tag:status.iterable.com,2005:Incident/189700142023-10-30T11:06:49-07:002023-10-30T11:06:52-07:00Iterable Shopify Traffic Down<p><small>Oct <var data-var='date'>30</var>, <var data-var='time'>11:06</var> PDT</small><br><strong>Resolved</strong> - As of 10:20 PT Iterable's Shopify traffic has resumed. Our Engineers will continue to monitor traffic for the integartion, but if you have any questions regarding this incident please reach out to Iterable Support.</p><p><small>Oct <var data-var='date'>30</var>, <var data-var='time'>10:12</var> PDT</small><br><strong>Identified</strong> - Starting approximately 8:13am PT our Shopify Integration Traffic has halted. Customers using the Shopify integration will notice Shopify data not passing into Iterable. Engineering has identified the cause and is working on resolving the issue. Next update at 12 pm PT or sooner.</p>tag:status.iterable.com,2005:Incident/187013052023-10-05T15:40:43-07:002023-10-05T15:40:43-07:00Slow journey performance and user API endpoints impacted on Cluster 6<p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>15:40</var> PDT</small><br><strong>Resolved</strong> - After a period of monitoring, the C6 cluster has reached a stable state, and this issue has been resolved</p><p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>13:08</var> PDT</small><br><strong>Monitoring</strong> - The team has successfully identified the root cause of the high CPU usage on the c6 cluster, which was attributed to the performance of certain queries. This issue has been addressed by excluding those queries from being used for insights followed by action items to optimize the queries for long-term resolution. As a result, the c6 cluster has regained its health and recovered from the high CPU usage. Monitoring of the cluster will continue for some time to ensure its stability</p><p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>12:27</var> PDT</small><br><strong>Update</strong> - The engineering team is actively investigating the root cause of the high CPU load on c6, which is leading to slow performance in Journey's and User data API calls. Our next update on the progress is at 1:30 PM PST. This slowness is for the customers on C6 cluster.</p><p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>11:24</var> PDT</small><br><strong>Investigating</strong> - Customers on cluster 6 may be experiencing delays with journey performance as well as 5XX errors to any API calls around user endpoints. Our engineering team is working to identify the cause. Emails are still being processed. The next update will be at 12:30PM PST.</p>tag:status.iterable.com,2005:Incident/185674552023-09-21T16:13:50-07:002023-09-21T16:13:50-07:00Ingestion stopped on C25<p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>16:13</var> PDT</small><br><strong>Resolved</strong> - Cluster 25 has recovered fully and is now processing normally.</p><p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>13:03</var> PDT</small><br><strong>Monitoring</strong> - Ingestion on cluster 25 is still recovering. We are monitoring the situation and expect it to catch up in the next several hours. Until then you may still experience overall slowness of journey processing and API calls. Next update will be at or before 4pm PDT.</p><p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>10:43</var> PDT</small><br><strong>Update</strong> - As of 9:09 AM PDT we have resumed ingestion on Cluster 25 and are working through the backlog of user events, updates, and triggers. No data has been lost. You may experience overall slowness of journey processing and API calls as we work through this backlog. Next update will be at or before 1PM PDT.</p><p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>09:08</var> PDT</small><br><strong>Update</strong> - Our engineers are continuing to remedy the ingestion issue on C25. Next update will be at 11AM PDT or before.</p><p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>08:21</var> PDT</small><br><strong>Identified</strong> - At 7:48 AM PDT we identified that ingestion has stopped on C25 and our engineers have identified the issue and are working to resolve it. Next update will be at 9AM PDT or sooner.</p>tag:status.iterable.com,2005:Incident/185579832023-09-20T15:09:08-07:002023-09-20T15:09:08-07:00Partial API Outage<p><small>Sep <var data-var='date'>20</var>, <var data-var='time'>15:09</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>20</var>, <var data-var='time'>12:35</var> PDT</small><br><strong>Monitoring</strong> - Engineering has made several changes to our infrastructure that has mitigated the issue. We are currently monitoring the situation and determining the root cause. Any API calls that returned a 5xx error code during the incident will need to be retried. The time lapse of intermittent API failures was from 8AM PDT to 11:30 AM PDT on 9/20/23.</p><p><small>Sep <var data-var='date'>20</var>, <var data-var='time'>11:36</var> PDT</small><br><strong>Update</strong> - Web and API have been affected by error spikes lasting roughly 1 minute each since 8AM PDT on 9/20/23. Our engineers are working to increase resources on our gateway to mitigate the issue at this time. We are still investigating the root cause. The next update will be at 1PM PDT.</p><p><small>Sep <var data-var='date'>20</var>, <var data-var='time'>10:37</var> PDT</small><br><strong>Update</strong> - We are currently investigating a large volume of API and Web errors. Next update will be at 11:15 AM PDT.</p><p><small>Sep <var data-var='date'>20</var>, <var data-var='time'>10:23</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating a large volume of API errors. Next update will be at 11:15 AM PDT.</p>tag:status.iterable.com,2005:Incident/184398432023-09-12T00:00:17-07:002023-09-12T00:00:17-07:00Monday, September 11 - Scheduled database maintenance<p><small>Sep <var data-var='date'>12</var>, <var data-var='time'>00:00</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep <var data-var='date'>11</var>, <var data-var='time'>22:30</var> PDT</small><br><strong>In progress</strong> - Engineering has completed majority of scheduled work of this maintenance and will continue additional scheduled maintenance during the our October 16, 2023 window. We will send out further communications closer to October 16th</p><p><small>Sep <var data-var='date'> 8</var>, <var data-var='time'>13:50</var> PDT</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Sep <var data-var='date'>11</var>, <var data-var='time'>22:30</var> PDT</small><br><strong>Scheduled</strong> - Monday, September 11th, starting at 10:30PM PDT for up to 90 minutes <br /><br />These scheduled maintenance windows are necessary to ensure the optimal performance and security of our systems, which is of utmost importance to continue to provide you with secure and reliable service. You can find more details below.<br /><br />September 11, 2023 Maintenance Window Between 10:30 PM PDT and 12:00 PDT<br /><br />During this maintenance window, users will be paused from entering into journeys. More details below:<br /><br /> * Any users currently in a journey will be unaffected and move through the journey as expected.<br /> * New users triggered into a journey during the maintenance window will be held in the start tile until the maintenance is complete, including transactional notifications sent through journeys. After completion, they will move through as expected.<br /> * Any users who would be exited from a journey due to Exit Rules will be queued up to exit, but they won’t exit the journey until completion of the maintenance.<br /> * All other services, including the send services for other campaigns, will remain operational without any disruptions.</p>tag:status.iterable.com,2005:Incident/183266522023-08-30T18:22:04-07:002023-08-30T18:22:04-07:00Users are having issues accessing and saving journeys and campaigns.<p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>18:22</var> PDT</small><br><strong>Resolved</strong> - The Engineering team resolved the underlying issues. If you are still experiencing issues, please contact the Iterable support team for further assistance.</p><p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>17:15</var> PDT</small><br><strong>Monitoring</strong> - The Engineering team implemented a fix, and we have recovered from the service degradation. We will continue monitoring the service for any additional degradation issues and provide another update at 6:00 PM PST.</p><p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>16:13</var> PDT</small><br><strong>Investigating</strong> - We detected that campaigns and templates cannot be accessed or saved. The outage is intermittent and the engineering team is working on restoring the service. The next update is at 5:00 PM PST.</p>tag:status.iterable.com,2005:Incident/183137482023-08-29T19:18:34-07:002023-08-30T11:37:50-07:00Unable to Save Templates<p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>19:18</var> PDT</small><br><strong>Resolved</strong> - There have been no new reports of issues with saving templates. This incident is now resolved. If you are still experiencing issues with saving any parts of your templates, please reach out to Support for additional instructions.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>17:27</var> PDT</small><br><strong>Monitoring</strong> - The Engineering team implemented a fix, which should allow customers to save templates once more. If you are still unable to save templates, please reach out to the Support team to receive additional instructions.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>16:57</var> PDT</small><br><strong>Investigating</strong> - We have received multiple reports that templates cannot be saved. The Engineering team is actively investigating the issue to restore this service. Next update at 5:30 PM PT</p>tag:status.iterable.com,2005:Incident/182835232023-08-27T13:10:21-07:002023-08-27T13:10:21-07:00Journey processing issues across clusters<p><small>Aug <var data-var='date'>27</var>, <var data-var='time'>13:10</var> PDT</small><br><strong>Resolved</strong> - The engineering team has confirmed that journey processing is healthy and our systems have caught up on their processing backlogs. For the most part, impact was between approximately 3:00am PT and 9:20am PT today. Clusters 6 and 9 took a little longer to recover, so impact for those was between approximately 3:00am PT and 12:00pm PT.</p><p><small>Aug <var data-var='date'>27</var>, <var data-var='time'>10:03</var> PDT</small><br><strong>Monitoring</strong> - The engineering team rolled out a remediation that has brought journey processing back to a normal level. The team will continue to monitor the situation. Impact was between approximately 3:00am PT and 9:20am PT today.</p><p><small>Aug <var data-var='date'>27</var>, <var data-var='time'>09:11</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Aug <var data-var='date'>27</var>, <var data-var='time'>09:10</var> PDT</small><br><strong>Investigating</strong> - The engineering team has identified an issue in our journey processing engine with implications across our customer base. The specific impact is delayed processing of journeys across multiple clusters. Impact began at approximately 3am PT today. No data has been lost. Next update at 10:00am PT.</p>tag:status.iterable.com,2005:Incident/182260522023-08-22T15:35:54-07:002023-08-22T15:35:54-07:00User updates, user deletions, list uploads, custom event, and journey processing delays<p><small>Aug <var data-var='date'>22</var>, <var data-var='time'>15:35</var> PDT</small><br><strong>Resolved</strong> - Our engineering team has confirmed processing across all clusters is healthy and our systems have caught up on their processing backlogs. Total impact was from approximately 1:45pm PT to approximately 3:20pm PT</p><p><small>Aug <var data-var='date'>22</var>, <var data-var='time'>15:02</var> PDT</small><br><strong>Update</strong> - Our engineering team has rolled out a fix. Processing across all clusters is returning to a healthy rate. The team will continue to monitor. Our engineering team estimates that our systems will be caught up on their processing backlogs in approximately 30 minutes. Next update at 3:30pm PT.</p><p><small>Aug <var data-var='date'>22</var>, <var data-var='time'>14:39</var> PDT</small><br><strong>Identified</strong> - Our engineering team has identified an issue that is causing delays on processing user updates, user deletions, list uploads, custom event, and user journeys. Impact started at approximately 1:45pm PT. No data is being dropped only delayed. This may also delay message sends associated with impacted journeys and API calls. The team has identified a fix and is actively working on rolling it. Next update at 3pm PT.</p>