This issue has been resolved and all our systems are healthy.
We thank you for your patience while our team addressed this.
Posted Mar 27, 2024 - 06:38 PDT
Monitoring
Our team detected the root cause for this issue in a high amount of variants for cached resources being created in certain scenarios. A fix has been implemented to ensure a more efficient caching strategy by reducing unwanted revalidation and preventing excessive variants for objects in entries critical for SDK initialization.
All indicators are currently healthy and validate that the fix has effectively mitigated the issue. We continue to monitor the results.
Posted Mar 25, 2024 - 13:57 PDT
Update
We detected a spike in 503 responses from our SDK API today between 15:04 and 15:06 GMT.
Our team continues to work on a fix for this issue.
Posted Mar 25, 2024 - 11:22 PDT
Identified
We are investigating some usage patterns affecting our CDN and sporadically causing slightly elevated error rates during short periods of time in certain regions. This can lead to initialization timeouts for SDKs initializing from an empty cache. Running SDK instances are unaffected, and synchronization will continue to work without issues. All of our SDKs have built-in retry mechanisms which means they will automatically recover once the event is resolved, typically within minutes of the onset of the issue.
Client side SDKs with a populated cache from a previous session are unaffected. Our mobile SDKs have persistent caching always enabled, and for our browser-based SDKs this can be enabled by setting the storage type to 'localstorage' through the SDK factory configuration (see: https://help.split.io/hc/en-us/articles/360020448791-JavaScript-SDK#configuration).
In order to minimize any potential impact to our customers' apps, we highly recommend following our best practices by handling SDK initialization timeouts and control treatments. More information on best practices to build a resilient integration with Split can be found here: https://help.split.io/hc/en-us/articles/25255992258701-How-to-build-a-resilient-integration.
Our team is currently working on a fix for to remediate this issue as a top priority, and it is expected to be deployed shortly.
We at Split sincerely apologize for the impact caused to any customers who were affected by this issue, and thank you for your patience this week as our engineering team investigated and addressed it.