Dealing With Synchronization Issues in Tests
Traditionally, one of the most difficult aspects of E2E testing is synchronizing the test scenario with the app.
Complex operations inside the app (like accessing servers or performing animations) often take a variable amount of time to complete; In each step, we can’t move on to the next one until they’ve completed (i.e. when the app goes idle), which in turn surfaces a challenge in continuously trying to understand when the right time to do so is.
Fortunately, Detox - which comes with a gray-box approach, cleverly performs the synchronization automatically, as explained here.
Mitigating Synchronization Issues
While Detox's auto-synchronization mechanism is powerful and efficient, it does come with at least one caveat: It imposes strictness over the app's behavior. By default, Detox will fail your tests (i.e. due to a wait-for-idle time-out), if, for example, following an app launch or a navigation to a new screen, timers or animations continue to run endlessly. While this could be considered an advantage (e.g. finding an animation or timer management leakage!), these type of issues may not:
- Be specifically related to the main coverage goal of your test.
- Be directly visible to or considered a significant bug by the end user.
Therefore, it may be something some would want to be able to limit or completely opt-out of. With this in mind, let's review the possible ways of mitigation, aimed at either finding and fixing a bug, or allowing for it to be overlooked.
Step 1: Understanding what's blocking your app
Detox's synchronization debugging mechanism generates output to Detox's log which provides useful synchronization debugging information. We recommend that you'd start by pinpointing what the busy resources are - those that are keeping your app from going idle, based on those logs. Turn the debugging mechanism toggle on if you need to (although, it is on by default), rerun your test(s) and follow Detox's logs.
Read about Detox's synchronization debugging mechanism in order to understand how to debug your app for unnecessarily busy resources. It can be specified either via the Detox configuration file or more directly, as a CLI argument:
detox test --debug-synchronization 5000
(Optional) Lower-level debugging (iOS only)
If the synchronization debugging mechanism does not provide all of the necessary information, on iOS you can add the following launch argument to your app (using launchArgs in your launchApp() call) to enable a very verbose logging of the idling resource system to the system log:
-DTXEnableVerboseSyncSystem YES -DTXEnableVerboseSyncResources YES
You can then obtain this log by running the following command:
xcrun simctl spawn booted log stream --level debug --style compact --predicate "category=='SyncManager'"
For example, change your device.launchApp() call like:
await device.launchApp({
newInstance: true,
launchArgs: { 'DTXEnableVerboseSyncSystem': 'YES', 'DTXEnableVerboseSyncResources': 'YES' },
});
Step 2: Applying the Most Suitable Solution
First and foremost, as explained, an app's inability to go idle might be an indication of that some resources are unnecessarily busy. Therefore, whether it's a network request that's been left unacknowledged, or an endless loader -
The best solution is to fix the problem! 👷♂️
Dealing with endless animation bugs (e.g. loaders)
Sometimes the resource that's holding the app back from turning idle is a mere animation of a single loader - running endlessly on the screen:
The synchronization logs telling you that, would usually look roughly like this:
09:04:20.170 detox[90417] i The app is busy with the following tasks:
• UI elements are busy:
- View animations pending: 2.
- Layers pending animations: 7.
- Layers needs layout: 147.
- View needs layout: 98.
- View needs display: 67.
- Layers needs display: 82.
• 1 enqueued native timers:
- Timer #1:
+ Fire date: none.
+ Time until fire: 0.000.
+ Repeat interval: 0.
+ Is recurring: YES.
Below are a few actual-bug scenarios where this can take place - all of which are easy to inspect, identify and fix.
i. A server is being non-responsive
Your app is waiting indefinitely to load all of the necessary data from the a non-responsive server / a bad network, and therefore cannot switch to rendering the expected UI. When this is the deal, additional synchronization logs also repeatedly show some in-flight network calls:
09:04:20.170 detox[90417] i The app is busy with the following tasks:
• 1 network requests with URLs:
- URL #1: https://nonresponsive-server.org/something?id=1337
- URL #2: https://unreachable-server.org/hello
This can be solved by finding out why the server is being non-responsive or unreachable, for example - by tracking network calls into the logs, or inspecting requests on the server-ends.
ii. A returned server error is not conveyed by the UI
Sometimes the servers do respond, but with an error. Often times, we fail to adjust our app code to display the necessary UI (e.g. replacing the loader with a friendly message). Rather, the loader just remains running indefinitely. Unlike in the previous case, you will not see synchronization-debug logs surfacing those unacknowledged network calls.
Scan through the app/device logs (not Detox log!) from the bottom up and find traces of errors. The app/device logs can be automatically recorded by Detox as a test-run artifact.
iii. An app error is ignored
Last but not least, this behavior can be the result of an app bug around handling valid data returned from the server, such as an error while parsing the data.
Look up the error - which usually comes with a stack-trace and is therefore very prominent, in the test summary in the (also Detox) logs. If it's not in the summary itself, you'd have to resort to searching those in the app/device logs. The app/device logs can be automatically recorded by Detox as a test-run artifact.
Dealing with looping animations
Animations are ultimately a way of making our apps friendlier and nicer to use. Sometimes a good user experience comes specifically from animations that are forever-looping on purpose (for example. animated .gifs).
Detox currently has no API's for "black listing" animations - namely, excluding them from the synchronization process (i.e. as it does with network calls). Therefore, you might want to consider removing the looping animation in your tests by rebuilding the app with such animating elements replaced by mocks.
Dealing with hidden animations
Not all synchronization issues around animations are trivial:
- The animation can be associated with an element that is rendered off-screen, such as an item in a long news feed that's been rendered beyond the screen's bound, or a loader in a screen associated with a bottom tab that hasn't been navigated-to since the beginning of the test.
- The animation can also be associated with elements which have been silently leaked (bug) under other UI elements. They are fully functional yet not visible to the user. For example: A compact loader accidentally showing under the app bar (android)/navigation bar (iOS).
These types of animations can be difficult to track down, and sometimes fix.
We are aiming to provide more advanced Detox-complementary tools that would help identify such animations more easily (follow Github issue #4734). Until we do, here are some things you can do, roughly, in order:
(Most recommended) Start Detox in debug mode, and then run the blocked test step-by-step - either from the beginning of it, or starting a breakpoint you've set over where you think the synchronization issue starts. Step over the Detox commands until Detox gets blocked and signals for synchronization issues (i.e. in the logs). Then, explore the app yourself (!), physically looking for animations (e.g. by scrolling lists, navigating through tabs, and so on. You might actually be able to see the animation in your own eyes, or witness Detox getting unblocked as your interactions release an otherwise endless loader. Do this iteratively until the exact animating UI element can be pinpointed.
Run the test and generate a view-hierarchy at the right timing, then inspect your app's layout in xcode's powerful view-hierarchy inspector. Look for off-screen or obscured elements.
Run your app (without Detox) on Android in debug mode using Android Studio, and use its fancy Layout Inspector tool in order to inspect your app.
Perform a "binary search" over the screen's content: Remove big blocks of content from it intermittently, rerunning the failing test each time, until synchronization stops blocking you. Then, slowly bring back repeatedly smaller and smaller sub-elements of the UI to see which one retriggers synchronization blockage. Eventually, you could pinpoint the exact culprit.
Dealing with network synchronization issues
Sometimes Detox synchronization gets block, waiting for networks call that are in fact associated with URL's that happen to be "noise", meaning - specifically related to some kind of long polling background tasks, or web-socket connections. Such network calls should NOT be considered blocking in a user-flow test.
This can be solved by black-listing the URL's in Detox, which would exclude it from Detox's synchronization over the network:
// Black-list (exclude) URL's in mid test-run:
await device.setURLBlacklist(['.*127.0.0.1.*']);
// Black-list URL's from as early as app launch:
await device.launchApp({
launchArgs: {
detoxURLBlacklistRegex: '(".*example.com/some-url/.*")',
},
});
// Reset black-list
await device.setURLBlacklist([]);
Dealing with endless timers
By default, Detox is designed to ignore JavaScript's setInterval() and will only wait for setTimeout()'s of up to 1.5 seconds.
If you have an endless polling loop with short intervals implemented with setTimeout, switch the implementation to setInterval. If possible, avoid aggressive polling in your app altogether, the poor single JavaScript thread we have doesn’t like it.
Last resort: Switching to manual synchronization
Disabling automatic synchronization makes sense in two cases:
You have a synchronization problem (as explained above) but you can't find the source of the problem, or simply decide not to fix it (temporarily...)
You have a screen with a transient element, such as a toast or a cheering UI element, that is displayed temporarily and disappears automatically:
Detox always has the fail-safe solution of turning off automatic-synchronization altogether and waiting manually for elements.
This isn’t the recommended approach as you'd be giving up Detox's synchronization super-powers and resort to manually defining timeouts, but hey, life is about trade-offs. You can do this with the main synchronization switching API's:
// Disabling in mid test-run:
await device.disableSynchronization();
// Launching the app with sync disabled from the start:
await device.launchApp({
launchArgs: {
detoxEnableSynchronization: 0
},
});
// To turn synchronization back on (this command will block on synchronization):
await device.enableSynchronization();
Mind that when this technique is applied, it effectively means that have to start using Detox API's differently (i.e. resort to using waitFor().withTimeout()) API's and possibly even migrate test code.
For example, instead of:
await element(by.id('write-message-btn')).tap(); // Navigate to message writing screen
await element(by.id('message-input')).typeText('Hello!'); // Type in a message
await element(by.id('submit-btn')).tap(); // Submit!
Your test code will have to look more like this:
await device.disableSynchronization();
await element(by.id('write-message-btn')).tap();
// Wait up to 4 seconds for the screen to fully load
await waitFor(element(by.id('message-input'))).toBeVisible().withTimeout(4000);
await element(by.id('message-input')).typeText('Hello!');
await element(by.id('submit-btn')).tap();
Hybrid manual/automatic synchronization
As of writing this, fine-tuned control over the various synchronized OS-systems that Detox monitors is not supported. Follow issue #1513 to keep track of that.