Detox Copilot: Write Tests in Natural Language​

We're excited to announce Detox Copilot, a groundbreaking feature that brings natural language testing to Detox. With Detox Copilot, you can now write end-to-end tests using plain textual commands, making test creation more intuitive and accessible than ever.

Detox Copilot leverages advanced Large Language Models (LLMs) to interpret natural language instructions and translate them into Detox actions and assertions. This means you can describe your test scenarios in everyday language, aligning perfectly with Behavior-Driven Development (BDD) principles.

Why Natural Language Testing?​

• Improved Collaboration: Teams can collaborate more effectively, as tests are written in plain language understandable by developers, QA engineers, and non-technical stakeholders alike.
• Faster Test Creation: Reduce the time spent writing and maintaining complex test scripts.
• Enhanced Test Coverage: Lower the barrier to writing tests, encouraging more comprehensive testing.
• Reduced Maintenance Costs: Thanks to the decoupling from specific matchers (e.g., avoiding brittle XPath selectors or relying on testID attributes commonly used in React Native apps), tests are less prone to breaking when the UI changes, leading to lower maintenance overhead.

Key Features of Detox Copilot​

Write Tests in Plain Text​

Detox Copilot allows you to write tests using natural language commands. Each step corresponds to a specific action or assertion within your app.

it('should navigate and add a product to the cart', async () => {
  await copilot.perform(
    'Navigate to the "Products" page',
    'Tap on the "Add to Cart" button for the first product',
    'Verify that the "Added to Cart" pop-up is displayed'
  );
});

Seamless Integration with Detox​

Detox Copilot is built into Detox and requires no additional installation. Simply initialize it in your test setup, and you're ready to start writing natural language tests.

LLM-Agnostic Design​

Detox Copilot uses LLMs to interpret instructions but is designed to be LLM-agnostic. This means you can connect it to your preferred language model service, offering flexibility and future-proofing your testing strategy.

How Detox Copilot Works​

Once you've written your tests using natural language instructions, Detox Copilot takes care of the rest. Here is a high-level overview of the execution flow:

1. Gather Context: Collect relevant app state, view hierarchy, and previous step results.
2. Interpret Intent: Use the LLM to interpret the natural language instruction.
3. Generate Code: Create the appropriate Detox commands.
4. Execute Action: Run the generated Detox code.
5. Cache Results: Store execution results to optimize future runs.
6. Provide Feedback: Return values or confirm actions for subsequent steps.

By combining these steps, Detox Copilot effectively bridges the gap between natural language instructions and concrete test actions.

Getting Started with Detox Copilot​

Getting started with Detox Copilot is easy. Simply initialize Copilot in your test setup and start writing tests using natural language instructions.

Check our Testing with Copilot guide for detailed instructions on setting up and writing tests with Detox Copilot.

Extending Beyond Detox​

Detox Copilot is built on a standalone core library called detox-copilot designed to interpret natural language testing instructions and generate test code. Though initially developed for Detox, it can be extended to work with other testing frameworks.

Learn More​

For detailed guidance, check out our Testing with Copilot guide and the Detox Copilot API Documentation.

Join the Future of Testing​

Detox Copilot represents a major step forward in making end-to-end testing more accessible and efficient. By embracing natural language testing, you can enhance collaboration, speed up test creation, and improve overall test coverage.

We're excited to see how you'll leverage Detox Copilot in your tests! Share your experiences, feedback, and suggestions with us as we continue to refine and expand this groundbreaking feature.

Today we're proud to announce the new major release, Detox 20 (codename "Ashán"), which brings:

• official support for Genymotion SaaS
• improved integration with test runners
• configurable logging subsystem
• headless mode for iOS via configs and CLI
• reversing TCP ports via Android app configs
• and more optimizations to land in the next minor versions.

Genymotion SaaS​

Highlights: Using Genymotion SaaS.

Two years ago we added elementary support for cloud-based Android emulators provided by Genymotion SaaS and started a beta testing phase across mobile projects at Wix.

Previously our mobile infrastructure engineers had been maintaining Android virtual devices on CI build agents on their own, so switching to cloud devices cleared up their time for more productive tasks. Another improvement was particularly noticeable for teams with a vast number of tests. We could reduce the duration of their CI pipelines almost by half after they scaled up from 2 parallel devices to 6¹.

This positive impact encouraged us to adopt Genymotion SaaS for CI as quickly as possible, ignoring some unresolved issues in the initial pull request. For the most part, those were minor problems in global lifecycle management. Yet that made us feel uncertain about releasing it as-is, so we decided to take time and gain more production experience before taking any direction.

The further experience was surprisingly smooth and rarely presented issues, spare for a few minor glitches in advanced scenarios. Admittedly, revamping the Detox lifecycle took us longer than expected, which is all the more reason for us to celebrate today.

We're looking forward to providing our users with more opportunities for testing in the cloud, and this step is only the first of many to come. We hope you'll utilize this new feature to your delight.

Integration with test runners​

Highlights: Config file > Test runner, Internals API, Dropping Mocha support.

It took about a few months of work to formalize the contract between Detox and a test runner. While there's still a lot of place for improvement, the new Detox release refines their interaction and lays the groundwork for third-party integrations.

Mocha was our first supported test runner, but unfortunately, it could not keep up with our scaling requirements as the number of end-to-end tests grew. By the time it acquired the ability to run tests in parallel, we already had to place bets on another horse, and that was Jest.

We attempted to keep compatibility with both Jest and Mocha, but the farther we went, the more obvious it was that we couldn't have it both ways. As it turned out, Jest wasn't easy to get along with – our first integration with it was too simplistic. Over a couple of years of use in production, we kept discovering various issues that forced us to rewrite our "glue" code from scratch twice, and this isn't over yet. All combined didn't leave much time and energy for tinkering with Mocha anymore.

In this release, we discontinued Mocha support to focus on the attunement of Jest with the new runner-independent test runner config and Internals API. If there's enough demand, now it is up to the open-source community to build a new integration between Detox and Mocha.

Configurable logger​

Highlights: Config file > Logger, Logger API.

The rigidity of the logging subsystem has always been showing itself since its very creation in the summer of 2019. Due to time constraints and existing tech debts, it was impossible to do it right the first time, so we lived about three years with a proof-of-concept rather than a full-fledged feature.

The inconveniences weren't fatal but quite noticeable, nevertheless. Here are a few syndromes you could have spotted if you have ever used Detox timeline and log artifacts, especially when running tests in parallel:

• an uncanny file array: detox_pid_7505.log, detox_pid_7505.log.json, detox_pid_7506.log;
• a relatively shallow detox.trace.json: test suites, test functions, and some user-defined segments.

The good news is that the new Detox release condenses all those numerous logs into two files:

• the plain, human-readable detox.log;
• the raw, machine-readable detox.trace.json for chrome://trace, Perfetto and other utilities.

With the help of the new Logger API, you can add custom duration events to the timeline, too, e.g.:

await detox.log.trace.complete('Login', async () => {
  await element(by.id('email')).typeText('john@example.com');
  await element(by.id('password')).typeText('123456');

  detox.log.info('Trying to log in...');
  await element(by.id('submit')).tap();
});

Besides, it is possible now to customize the console output of Detox via the new logger config, e.g.:

/** @type {Detox.DetoxConfig} */
module.exports = {
  // ...
  logger: {
    options: {
      showDate: false,
      showLoggerName: false,
      showPid: false,
      prefixers: {
        ph: null,
      },
    },
  },
};

In the example above, we minimize all the metadata around the log messages – see the screenshot below:

Minor features​

Headless iOS​

One of Detox known issues was always booting iOS simulators in a hidden mode. You could see tests running on your local simulator only if you had manually opened the Simulator app beforehand. So, we unified the headless property for both iOS and Android, and now both the platforms visibly boot a device unless you configure it otherwise, e.g.:

/* @type {Detox.DetoxConfig} */
module.exports = {
  devices: {
    iphone: {
      type: 'ios.simulator',
      headless: process.env.CI ? true : undefined,
      device: {
        type: 'iPhone 14'
      },
      /* ... */
    }
  },
};

or, via CLI:

detox test -c ios.sim.release --headless

Reverse ports​

Your apps might try to access some localhost:* addresses (e.g., mock servers), but this is a bit more problematic in the case of Android. The Android emulators are separate virtual devices with their own loopback network interface. In such cases, you must set up reverse port forwarding via adb reverse.

Local servers are quite a common prerequisite for apps in debug mode – one could recall React Native bundler on port 8081, Storybook server on 9009, etc. That's why we decided to add an optional config property for Android apps, reversePorts:

/** @type {Detox.DetoxConfig} */
module.exports = {
  // ...
  apps: {
    'android.debug': {
      type: 'android.apk',
      binaryPath: '...',
      reversePorts: [8081, 3000],
    },
  },
};

In other words, this is a convenience API that tells Detox to run device.reverseTcpPort(portNumber) after installing the app. It should be helpful for anyone who prefers to keep such things as configs rather than as code.

Read-only emulators by default​

The -read-only flag appeared in Android emulator 28.0.16. Detox promptly adopted it since the read-only mode allowed it to run multiple instances of a single Android virtual device (AVD) concurrently. This feature helped us to implement parallel test execution support for Android back then.

Being overcautious, we implemented that partially, only for cases when the user starts multiple concurrent workers. This decision created a moderately annoying UX issue. Imagine you run tests sequentially first, using one worker only. That provides you with a regular AVD instance, i.e., not a read-only one. After that, you switch to multiple workers only to get an error from the Android emulator, complaining about mixing regular and read-only instances.

While the fix itself has always been straightforward – close the running AVD and try again – this entire overcaution brought more issues than solving them. That's why, from now on, Android emulators will always be starting in -read-only mode unless you configure readonly: false in your device config.

Reset lock file​

This release adds a small CLI tool, detox reset-lock-file, to help users with one specific use scenario.

Imagine you want to run tests for multiple Detox configurations at once, e.g.:

detox test -c iphoneSE2020.release e2e/ui.test.js
detox test -c iphone14ProMax.release e2e/ui.test.js

The problem is that Detox uses a file-locking mechanism to avoid situations when parallel test workers would take control of the same device. The detox test command, upon start, erases that file contents, creating a race condition risk.

To eliminate that risk, use a combination of detox reset-lock-file and --keepLockFile like this:

detox reset-lock-file & \
detox test --keepLockFile -c iphoneSE2020.release e2e/ui.test.js & \
detox test --keepLockFile -c iphone14ProMax.release e2e/ui.test.js & \
wait

In the future, we plan to minimize using lock files so that you don't have to think about this low-level implementation detail. So, this tool adds some convenience until we provide a next-gen solution.

Deprecations​

Detox 20 executes many pending deprecations, so make sure to check out our Migration Guide before upgrading:

• JS: minimum supported Node.js version is 14.x;
• JS: minimum supported Jest version is 27.2.5;
• JS: Mocha test runner is no longer supported;
• JS: discontinued old adapters for Jest (jest-jasmine, first generation of jest-circus adapter);
• JS: discontinued { permanent: true } option in device.appLaunchArgs.* methods (#3360);
• CLI: dropped -w, --workers and -o, --runner-config args – see a dedicated section in the migration guide;
• CLI: dropped deprecated --device-launch-args (#3665);
• Config: discontinued kebab-case properties: test-runner, runner-config (#3371)
• Config: discontinued skipLegacyWorkersInjections property ((#3286))
• Config: deprecated specs and runnerConfig properties
• Config: changed semantics of testRunner property
• Config: dropped support for all-in-one configurations (#3386);
• Android: remove deprecated native IdlePolicyConfig (#3332)
• iOS: discontinued ios.none device type – see the new way to debug native code (#3361)

Afterword​

Over the last year and a half, we have established a centralized configuration system for more than 50 projects using Detox at Wix. While it never seemed to be a cakewalk, the entire experience of troubleshooting over a hundred issues across the organization did not leave us unchanged.

We see numerous things to improve in Detox, but most of them boil down to the same thing – scaling. Surprisingly, "scaling" makes an excellent umbrella term for nearly every challenge we've been encountering lately:

• scaling up the number of users requires us to improve the onboarding and troubleshooting experience;
• scaling up the number of projects forces us to centralize scattered configs into flexible organization presets;
• scaling up the number of tests prompts us to optimize the codebase and incline it towards cloud and remote execution.

Our core team has been facing challenges of limited human resource constraints and growing scaling needs for a long time. In many ways, that has shaped a specific mindset within the team. We evaluate every discussed feature by asking a simple question: will it save other people and us time to focus on more important things? Teaching a man to fish is better than giving fish, so our success at preventing support issues matters more than our success at solving them ourselves.

That's why we'll be making subsequent efforts in these three areas, hoping to get back to you soon with even more exciting updates.

Enjoy your drive with Detox 20!

Cheers! 👋