Tracking frontend coverage of your e2e tests with Playwright

By Max Schmitt, Published on 6/22/2020

Introduction

End-to-End tests play a critical role in modern software development nowadays. As Guillermo Rauch (CEO of Vercel, formerly ZEIT) outlined in his last blog article about software development, software engineering teams attend to prioritize and implement End-to-End tests over normal unit tests. As an example, they start the real application setup and their dependencies to use it as a real user by browser emulation instead of testing e.g. React components and comparing input and output of them. Especially integrated into the Continuous Integration lifecycle with a CI provider like GitHub Actions ensures always, that the End-to-End tests are passing before they are merged and deployed to production.

With this mindset, its especially good for the team to have a way to track the actual coverage down to the code level, to see which functions and pages were executed with the End-to-End tests in our case with Playwright. Doing this manually is very time inefficient and error-prone. So we are focusing in this article, what tools and components you need, to automatically track the coverage and send it to your preferred coverage visualization provider like Codecov or Coveralls to have it visualized it like here on Coveralls.

Overview

There are multiple ways, to get the coverage data of your tests, you can use the Chrome specific coverage tracking feature, which gets the coverage data directly from the V8 runtime, but this is browser-specific and has no support for 1:many source-maps if e.g. your bundle is minified it wouldn't yet work, see here for more information.

In this tutorial, we are focusing on the usage with the Babel plugin babel-plugin-istanbul which you have to add during the build process (transcompilation) to your project. This will hold then the coverage data in a global variable called __coverage__ on the window object. Jest-Playwright, will automatically take care of storing and merging the coverage data. If you want to do it manually, you have to store it after each reload and page navigation because it's stored on the Window object which is not persistent.

Setting up Babel

The configuration with Babel depends on each application. In general its framework agnostic, so it can be used with e.g. React, Vue or Angular. In our example, we are using a React application with create-react-app which per default does not accept any additional Babel plugins, that's why we have to use a third-party tool called react-app-rewired to make it configurable. So in this case we have to create a file called config-overrides.js, to add the babel-plugin-istanbul plugin there. See here for the full file on GitHub and here for getting started with react-app-rewired.

const { override, addBabelPlugins } = require("customize-cra")

module.exports = override(
  process.env.E2E_TESTS && addBabelPlugins(
    "babel-plugin-istanbul",
  )
)

It's common to only add the babel plugin when you actually need the coverage information, for that we have there the test on the E2E_TESTS environment variable in place which we only set, when we bundle the web application for the end-to-end tests.x

Setting up jest-playwright

jest-playwright acts as a Jest environment which provides a Playwright Page instance for each test. After installing it with:

npm install -D jest jest-playwright-preset playwright

You can enable it in your Jest configuration jest.config.js:

module.exports = {
    preset: "jest-playwright-preset"
}

For a full reference for jest-playwright check out the official docs. Its also recommended using a separate Jest configuration for the e2e tests.

Collecting coverage with jest-playwright

jest-playwright will provide a method for saving the test coverage after each test and merging it after all the tests are completed. First you have to enable the coverage collection in the configuration in the jest-playwright.config.js file as follows:

// https://github.com/playwright-community/jest-playwright/#configuration
module.exports = {
    browsers: ["chromium", "firefox", "webkit"],
    serverOptions: {
        command: "npm start",
        port: 3000,
        launchTimeout: 10000,
        debug: true,
        options: {
          env: {
            E2E_TESTS: "true"
          }
        }
    },
    collectCoverage: true
}

In this example, we are also using the jest-process-manager to automatically start the React application before we are running the Jest test suite.

Once this is setup, jest-playwright, will start the development server and save the coverage data for all the pages which you are using inside the tests.

beforeEach(async () => {
  await page.goto("http://localhost:3000")
})

test("use Turquoise as a default background color", async () => {
  await expect(page).toHaveSelector("text=#1abc9c")
});

test("use Red as a background color", async () => {
  await page.click("text=Red")
  await expect(page).toHaveSelector("text=#e74c3c")
});

At that point, you can run your tests as usual with the jest command which is ideally configured in the script section of your package.json. When everything was successfully configured, you will see that the coverage data was successfully written to the file.

The output will be placed under the .nyc_output/coverage.json path, so it can be consumed using the Istanbul command line interface.

Visualize it using NYC

Once this is done, you can directly work with your coverage data by using the npx nyc report --reporter=html command which will create an HTML website in the coverage directory. You can then open the coverage/index.html file with your favorite browser to analyse it which will look in our example like as follows.

Uploading it to Coveralls/Codecov

It depends now on the coverage provider. Coveralls for example needs the lcov data format which is different from the format which we have generated (coverage.json). By using the nyc report --reporter=lcovonly command you can convert it to the lcov data format which will save the file under coverage/lcov.info. It's also useful to add it with the && syntax in the script section of your package.json to always generate the lcov data after running the test command.

Codecov on the other hand will accept the coverage.json file, so there is no need to convert it.

It also depends on the CI provider of your choice to upload it to the coverage provider. For GitHub, it's recommended to use the available official GitHub Actions:

• Coveralls GitHub Action
• Codecov GitHub Action

Summary

In this tutorial, we went through the setup which is required to configure Playwright and Jest for storing and merging the coverage data of your End-to-End tests. You'll find the full repository on GitHub as a reference on GitHub.