Measuring code coverage by Protractor end-to-end tests

Was just setting up new JavaScript project based on Grunt. I scaffolded the project template by Yeoman with usage of angular-fullstack generator. I decided to try MEAN stack without MongoDB for my new project (DB isn’t needed).

Next step was integrating Require.JS and configuring measurement of code coverage on client and server by Instanbul. When was this all done I was wondering if it is possible to measure code coverage by Protractor end-to-end testing.

After quick search I found that Ryan Bridges recently released grunt-protractor-coverage plugin. Interesting coincidence. So I decided to try it and can confirm that it’s working fine with mentioned stack. Configuration was smooth and Ryan fixed small issue very promptly. It’s based on grunt-protractor-runner plugin.

I created separate Grunt configuration file just for this purpose not to mess around with normal build. I had also problems to run ‘makeReport’ task of grunt-istanbul plugin for two different directories (Mocha server side code coverage measurement is using same task).

So here is the Grunt flow. First we need to copy non JavaScript files into target directory. It needs to be in separate directory because JavaScript files will need to be instrumented.

copy: {
  coverageE2E: {
    files: [{
      expand: true,
      dot: true,
      cwd: '<%= dirs.app %>',
      dest: '<%= dirs.instrumentedE2E %>/app',
      src: [
        '*.{ico,png,txt}',
        '.htaccess',
        'bower_components/**/*',
        'images/**/*',
        'fonts/**/*',
        'views/**/*',
        'styles/**/*',
      ]
    }]
  },
},

Next step is instrumentation of the code. It is needed for gathering coverage stats. Each line is decorated by special instructions that helps during measurement. Pay attention to fact that we are instrumenting server and client side code. Instrumented code is placed into target directory represented by placeholder <%= dirs.instrumentedE2E %>.

instrument: {
  files: ['server/**/*.js', 'app/scripts/**/*.js'],
  options: {
    lazy: true,
    basePath: '<%= dirs.instrumentedE2E %>/'
  }
},

Next we start Express from target directory.

express: {
  options: {
    port: process.env.PORT || 9000
  },
  coverageE2E: {
    options: {
      script: '<%= dirs.instrumentedE2E %>/lib/server.js',
      debug: true
    }
  },
},

And the protractor_coverage task of grunt-protractor-coverage plugin. Configuration should be the same as for grunt-protractor-runner plugin.

protractor_coverage: {
  options: {
    configFile: 'test/protractor/protractorConf.js', // Default config file
    keepAlive: true, // If false, the grunt process stops when the test fails.
    noColor: false, // If true, protractor will not use colors in its output.
    coverageDir: '<%= dirs.instrumentedE2E %>',
    args: {}
  },
  chrome: {
    options: {
      args: {
        baseUrl: 'https://localhost:3000/',
        // Arguments passed to the command
        'browser': 'chrome'
      }
    }
  },
},

Last step is generation  of coverage report.

makeReport: {
  src: '<%= dirs.instrumentedE2E %>/*.json',
  options: {
    type: 'html',
    dir: '<%= dirs.coverageE2E %>/reports',
    print: 'detail'
  }
},

Finally, this is grunt task gathering all steps.

grunt.registerTask('default', [
  'clean:coverageE2E',
  'copy:coverageE2E',
  'instrument',
  'express:coverageE2E',
  'protractor_coverage:chrome',
  'makeReport'
]);

EDIT: Notice that following Github link was changed to branch, because project structure was significantly changed:

Source code for this project can be found on GitHub.

For running end to end Protractor test you have to have webdriver-manager running. See  Protractor documentation.

To install and start Selenium Webdriver:

npm install -g protractor
webdriver-manager update
webdriver-manager start

To install project dependencies:

npm install
bower install

To run this end-to-end testing with coverage measurement (webdriver-manager has to be running):

grunt --gruntfile Gruntfile-e2e.js

6 thoughts on “Measuring code coverage by Protractor end-to-end tests

      1. my problem with makeReport task was, that grunt-istanbul can handle only one report per grunt file. That is why I separated istanbul code coverage measurements into separate files. If you aren’t using istanbul also for unit and e2e test coverage, you aren’t facing problems I mentioned. I would suggest to double check if you have grunt-istanbul installed via npm or if directories mantioned in makeReport task exist or are created by previous istanbul grunt tasks. It seem to me as such fundamental problem. Hope this helps.

  1. Hi,

    I am trying to set up protractor coverage for my web application, the problem that I am facing is that when I run the tests it doesn’t pick up the instrumented code, I’ve tried every possible way to fix it but no luck. I have also compared my configuration with some examples and couldn’t find the problem.

    1. Hi Parham,

      I can’t give you any advice without looking into your build. Modern JavaScript tooling is changing a lot. I already abandoned Grunt from my build.

      There is link to repository which was working with this configuration, but dependencies may be outdated. I also believe that there was problem instrumenting client side code. I recommend to pull out Github repository and play around with it.

Leave a Reply to Lubos Krnac Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.