Category Archives: JavaScript

Select Video.js subtitle track automatically

We at Dotsub are using videojs as video player for our sites. One of the Video.js main benefits is customizability via its plugin system. Recently we had a need to automatically select certain subtitle track after user started video on Video.js player. This is handy when we know the language user is most probably going to need translation into. So we created simple open-source plugin to save few user clicks needed for selecting default subtitle track.

The plugin was named videojs-select-subtitle and is hosted on Github. Assuming you know how to use Video.js plugin for you video player, we jump straight to explaining how to configure this new plugin.

Installation

Most modern JavaScript projects are using some kind of Node.JS based build process with NPM dependency management. So easiest way to install plugin is to use NPM:

npm i --save videojs-select-subtitle

Your other option is to clone Github project build the project with command:

npm run build

This command creates JavaScript files in sub-directory dist. You can include minified or non-minified version of JavaScript plugin file into your project with whatever mechanism you are used to.

Usage

After after videojs player is initialized and plugin is installed in our project, we can execute it with command:

player.selectSubtitle({ trackLanguage: 'es' });

Object passed as parameter into selectSubtitle function is option required by plugin to select correct subtitle/caption track. Options object has to have trackLanguage key and value needs to define language attribute of caption/subtitle track to be selected. Of course such track must be available in videojs player, otherwise plugin can’t select it at all. Both key and value of options object are case sensitive.

Usage on Brightcove Video Cloud

Video.js project was created by Brightcove. Therefore they made it very easy to embed Video.JS plugins into their Video Cloud service. If you are using this service as your online video platform, you can install videojs-subtitle-plugin via their UI.

In order to do this, you need to host built plugin file somewhere on the internet. After plugin is accessible by Brightcove Video Cloud, you can refer to it from your Brightcove player. It is important to configure name of the plugin to selectSubtitle and trackLanguage option for the plugin:

Screenshot_2016-09-02_14-18-59

After the user starts video, plugin finds desired caption/subtitle track and show it automatically.

Promises vs Callbacks - Code comparison

Promises vs Callbacks – Code comparison

I am not going to highlight pros of promises and cons of callbacks. There is plenty of good reading about this topic out there. I was recently writing simple Node module and decided to learn promises during its implementation. The result  can be interesting as comparison of Promises vs Callbacks approach applied to the same problems, because project contains

These are glued with Gulp build system to execute tests with all possible combinations. So for example Callbacks based test is executed also against Promises based module.

I named the project Jasstor. I am planning to use it for storage credentials into JSON file, hashing and verification of credentials. It stores user name, hashed password and user’s role in string format. Here is Github branch dedicated to this blog post, so that it’ll stay consistent. Please bear in mind, that I am learning Node development, ES6 features, Promises and Gulp on this project. So I could easily miss handy tricks or misused some features.

Project uses these main technologies:

I decided to have these constraints for Promises

  • Mocha will be excluded from promisification, so describe and it will be used with callbacks.
  • Jasstor‘s API will follow standard Node JS patterns
    • All functions are asynchronous with callback as last parameter
    • First parameter of callback is always error

When function signatures follow Node JS patters, it allows for promisification of modules. Such modules can be integrated into promise chain easily. But at the same time API isn’t tied to Promises at all. I like this approach because both camps (Promises or Callbacks fans) are happy.

Let’s take a look at code. I will explain and compare only most verbose use case and leave the rest for curios readers. You can find the code here.

Callback vs Promises – Tests comparison

Enough talking, let’s take a look at code. I’ll start with tests explanation as it promotes TDD thinking. Use case should test if existing password is overwritten when credentials for same user are stored. There are these phases in the test:

  1. Credentials file with initial password is created
  2. Read initial password
  3. Overwrite initial password with different one
  4. Read new password
  5. Verify that new password is different to initial one

Callbacks based test

var credentialsFile = 'testCredentials.txt';

var checkError = function(err, done) {
  if (err) {
    done(err);
  }
};

var readPassword = (credentialsFile, done, callback) => {
  fs.readFile(credentialsFile, (err, data) => {
    checkError(err, done);
    var jsonData = JSON.parse(data);
    callback(jsonData.user.password);
  });
};
describe('jasstor', () => {
  var jasstor = new Jasstor(credentialsFile);

  describe('when creadentials file already exist', () => {
    beforeEach(done => {
      fs.unlink(credentialsFile, () => {
        jasstor.saveCredentials('user', 'password', 'role', done);
      });
    });

    it('should overwrite existing password', done => {
      readPassword(credentialsFile, done, (originalPassword) => {
        jasstor.saveCredentials('user', 'password1', 'role', err => {
          checkError(err, done);
          readPassword(credentialsFile, done, (newPassword) => {
            newPassword.should.be.ok;
            originalPassword.should.be.ok;
            newPassword.should.not.equal(originalPassword);
            done();
          });
        });
      });
    });
  });
});

jasstor is testing object and jasstor.saveCredentials() is testing function. There is created helper function readPassword because password needs to be read twice during test. Pretty straight forward callbacks pyramid. I don’t like calling checkError at the beginning of each callback. Annoying Node pattern.

Promises based test

var fs = Bluebird.promisifyAll(require('fs'));
var credentialsFile = 'testCredentials.txt';

var readPassword = (credentialsFile, userName, done) => {
  return fs.readFileAsync(credentialsFile)
    .then(JSON.parse)
    .then(jsonData => {
      return jsonData[userName].password;
    }).catch(done);
};

var ignoreCallback = () => {};

describe('jasstor tested with promises', () => {
  var jasstor = Bluebird.promisifyAll(new Jasstor(credentialsFile));

  describe('when creadentials file already exist', () => {
    beforeEach(done => {
      fs.unlinkAsync(credentialsFile)
        .finally(() => {
          jasstor.saveCredentials('user', 'password', 'role', done);
        }).catch(ignoreCallback);
    });

    it('should overwrite existing password', done => {
      var originalPassword = readPassword(credentialsFile, 'user', done);
      var newPassword;
      jasstor.saveCredentialsAsync('user', 'password1', 'role')
        .then(() => {
          newPassword = readPassword(credentialsFile, 'user', done);
          newPassword.should.be.ok;
          originalPassword.should.be.ok;
          newPassword.should.not.equal(originalPassword);
          done();
        }).catch(done);
    });
  });
});

Important here is promisification of fs library (first line). It patches fs module to have additional methods with Async suffix. These methods return Promise and doesn’t take callback as parameter. This effectively converts existing API to promise based API. Same is done to testing object jasstor.

It was slight surprise to me that Promises actually doesn’t enable for less verbose code. Few facts are pretty obvious to me after this comparison:

  • Much more elegant error handling. As long as error callback has error as first parameter, you can just pass it to catch block as function reference.
  • Callbacks pyramid is flattened. This can improve readability. But readability is probably matter of maturity with certain approach.

Callback vs Promises – Node module comparison

Now I am going to compare code that was tested by tests above.

Callbacks based code

var hashPassword = (password, callback) => {
  bcrypt.genSalt(10, (err, salt) => bcrypt.hash(password, salt, callback));
};

var readJsonFile = (storageFile, callback) => {
  fs.exists(storageFile, (result) => {
    if (result === false) {
      callback(null, {});
    } else {
      fs.readFile(storageFile, (err, data) => {
        var jsonData = JSON.parse(data);
        callback(err, jsonData);
      });
    }
  });
};

module.exports = class Jasstor {
  constructor(storageFile) {
    this.storageFile = storageFile;
  }

  saveCredentials(user, password, role, callback) {
    readJsonFile(this.storageFile, (err, jsonData) => {
      hashPassword(password, (err, hash) => {
        jsonData[user] = {
          password: hash,
          role: role
        };
        var jsonDataString = JSON.stringify(jsonData);

        fs.writeFile(this.storageFile, jsonDataString, callback);
      });
    });
  }
};

Here we have ES6 class Jasstor with constructor and method that saves credentials into JSON file. There are two helper methods hashPassword and readJsonFile to help with repetitive tasks across the Jasstor class. We can see callback pyramid again. It is slightly simplified by helper functions.

Promises based code

var fs = Bluebird.promisifyAll(require('fs'));
var bcrypt = Bluebird.promisifyAll(require('bcrypt'));

var hashPassword = password => {
  return new Promise(resolve => {
    bcrypt.genSaltAsync(10)
      .then(salt => {
        return bcrypt.hashAsync(password, salt);
      }).then(resolve);
  });
};

var readJsonFile = storageFile => {
  return new Promise((resolve, reject) => {
    fs.exists(storageFile, result => {
      if (result === true) {
        fs.readFileAsync(storageFile)
          .then(JSON.parse)
          .then(resolve)
          .catch(reject);
      } else {
        resolve({});
      }
    });
  });
};

module.exports = class Jasstor {
  constructor(storageFile) {
    this.storageFile = storageFile;
  }

  saveCredentials(user, password, role, callback) {
    readJsonFile(this.storageFile).then(jsonData => {
      hashPassword(password).then(hash => {
        jsonData[user] = {
          password: hash,
          role: role
        };
        return jsonData;
      }).then(JSON.stringify)
        .then(jsonDataString => {
          fs.writeFile(this.storageFile, jsonDataString, callback);
        }).catch(callback);
    }).catch(callback);
  }
};

Same implementation packed into Promises is more verbose (hopefully I missed some tricks that could made it shorter). I like again simplified error handling. You maybe spot  that fs.exists isn’t promisified. If you take a look at its API, callback doesn’t have error as first parameter. I suspect, this is why fs.existsAsync doesn’t work correctly. Not sure if this is limitation of Bluebird promises library I am using or Promises A+ specification.

Conclusion

Promises are very nice approach that could totally change style of your programming. But I have to admit that I am not 100% sold to it yet. It took me some time to wrap my head around the concept. Promises also seem to me slightly more verbose than callbacks. When you have functions with one parameter and return value, you can nicely chain them with just passing function references into Promise chain. But mostly you don’t have such comfortable APIs and you end up doing “flattened callbacks pyramid”.

I would suggest to try Promises on your project (or small library) and make own opinion. Examples aren’t enough challenging to push the Promises into its limits.

Watch file changes and propagate errors with Gulp

Gulp fever infected me. Streaming model is very interesting and modern. After initial excitement, I started to experience first pitfalls. This is understandable for such young project. I am going to describe my problem with watching file changes and propagating errors.

Error in Gulp by default breaks the pipe, terminates the build/test and whole Gulp process with some error code. This is fine for CI process. But breaking the pipe stops file watch task also. This is big problem when developer wants to watch file changes and re-run particular tasks (e.g. tests).  You have to start watch task again after error occurs. This makes default watch task in Gulp pretty much useless. It is known and not the only problem of Gulp file watcher.

Fortunately Gulp 4 version if going to fix this. But I needed to come up with solution now. Google search points you to gulp-plumber. Idea behind it is to use gulp-plumber at the beginning of each pipe.

var plumber = require('gulp-plumber');
var coffee = require('gulp-coffee');

gulp.src('./src/*.ext')
    .pipe(plumber())
    .pipe(coffee())
    .pipe(gulp.dest('./dist'));

It prevents unpiping on error and forces the build process to continue regardless of error. Nice. Looks like problem with watch task solved.

I applied this approach on my pet project. It is simple node module that should store encrypted passwords into JSON file. But domain is not important for this blog post. When I checked in build process with gulp-plumber, I started to get false positives by drone.io CI server [EDIT: Build link doesn’t exist anymore].  Drone.io is using process error propagation, where each process returns error code. Non-zero value indicates error and zero means that process finished without error. gulp-plumber forces gulp process to continue and just writes errors to the console. Result is always zero error code from Gulp process.

So my goal is to use gulp-plumber to be able to continuously watch file changes and have fast feedback loop but also force Gulp process exit with non zero result when some error occurs.

First I declared variable to gather if error occurred.

var errorOccured = false;

Created handler for error recording.

var errorHandler = function () {
  console.log('Error occured... ');
  errorOccured = true;
};

Use gulp-plumber together with error handler for each Gulp pipe.

var transpilePipe = lazypipe()
  .pipe(plumber, {
    errorHandler: errorHandler
  })
  .pipe(jshint)
  .pipe(jshint.reporter, stylish)
  .pipe(jshint.reporter, 'fail')
  .pipe(traceur);

//Compiles ES6 into ES5
gulp.task('build', function () {
  return gulp.src(paths.scripts)
    .pipe(plumber({
      errorHandler: errorHandler
    }))
    .pipe(transpilePipe())
    .pipe(gulp.dest('dist'));
});

//Transpile to ES5 and runs mocha test
gulp.task('test', ['build'], function (cb) {
  gulp.src([paths.dist])
    .pipe(plumber({
      errorHandler: errorHandler
    }))
    .pipe(istanbul())
    .on('finish', function () {
      gulp.src(paths.tests)
        .pipe(plumber({
          errorHandler: errorHandler
        }))
        .pipe(transpilePipe())
        .pipe(gulp.dest('tmp'))
        .pipe(mocha())
        .pipe(istanbul.writeReports())
        .on('end', cb);
    });
});

This replaces gulp-plumber default error handler. It allows to record any error. (Example uses Lazypipe module. It can declare reusable pipe chunks. Lazypipe isn’t integrated with gulp-plumber, so it is needed also in sub-pipe.)

Next we need error checking Gulp task. It exits process with non-zero error code to indicate error state to Gulp process environment.

gulp.task('checkError', ['test'], function () {
  if (errorOccured) {
    console.log('Error occured, exitting build process... ');
    process.exit(1);
  }
});

Finally we call error checking task at the end of main Gulp task (right before submitting test coverage to coveralls.io in this case).

gulp.task('default', ['test', 'checkError', 'coveralls']);

Watch task is pretty standard, but doesn’t stop on error now.

gulp.task('watch', function () {
  var filesToWatch = paths.tests.concat(paths.scripts);
  gulp.watch(filesToWatch, ['test']);
});

And that’s it. Drone.io CI server properly highlights errors. I can also continuously watch file changes and automatically re-run tests. I agree that solution is little bit verbose, but I can live with that until Gulp 4 will be out.

Source code for this blog post can be found on Github.

Multi module JavaScript project

Multi module JavaScript project with Grunt

Writing blog post how I managed to configure multi module JavaScript project with Grunt for my spare time project. It is using Protractor for end-to-end testing, but I believe that this multi module approach would be easily portable onto non-Angular stack.

In my spare time I work on pet project based on EAN stack (Express, Angular, Node.JS). (Project doesn’t need DB, that’s why MongoDB is missing from famous MEAN stack). Initial draft of the project was scaffolded by Yeoman with usage of angular-fullstack generator. Build is based on Grunt. Apart from that generator was using Grunt, I chose it over Gulp, because it would be probably more mature. Also Grunt vs Gulp battle seem to me similar as Maven vs Gradle one in Java world. I never had a need to move away from Maven. Also don’t like idea of creating  some custom algorithms in build system (bad Bash and Ant experience in the past). Grunt is similar to Maven in terms of configuration approach. I can very easily understand any build in Maven and expect similar build consistency from Grunt.

Nearly immediately I started to feel that Node.JS and Express back-end build concerns (Mocha based test suite) are pretty different to concerns of Angular front-end build (minification, Require.js optimalization, Karma based test suite, …). There was clear distinction between these two.

My main problem was having separate test suites. Karma makes generation of unit test code coverage very easy. Slightly tricky was setting up generation of code coverage stats for Mocha based server unit test suite. I managed to do that with Instanbul. So far so good. But when I wanted to send my stats to Coveralls server I could do that only for one suite. Coveralls support one stat per project. Combining stat files didn’t work nicely for me.

Multi module JavaScript project

So I felt a need for splitting the projects. As I’m developer with Java background, this situation reminded me Maven multi module project. In this concept you can have various separated projects/sub-modules that can evolve independently. These can be grouped/integrated together via special multi module project. This way you can build large enterprise and also modular application.

So I said to myself, that I wouldn’t give a try to this stack until I figure out how to configure multi module project. I separated main repository called primediser into two:

(Notice I created branch blog-2014-05-19-multi-module-project to to have code consistent with blog post)

So now I am able to set up Continuous Integration for each project and submit coverage stats separately. But how to integrate these two together? I created umbrella project, that doesn’t contain any JavaScript production code (similar to multi module project in Maven world). It will contain only Protractor E2E tests and grunt file for integration two modules.  This project is located in separate Github repository called primediser.

It uses various Grunt plugins and one conditional trick to do the integration:

grunt-git

This plugin is used to clone mentioned sub-projects from Github:

    gitclone: {
      cloneServer: {
        options: {
          repository: 'https://github.com/lkrnac/<%= dirs.server %>',
          directory: '<%= dirs.server %>'
        },
      },
      cloneClient: {
        options: {
          repository: 'https://github.com/lkrnac/<%= dirs.client %>',
          directory: '<%= dirs.client %>'
        },
      },
    },

I could use grunt-shell plugin for this (I am using it anyway if you read further), but this one seems to be platform independent. Grunt-shell obviously isn’t.

Conditional cloning

Git can clone repository only once. Second attempt fails. Therefore we need to clone sub-projects only when they don’t exist. It is obviously up to developer to

  var cloneIfMissing = function (subTask) {
    var directory = grunt.config.get('gitclone')[subTask].options.directory;
    var exists = fs.existsSync(directory);
    if (!exists) {
      grunt.task.run('gitclone:' + subTask);
    }
  };

  grunt.registerTask('cloneSubprojects', function () {
    cloneIfMissing('cloneClient');
    cloneIfMissing('cloneServer');
  });

My setup expects that developer would update sub-projects as needed. Also expects that Continuous Integration system that throws away entire workspace after the build. If you would be using Jenkins, you could use similar conditional trick in conjunction with gitupdate maven task that grunt-git provides.

grunt-shell

After cloning, we need to install dependencies for both sub-projects. Unfortunately I didn’t find any platform independent way of doing this (Have to be honest I didn’t look very deeply though).

    shell: {
      npmInstallServer: {
        options: {
          stdout: true,
          stderr: true
        },
        command: 'cd <%= dirs.server %> && npm install && cd ..'
      },
      npmInstallClient: {
        options: {
          stdout: true,
          stderr: true
        },
        command: 'cd <%= dirs.client %> && npm install && bower install && cd ..'
      }
    },

grunt-hub

Next step is to kick off builds of sub-projects via grunt-hub plugin:

    hub: {
      client: {
        src: ['<%= dirs.client %>/Gruntfile.js'],
        tasks: ['build'],
      },
      server: {
        src: ['<%= dirs.server %>/Gruntfile.js'],
        tasks: ['build'],
      },
    },

Tasks configurations

  grunt.registerTask('npmInstallSubprojects', [
    'shell:npmInstallServer',
    'shell:npmInstallClient'
  ]);

  grunt.registerTask('buildSubprojects', [
    'hub:server',
    'hub:client'
  ]);

  grunt.registerTask('coverage', [
    'clean:coverageE2E',
    'copy:coverageStatic',
    'instrument',
    'copy:coverageJsServer',
    'copy:coverageJsClient',
    'express:coverageE2E',
    'protractor_coverage:chrome',
    'makeReport',
    'express:coverageE2E:stop'
  ]);

  grunt.registerTask('default', [
    'cloneSubprojects',
    'npmInstallSubprojects',
    'buildSubprojects',
    //'build',
    'coverage'
  ]);

As you can see there is one task I didn’t mention called coverage:

  • This one gatheres builds of sub-projects into dedicated sub-direcotory
  • Instrument the files
  • Run the Express back-end
  • Kicks Protractor end to end tests
  • and measures front end test coverage

Main driver in this task is grunt-protractor-coverage plugin. I already wrote blog post about this plugin. That blog post was done at stage when there wasn’t multi module configuration in place, so you can expect differences (There is also branch dedicated for that blog post also). Backbone should be the same though.

Measuring code coverage by Protractor end-to-end tests

Was just setting up new JavaScript project based on Grunt. I scaffolded the project template by Yeoman with usage of angular-fullstack generator. I decided to try MEAN stack without MongoDB for my new project (DB isn’t needed).

Next step was integrating Require.JS and configuring measurement of code coverage on client and server by Instanbul. When was this all done I was wondering if it is possible to measure code coverage by Protractor end-to-end testing.

After quick search I found that Ryan Bridges recently released grunt-protractor-coverage plugin. Interesting coincidence. So I decided to try it and can confirm that it’s working fine with mentioned stack. Configuration was smooth and Ryan fixed small issue very promptly. It’s based on grunt-protractor-runner plugin.

I created separate Grunt configuration file just for this purpose not to mess around with normal build. I had also problems to run ‘makeReport’ task of grunt-istanbul plugin for two different directories (Mocha server side code coverage measurement is using same task).

So here is the Grunt flow. First we need to copy non JavaScript files into target directory. It needs to be in separate directory because JavaScript files will need to be instrumented.

copy: {
  coverageE2E: {
    files: [{
      expand: true,
      dot: true,
      cwd: '<%= dirs.app %>',
      dest: '<%= dirs.instrumentedE2E %>/app',
      src: [
        '*.{ico,png,txt}',
        '.htaccess',
        'bower_components/**/*',
        'images/**/*',
        'fonts/**/*',
        'views/**/*',
        'styles/**/*',
      ]
    }]
  },
},

Next step is instrumentation of the code. It is needed for gathering coverage stats. Each line is decorated by special instructions that helps during measurement. Pay attention to fact that we are instrumenting server and client side code. Instrumented code is placed into target directory represented by placeholder <%= dirs.instrumentedE2E %>.

instrument: {
  files: ['server/**/*.js', 'app/scripts/**/*.js'],
  options: {
    lazy: true,
    basePath: '<%= dirs.instrumentedE2E %>/'
  }
},

Next we start Express from target directory.

express: {
  options: {
    port: process.env.PORT || 9000
  },
  coverageE2E: {
    options: {
      script: '<%= dirs.instrumentedE2E %>/lib/server.js',
      debug: true
    }
  },
},

And the protractor_coverage task of grunt-protractor-coverage plugin. Configuration should be the same as for grunt-protractor-runner plugin.

protractor_coverage: {
  options: {
    configFile: 'test/protractor/protractorConf.js', // Default config file
    keepAlive: true, // If false, the grunt process stops when the test fails.
    noColor: false, // If true, protractor will not use colors in its output.
    coverageDir: '<%= dirs.instrumentedE2E %>',
    args: {}
  },
  chrome: {
    options: {
      args: {
        baseUrl: 'https://localhost:3000/',
        // Arguments passed to the command
        'browser': 'chrome'
      }
    }
  },
},

Last step is generation  of coverage report.

makeReport: {
  src: '<%= dirs.instrumentedE2E %>/*.json',
  options: {
    type: 'html',
    dir: '<%= dirs.coverageE2E %>/reports',
    print: 'detail'
  }
},

Finally, this is grunt task gathering all steps.

grunt.registerTask('default', [
  'clean:coverageE2E',
  'copy:coverageE2E',
  'instrument',
  'express:coverageE2E',
  'protractor_coverage:chrome',
  'makeReport'
]);

EDIT: Notice that following Github link was changed to branch, because project structure was significantly changed:

Source code for this project can be found on GitHub.

For running end to end Protractor test you have to have webdriver-manager running. See  Protractor documentation.

To install and start Selenium Webdriver:

npm install -g protractor
webdriver-manager update
webdriver-manager start

To install project dependencies:

npm install
bower install

To run this end-to-end testing with coverage measurement (webdriver-manager has to be running):

grunt --gruntfile Gruntfile-e2e.js