Ember Blueprints part 3: executing our tests

This is the third in a multi-part series looking at Ember Blueprints. Part one covered building an Ember Blueprint. Part two moved it into an add-on and put the initial test scaffolding in place.

By the end of this post we will have a working Ember unit test running against the dynamically generated model-crud Blueprint. I hope this sounds as awesome to you as it did to me! I’m not aware of anywhere else you can find this information without having to figure it all out from source code.

It’s been a few weeks since we last spoke, so in true TV style:

Last time, on Ember Blueprints…

We finished up with the our main test runner looking like this:

The whole state up to this point can be found in this commit on Github.

Therefore we need to fill in the gaps. Broadly speaking these are:

  • Add the tests we want to run against the generated Ember app.
  • Install dependencies for the generated Ember app.
  • Generate code from the model-crud Ember Blueprint
  • Run the tests for generated Ember app.

Everything looks so much easier when written in bullet-points, so with a sense of optimism, let’s dive straight in.

Add tests

There are a couple of parts we need to address here:

  • Write the unit test
  • Remove any Phantom.js dependencies by writing a custom testem.js
  • Copy them both into the generated app as more fixture data

Write a test

Starting with the most simple case, we can simply use a route unit test that checks the route exists. From the previous posts, we know we’re going to configure three new routes /new-book, /book/<book_id>/view and /book/<book_id>/edit. Since at this stage we don’t want to worry about valid book_id values, we’ll pick new-book.

If this code looks familiar, then you’ll know this is simply the test created on a normal Ember project by running ember g route new-book. We don’t need anything more complicated than this right now.

As long as you save it inside the dummy/tests folder, the rest of the structure is optional. In this article, I’ve chosen to duplicate the structure created when generating tests automatically.

Remove Phantom.js Dependency

This is somewhat optional, but thoroughly recommended.

In order to run these tests we’re going to need to install all our dependencies beforehand. Phantom.js is a bit of a monster if you don’t already have it.

You probably already have Chrome, and we can use that to run our tests. In fact, in the very latest versions of Ember CLI, it’s the generated default, so you might find this step unnecessary after all.

The code below gives you a nice testem.js that will run your tests in headless Chrome. It’s a wonderful thing, and while we’re all very grateful for the options Phantom.js has provided to us over the years, we won’t be sad to see it replaced.

This testem.js file simply sits in the root of the dummy folder:

Copying the Test Files

If you recall from above, we’re already copying the models into the generated Ember instance, and we need to do the same here. The technique is exactly the same however.

Here is the updated model-crud-test.js file:

As you can see, the dummy constant has been modified slightly to make it more flexible for use in copying the test files. The tests folder and the testem.js file have simply been copied over to the correct places in the generated project, exactly where they would be for any Ember app.

To see where we’re up to at this point, view this commit on Github.

Installing Dependencies

If you create a new Ember project without installing any npm modules (e.g. ember new project_name --skip-npm), and then run ember test, you’ll get the following error:
node_modules appears empty, you may need to run 'npm install'

The same will happen when we try to run tests against the generated Ember app, therefore we need to install the project’s dependencies.

Unfortunately this process is pretty slow and dependent on a good internet connection. In tests, I found that yarn was about three times faster than npm on the configuration I’m currently running, so we’re going to use that.

The Ember CLI code ultimately uses execa to run a shelled yarn command when creating a new Ember instance. execa bills itself as “a better child_process”, but to avoid an extra dependency, we’re just going to use child_process.

Add this to the top of the model-crud-test.js file:

const { exec } = require('child_process');

Then we can insert usage of this command at the point in the code where the //todo: npm install comment exists.

If we were to now run npm run nodetest we’d get an error that looks similar to this:

What’s going on here? Well, our generated Ember app doesn’t know that mediasuite-ember-blueprints is the item under test, and not actually an npm module to be installed. There’s no way to indicate to the install process that it needs to skip this, but fortunately the ember-cli-blueprint-test-helpers library we’ve been using has a handy helper function for this.

It took a little digging through the source code, but if we now modify our code in the following way, it’ll all work:

Note how we use the modifyPackages function to remove the mediasuite-ember-blueprints package, and then add it back into the devDependencies once the install is complete.

Why are we adding it back in? Well, when we complete the next step (generating the code from our model-crud Blueprint), we’ll need it to be in the package.json file.

By adding this code, we’ve made a test cycle take significantly longer. If your goal is a quick test cycle whilst developing then this will no doubt grind your gears. I know it does for me. However, it wouldn’t be too tricky to copy node_modules to /tmp the first time you do this and then check that location before running yarn in the future. That way you could cache on an informal, per-session basis. I’ll leave that to you to decide if you fancy adding that optimisation into your own code.

There is a problem with timing that needs dealing with however. Unless you have a very high-speed internet connection and a top of the range computer, it’s likely the test will time-out when being executed. Mocha has a 20 second default for test execution. For me, this test takes about 85 seconds. It’s trivial to increase that timeout however – just drop a line of code after the test declaration.

All of this can be found at this commit.

Generate the Ember Blueprint

The final step before we can actually run tests is to generate our Blueprint. The end is firmly in sight at this stage, and I don’t mind admitting that I’m doing the knee dance at my standing desk…

‘Knee dance’ – dancing from the hips down, mainly in the knees, such that the arms and head stay still so as to not impact typing. Some gentle swaying allowed.

Anyway…

The awesome ember-cli-blueprint-test-helpers comes through for us again with the emberGenerate function.

Generating our Blueprint is as simple as including emberGenerate in the modules imported from bluePrintHelpers and returning the promise from the command into the promise chain. We are generating the crud forms for the book model, so we need to represent that in the arguments.

Boom! Simple, and working. When you run the test (npm run nodetest) again it should spit out the text to add into your router.js, so you know we’re on the right track.

The code up until this point can be found in this commit.

Executing the Tests

We’ve got two options here. We can try to shell the ember test command out as we had to do for installing dependencies, or we can use methods from Ember CLI code which are proxied through the ember-cli-blueprint-test-helpers library.

If we do try shell out the command, we end up with the following problem:

ember test requires all packages listed in package.json to be installed when it runs. Now, we can absolutely modifyPackages([{name: 'mediasuite-ember-blueprints', delete: true}]) ahead of this command, and everything will run just fine.

However, if we look at the Ember CLI alternative, it’s pretty interesting.

We can call the ember function with an array of arguments for a neater solution, so the code now looks like:

Run npm run nodetest now and you’ll see this glorious output:

The key here is the passing unit test. The test we wrote specifically for checking the output of the generated code is executing, and passing.

The whole temporary directory that the Ember application is generated inside of is removed as part of the ember-cli-blueprint-test-helpers test harness, so we don’t need to do any further tidy-up. Therefore, we can have the second greatest feeling in software development by removing code and pulling out that finally function.

This can all be found in the final commit.

Wrapping it up

We’re now able to write tests and have them execute against generated code. From this point forward we can write as many tests as we like in order to test our Blueprint’s generated code. It’s definitely good times.

This same technique can be used for any Blueprint you want to write, which in turn really opens the door into more ambitious code generation. We’re no longer limited to simply spinning up a new component or route, but instead anything we can dream of. Want to connect to a database, read the structure of a view and then generate models and routes? Entirely possible. Want to create search pages for different types of model, build it once as a Blueprint and then generate them as you need to? You can!

The only limit we have now is what we can do in a node script, which is really no limit at all. With the ability to be able to test our output, we can vastly reduce the time taken to author these Blueprints and make them viable on a per project basis.

I hope after reading through this lengthy series you feel empowered to try some more ambitious Blueprints yourself.

Discovering the joy of unit testing

With conservative estimation, each developer saves at least 3 hours a week on running angular unit tests. Before I was spending about 50 minutes a day waiting for tests to run. Now, tests run so fast I hardly have time to get distracted by banter in #random.

Don’t load more than you need too

The biggest feedback loop win has been the cryptically named NO_ERRORS_SCHEMA. The official documentation says, “Defines a schema that will allow any property on any element.”. It could be better described as “Render only components that are declared in your module, while courageously ignoring errors related to unknown components.”. Each test ran between 3 to 5 times faster after implementing this schema.

Each component test declares a very minimalist module. The module provides us with only the components which are important for executing the tests.

We learned that in most cases you only need to render a single component for each test.

Each component test defines a test module. The test module can import components by defining import modules. Previously we were defining over 100 components in the test module.

The key is in only declaring the component under test and adding NO_ERRORS_SCHEMA to your test module. This means that we don’t render the whole tree of components. A three to five time speed increase for each test was realised with this change.

On line 14 we include the NO_ERRORS_SCHEMA which allows us to declare only the component under test.

You should consider some of the disadvantages of NO_ERRORS_SCHEMA. When a component renders all child components, a lot of incidental testing happens.

Testing component interactions is something you need to be explicit about when using this schema. Each component that causes or reacts to side effects will need to be declared in the testing module.

Run only the relevant tests

When you’re developing tests for a new component, focus on the tests that are important. In jasmine, we have the “fdescribe” and “fit” function for this purpose.

Read about “fdescribe” and “fit” here:  https://jasmine.github.io/2.1/focused_specs.html

As you work on tests, natural groups of tests will appear. When this happens, it makes sense to wrap the tests in a “describe” function.  Focusing on this logical group of tests is now possible!

Replace PhantomJS with Chrome headless

When a test fails on CI while passing locally developers around the world perform a collective facepalm. In our project, phantomJS is frequently the culprit. As of April 2017 chrome headless is the solution.

Using chrome, our tests run about 30% faster in the CI environment. This was just the icing on the cake though. Our true win has been in having a consistent environment for local and CI test running. We have had no DOM behaviour inconsistencies between test environments.

If you’re using Angular CLI with Jasmine/Karma for tests, try it out.

  1. We need karma-chrome-launcher first. “npm install –save-dev karma-chrome-launcher”
  2. Copy the customLaunchers section from line 52

3. Update your test command to use the new launcher

ng lint && ng test --single-run --no-sourcemap --browsers 'chrome_headless'

// see the full example: https://gist.github.com/ebuckley/980baffb8a46a3cfe1b14434123e07ad#file-package-json-L13

Wrapping things up

If I tried to list all the ways you can speed up tests, you would never have time to go and make these easy wins. We refactored over 100 tests and the developer happiness really shows.

If your stack looks like ours, then you can probably implement these speed ups too. Go out, be courageous and take action. Make your tests fast, your team will not regret it.