Getting started with Broccoli and Ember.js

Broccoli is a new build tool released recently by Jo Liss.

The launch of the project was met with great enthusiasm and community support, especially from the Ember.js developers. But due to the recent release of the project, you might find it hard to use or integrate with an existing application.

Matt and I took the project for a spin last week, and we were impressed by the benefits of using Broccoli.

A bit of disclaimer.
Although I’m writing Ember.js apps for some time, this is the first time I decided to avoid using Ruby for builds. Nor Matt, nor I ever used existing Node.js build tools and this article is in no way an attempt to compare Broccoli with any of them.

Our task was simple. We wanted to achieve a bootstrapped Ember.js application, similar to what Stefan’s AppKit provides.

To list a few of the features:

• Ember.js idiomatic application structure with ES6 modules
• A persistence layer provided by Ember Data (think of it as an ORM for your application)
• Production, Development (Fixtures) and Test environments support
• Testing support using QUnit and Ember Testing package
• Continuous Integration cross-browser ready test runner
• CoffeScript support
• Less/SCSS support (we went using Less along with Bootstrap 3)

We started by forking Jo Liss fork of AppKit, but we ended up starting from scratch which allowed us to understand more deeply the integration with Broccoli and to shape a more streamlined build process taking into account configuration environments. Don’t get me wrong, both the original AppKit and the forked changes helped us a lot. Thanks to the authors.

Broccoli Filters Support #

Broccoli is especially powerful due to tree filters, although there are limitations to using them (limited support for complex integrations, ex: Emblem templating, limited integrations with processing libraries for CSS and HTML).
With all of these, CoffeeScript and Less implementations exist and are doing their job pretty well.

If you are familiar with the Sprockets or Rails Asset Pipeline, you will notice that it’s similar to Broccoli filters. Here’s an example how we chained Handlebars with CoffeeScript and Less file filters:

// Brocfile.js
function preprocess (tree) {
  tree = filterTemplates(tree, {
    extensions: ['hbs', 'handlebars'],
    compileFunction: 'Ember.Handlebars.compile'
  });
  tree = filterCoffeeScript(tree, {
    bare: true
  });
  tree = filterLess(tree, {
    compress: env === 'production',
    paths: ['.', './stylesheets', './vendor/bootstrap/less']
  });
  return tree;
}

Now you can pass any trees to the preprocess and it will make sure the matched files are picked by the appropriate filters.

You can also spot the Less integration with Bootstrap of which I will write a bit later.

The application structure #

There are almost no differences in terms of code organization if to compare with AppKit. The details we were missing were mostly related to the configuration environments support which we adopted and integrated into the application build process.

Note: Broccoli has an environment extension, but at this moment, all it does is to just to provide conventional access to the Node’s process.env where BROCCOLI_ENV attribute value is being read as the current configuration environment name and is limited to just to production and test. I suggest using this convention if you aren’t already, at least until the purpose of broccoli-env is fully decided.

By using Broccoli integration with Bower, access to the third party libraries was simple and easy. The snippet bellow showcases the purpose of broccoli.MergedTree and it shows how easy it is to manage trees using it.

// Brocfile.js
var appTree = broccoli.makeTree('app');
var configTree = broccoli.makeTree('config');
var vendorTree = broccoli.makeTree('vendor');
//...
var appFilesToAppend = [
  'jquery.js',
  'handlebars.js',
  'ember.js',
  'ember-data.js',
  'ember-resolver.js',
  'bootstrap.js',
  'config/environment.js',
  'config/environments/' + env + '.js'
];

configTree = pickFiles(configTree, {
  srcDir: '/',
  destDir: 'config'
});
configTree = preprocess(configTree);

appTree = pickFiles(appTree, {
  srcDir: '/',
  destDir: 'app'
});
appTree = preprocess(appTree);

appAndVendorTree = [appTree, configTree, vendorTree].concat(  broccoli.bowerTrees());
appAndVendorTree = new broccoli.MergedTree(appAndVendorTree);

appJs = compileES6(appAndVendorTree, {
  loaderFile: 'loader.js',
  ignoredModules: [
    'resolver'
  ],
  inputFiles: [
    'app/**/*.js'
  ],
  legacyFilesToAppend: appFilesToAppend,
  wrapInEval: env !== 'production',
  outputFile: '/assets/app.js'
});

if (env === 'production') {
  appJs = uglifyJavaScript(appJs, {
    mangle: true,
    compress: true
  });
}

Once the configuration environment integration was done, it allowed us to load environment related adapter, test libraries and fixtures.

We needed a fixtures adapter when developing the application, while for production and tests we wanted to use a real adapter with no fixtures and mocked HTTP responses.

Here’s what we came up with.
We made our adapter aware of the current environment:

// app/adapters/application.coffee
Adapter = DS.ActiveModelAdapter.extend
  namespace: 'api/v1'

FixtureAdapter = DS.FixtureAdapter.extend()

if window.ENV.development
  Adapter = FixtureAdapter

`export default Adapter`

And provided an initializer that will load any fixtures we provide in our configuration environment:

// app/initializers/fixtures_preloader.coffee
initializer =
  name: 'Fixture Pre-loader'
  after: 'store'
  initialize: (container, application) ->
    store = container.lookup('store:main')

    Ember.keys(requirejs._eak_seen).filter((key) ->
      !!key.match(/^app\/models\//) and DS.Model.detect(require(key)['default'])
    ).map (key) ->
      type = require(key)['default']
      typeKey = key.match(/^app\/models\/(.*)/)[1]

      type.FIXTURES = window.ENV.FIXTURES[typeKey]
      store.pushMany typeKey, type.FIXTURES if type.FIXTURES

`export default initializer`

Now defining fixtures is as easy as just changing the appropriate environment file:

// config/environments/development.coffee
window.ENV.development = true

window.ENV.FIXTURES = {
  user: [
    { id: 1, full_name: 'Tom Dale' }
    { id: 2, full_name: 'Yehuda Katz' }
    { id: 3, full_name: 'Jo Liss' }
  ]
}

I’m not completely happy having the flags all-over in the application files, but so far this is what worked great for us. If you think there are better solutions, please share them with us.

The application CSS and Bootstrap #

Broccoli has a LESS filter written by Sindre. Using Bower integration, and the concate tree transformation, it was a breeze to handle our CSS.

// Brocfile.js
var appCss = null;
var cssTree = broccoli.makeTree('stylesheets');
var cssFiles = [
  'qunit.css',
  'assets/app.css'
];

appCss = pickFiles(cssTree, {
  srcDir: '/',
  files: ['app.less'],
  destDir: 'assets'
});

if (env !== 'test') {
  cssFiles.shift();
}

appCss = new broccoli.MergedTree([appCss].concat(broccoli.bowerTrees()));
appCss = preprocess(appCss);

appCss = concatFiles(appCss, {
  inputFiles: cssFiles,
  outputFile: '/assets/app.css'
});

Since we told the preprocess Less filter we will be using Bootstrap, all we needed, is just to merge the Bower trees to provide the integration.
And since Bower trees will also include vendored packages like QUnit, we just included the qunit.css into the build. Broccoli knows where to search for it.
Thanks to our environments, we can dynamically remove it from our build if we are not testing.

Testing the build #

For testing Ember.js applications, QUnit framework has great support by using the Ember.Test package. We didn’t bother searching for other alternatives.

While the application build process seemed straight forward, the testing build might be a bit trickier. Mostly because we needed more libraries/trees included into the build and proper support for the continuous integration test runner.

While Ember.js AppKit encourages the use of the Trek’s httpRespond helper for mocking HTTP responses (think ajax/adapter requests), we had big issues making it work with the latest version of Ember (see #10. So we used jQuery Mockjax for the same purpose as fakehr.

To add testing support to our Brocfile.js we had to change the current application build process by extending the trees it uses and returning the appropriate result based on the used environment.

// Brocfile.js
var testsTree = broccoli.makeTree('tests');
//...
var testAppFilesToAppend = appFilesToAppend.concat([
  'qunit/qunit/qunit.js',
  'jquery-mockjax/jquery.mockjax.js',
  'tests/test_helper.js'
]);

testsTree = pickFiles(testsTree, {
  srcDir: '/',
  files: ['*.coffee', '**/*.coffee'],
  destDir: 'tests'
});
testsTree = preprocess(testsTree);

testsJs = new broccoli.MergedTree(
  [appAndVendorTree, testsTree].concat(broccoli.bowerTrees()));
testsJs = compileES6(testsJs, {
  loaderFile: 'loader.js',
  ignoredModules: [
    'resolver'
  ],
  inputFiles: [
    'app/**/*.js',
    'tests/*/*.js'
  ],
  legacyFilesToAppend: testAppFilesToAppend,
  outputFile: '/assets/app.js'
});

// If we are testing, return the compiled file with testing sources.
if (env === 'test') {
  appJs = testsJs;
}

You can see how flexible this step was to us, just because of the way Broccoli manages the filters and transformations. If we wanted to make this shorter, we could just integrate the application build process along with the testing environment build process.

Now returning our tress, is the last thing we need to do to get the build started.

// Brocfile.js
// ...
return [publicTree, appJs, appCss];

Picking the test runner for cross-browser testing was another issue. Initially we tried to setup Karma runner, but it simply refused to work for us because of some strange issues (first the ES6 conflicts, then the lack of integration with the launchers I needed, i.e. Chromium). We ended up using Testem, which is probably the most flexible test runner to integrate with and mostly Just Works™.

Now integrating Broccoli builds with Testem was as easy as using broccoli build and tweaking testem.json to point to the build directory.

// testem.json
{
    "framework": "qunit",
    "cwd": "build/",
    "test_page": "index.html"
}

And now putting everything together:

$ rm -rf ./build && BROCCOLI_ENV=test broccoli build ./build && testem ci -l phantomjs,firefox

To summarize the whole experience… Of course there were situations where the lack of documentation and examples created frustrations. Or the lack of integration with existing tools required reading the source code. But overall, Broccoli seems to be a fast and flexible build tool.
Some of the aspects I really liked and was happy to see are related to the similarities with the existing Ruby tools, simplicity and modularity.

We published our setup as a boilerplate on Github. Make sure you check it out and give Broccoli a try.

Ember.js Broccoli Boilerplate

Thanks to Alex Ciobica, Matthew Johnson and Flaviu Simihaian for reading and reviewing the drafts of this post.

Discuss on Hacker News.