Magento 2: Building Assets

I'm planning to do some investigation and experimentation with Magento 2's front end build process over the next few months. Consider this an introduction to my studies. I'll try to keep updates of anything cool I find on my blog, but I cannot promise that. I am a lousy correspondent. Anyway, what I've picked up already might be of use to people, so here goes.

The most fundamental change for front end developers (or, as I like to call them: developers) in Magento 2 is the notion that front end assets have to go through some sort of build step.

What do I mean by a build step? Well, in this context a build step is the idea that:

We create our front end source files in a location different to where they eventually get served from. Furthermore, these source files may be written in a different language or format to what is served.
In this case, the build step is the relocation of these files, coupled with the optional transformation process into the finished product.

By default, Magento's build step isn't up to much. But that doesn't matter, because the key fact is that it's easier to make a build step better or more comprehensive, than it is to introduce one in the first place. As long as the command or commands I have to run to compile my static assets to be ready for a browser to consume doesn't change, I don't need to know or care what's actually going on. All that happens is my life gets easier.

Having a build step opens up many opportunities, and I've spent a lot of time thinking about what the next steps might be for this. I'll be exploring these over the coming months, but first, I want to really understand the invariants of Magento's structure. What we have available, and what, if anything, we can't have.

Theme Fallback

One of the key features of Magento 1 was the idea of "theme fallback". Themes inherit from one another, and a particular static asset can be in a theme, or in any of its parents. As with many of the tentpole features of Magento 1, this has been turned up to 11 in Magento 2.

Let's recap. In Magento 1, a module could export static assets to a few different places:

  • /js/: This directory is the place where modules are supposed to put their JavaScript libraries, and module specific code. It's one big bucket, and nothing happens to it once it's put in here. This directory is accessible to the wider internet, and files just get served. If you wanted to override one of these files, you're out of luck.
  • /skin/frontend/base/default/: This is a directory, similar to /js/ in that it's accessible to the wider world. Every theme gets a spot in the /skin/ directory, but base/default/ is special because it pertains to a theme that you know for a fact is always installed, and always found. So you can put files here, get a skin URL in your layout, and trust that it will fall back. This allows other developers to customise your file by overwriting it and leveraging Magento's fallback system. As a footnote to certain developers: Yes, this is exactly like rewriting a class via a codepool hack. No, it's not ideal, but it's a lot more common in JavaScript, because JavaScript is more misunderstood and less customisable from the outside.
  • /skin/frontend/<package>/<theme>/: This is the general case of the above, and is where your theme's JavaScript should go. And, if you chose to overwrite any JavaScript from base/default/, that would go here, too.
  • /media/: The media directory is not usually checked into source control, so it's for generated files. Like the publicly accessible version of the /var/ directory. For example, if you're brave enough to turn on Magento's JavaScript combining feature, it dumps files here. I once wrote an extension that tried to do something similar.

That's your lot in Magento 1, and it serves us up a veritable smorgasbord of issues:

  1. Overriding things is hard: There's no real concept of dependency injection, so your options are to override the file (which you can only sometimes do), or add a new file that extends the old one (which only works sometimes, as well, with the added bonus of an extra HTTP request).
  2. Having a build step is harder than it needs to be. Sure, you can have a src/ directory in your theme, and build to a dist/ directory, and use those URLs in your layout. But what if you are using something like Sass, and you want to be able to request assets from a different theme? Nope.
  3. Dependency management doesn't exist: You want to use NPM, or something, in literally any capacity? Good luck, ma'am.

I'm delighted to inform you that Magento went to great lengths to totally solve (1), and do everything in (2) except actually build stuff. And make it fast. Their attempt at building assets is cripplingly slow. And, they punted on (3), which I'm happy about, because there are several complex issues in managing a separate set of dependencies, that many different entities care about.

So what actually changed? Well:

  • As part of the Great Module Reunification Effort, individual modules are now in control of their view layer, including their JavaScript and CSS. This sits in a directory view/<area>/web/ under the module root. This actually forces a build step upon us, unless you want to make all of your PHP files publicly accessible, or have a ridiculous set of access rules in your server configuration.
  • All static assets are served from exactly one directory. It's called /static/<area>/<themeVendor>/<theme>/<locale>/<assetPath>.
  • There is still a big bucket of global static assets, but it's called /lib/web/ rather than /js/. This just forms the base coat of what ends up in the aforementioned destination folder for our assets, and it's totally overridable in any theme.
  • Internationalisation is respected, and you can customise static assets per locale in a theme, rather than having to make a whole new theme. I love this feature, but it doesn't seem to have gotten too much love.

So all the assets in a module's view directory, or in lib/web, or in any of your themes' multiple web directories end up in /static/<area>/<themeVendor>/<theme>/<locale>/. In you're project, you can find the /static directory under /pub.

And no assets apart from those in /static are publicly accessible. Instead, Magento will take a URL, parse it, locate an asset, and copy that original file to a place in /static, and then serve a plain file for the rest of time. This process can happen on demand (in developer mode), or ahead of time (in production mode). (Note that for the purposes of this section, I'm not discussing any kind of build time processing, although Magento has some support for this.)

The rules for exactly how Magento translates a URL into a file in your project are comprehensively documented on Magento's Devdocs site, and I encourage you to check them out.

You Were Talking About a Build Step

Yes, I was talking about a build step. A build step is something we can run before we deploy our site to get everything ready. As you might have gathered, it should reconcile the theme fallback into a flat and complete hierarchy of files for a given theme and locale, so that Magento doesn't have to spend time compiling it.

Since we're modern developers now, it should also do some other stuff:

  • Compile our stylesheets with some preprocessor, like LESS, or Sass,
  • Collect all translation strings used in the static assets, and assemble the translated versions in some file ready for use on the front end.
  • Minify our JavaScript
  • Lint and test our code

Magento provided tools in Magento 2 that do all of the above. They did some of it in PHP, and finished it off with Grunt. But, I'm dreaming bigger than that:

  • I want to write my JavaScript in some wacky language, like ClojureScript, Elm,  BuckleScript, or even ES6. Can I compile that to a RequireJS compatible ES5 script?
  • I want to use PostCSS to do things like RTL transformations.
  • I want everything to be at least 10 times faster. More, if possible.
  • I want to be able to watch for changes across the entire project, and have it be intelligent about incrementally building things.
  • I want to apply some innovative front-end optimisations beyond minification, like concatenating modules that are often used together into bundles, to reach a compromise between HTTP requests and wasted JavaScript

Some or all of the above are provided in the magnificent Frontools from Snowdog, which is a wonderful example of the Magento community working together to improve Magento 2 for everybody. I would like more like this, please.