Loading external libraries

[andrewn]

@pixelblend and I discussed how to download external libraries into neue-radio Apps last week. We decided to use the standard in the JavaScript ecosystem which is specifying them in a package.json. The dependencies are downloaded into a folder called node_modules in the app directory via NPM. The PR #58 addresses this so we now have a good way of downloading this library code.

Once you have the code downloaded, you need to load it into the browser. That's what this issue is about.

Ways of loading

When writing Apps for neue-radio there are two ways of loading code:

1. script tag in the HTML

<script src="./app.js"></script>

This approach works but puts all your code in the global scope and means that for every script file you want to include, you must have a script tag. It also requires that the external library code is bundled so it can be included in this way.

2. ES6 modules

<script type="module">
  import app from "./app.js";
  app();
</script>

This allows you to split your code into modules that depend on each other. The browser will load the code for you asyncronously and keep it out of the global namespace. These kind of modules cannot currently be used in Firefox which means our external pages don't work.

Example code

Here's an example of how we might want to load and use external modules. I want to use the emojilib and lodash utilities in my App as follows:

import emoji from 'my-app/modules/emojilib/index.js';
import lodash from 'my-app/modules/lodash/lodash.js';

export default () => {
  const categories = lodash.countBy(emoji.lib, ({ category }) => category);
  console.log(`Here are the number of emoji in each category`, categories);
}

This currently isn't possible because:

• emojilib:
  • uses the CommonJS method of exporting code using module.exports =require('./emojis')
  • requires a JSON file (which ES6 modules do not allow to be imported)
• lodash:
  • uses the UMD method of exporting code which either uses the CommonJS approach or exports a global variable

Possible solutions

The easiest approach for App developers might be to transparently transform the JavaScript code in the manager Service so that they could load code without worrying too much about the bundling.

Unfortunately, all the solution are filled with compromises, but here's a list of them.

Convert CommonJS modules into ES6 modules

The Rollup bundler can import CommonJS and Node modules but export ES modules so we can continue our favoured approach of using the future ES6 module standard that browsers are adopting.

This wouldn't help us with our current Firefox issue since we'd still be outputing ES6 modules.

Export a UMD bundle

This would transform all the ES6 modules into CommonJS so that it could be loaded by all browsers. We'd no longer be able to use the native ES6 modules via <script type="module"></script> but would instead just include the entrypoint of the app into the page via a <script src="./main.js"></script> tag.

express-middleware-rollup has express middleware for this.

browserify-middleware and babelify-middleware with the rollupify transform 🙄 both provide express middleware to achieve this.

Some of these middlewares don't seem to be actively maintained.

Support for full app bundling

An extreme option might be to use webpack or parcel to bundle the whole app which would also allow bundling assets and CSS. Although then we'd need to figure out what configuration to use and whether this could be overriden.

Tell App developers to do it themselves

They'd use a bundler on their computer and commit the bundled code. This would be very flexible but it introduces a disparity between development and production that I'm not sure we want?

[andrewn]

I've just come across lodash-es which is "The Lodash library exported as ES modules" which is very handy.

I think for libraries that are actively maintained and popular, this problem will go away eventually. But, for other modules, or those targeting node.js but that can run in the browser then this is still an issue.