Building native multi-platform tools with ReasonML, esy and Azure Pipelines

In this post we will go over how I implemented multi-platform packages for Reason with the help of esy and Azure Pipelines and publish them on npm. The high-level overview of the process is this:

  1. Create a npm release for each platform via esy release
  2. Copy all the different releases into a folder
  3. Create a package.json for the whole package and a postinstall.js file
  4. Copy README and LICENSE
  5. Create a placeholder file that we can replace in postinstall
  6. Package up and release to npm

If you just want to get this setup for free you can clone and use use hello-reason as a base or use the excellent esy-pesy tool to bootstrap your project.

What is esy?

esy is a package management tool that has a similar workflow to npm but built from the ground up to be fast, handle project isolation and work with native code. It is currently optimized for Reason and OCaml but can in theory be used for any language, there are multiple examples of C/C++ projects packaged as a esy-package.

I previously wrote a short post about ReasonML and esy that you can find here.

What is Azure Pipelines?

Azure Pipelines is part of Microsofts DevOps offering. It provides both CI and CD via builds and releases. Using different “agents” it’s possible to build on the three major platforms so there is no need for multiple CI platforms. It’s free for open source and has a about 30 hours free for private projects, and you can always host your own agent to have infinite free builds.

It has something they call jobs that can be used either to provide parallelism of tasks that don’t depend on each other and more interesting for this use case, multiple platforms in a single job. Jobs can also depend on each other and one job can provide input to the next.

There is a concept of artifacts that is something produced by the continuous integration pipeline that can either be consumed by another job a release or downloaded. Another useful feature is that it’s possible to split up our build definition in multiples files using templates. A template can almost be seen as a reusable function, you can break out some part of the job to a template and then pass parameters to it.

So how does the setup really work?

The rest of this blogpost will be going over the ins and outs of the setup I created for reenv, a dotenv-cli and dotenv-safe replacement written in Reason and compiled natively for Windows, macOS and Linux. It’s installable via npm but because it’s compiled to a native binary it’s about 10-20 times faster than the node equivalent.

This is the first part of the build definition. We start off by giving it a name and declaring when it should trigger a build, in this case any push to master, tag or PR will trigger a build. Then we use the template feature to declare the same job three times, once for each platform.

Building the package

Let’s go over the template that we use to build the different platforms.

First we declare default parameters, in this case we just use macOS as default. Then we set the name of the current job and the agent OS from the parameters.

The steps we go through to build for each platform is. Make sure the agent has a recent node version installed and install esy via npm. Install all dependencies with esy, run esy pesy to make sure the build config is correct and then build with the command specified in the package.json. We then create docs and copy then publish the generated artifacts. We will go over the testing template in more detail next. And the last step is to create the npm release for the current platform upload the artifact. The release is done via the esy release command that bundles the whole sandbox that esy has for the project and creates relocatable dlls and executables.

Running tests

We run esy test that will run the command that we have declared in the package.json and tell it to continue even if there is an error. Then we save the path to the junit.xml in a variable that we can use to publish a test report. Sadly, there is some difference in where the report is put on macOS/Linux and Windows so there is two different publishing, one when it’s not Windows and one when it is.

It will then generate a beautiful report like this, and yes, the tests are really that fast.

Creating the release

If this was a PR then we would be done as everything builds and we don’t have to create a release for it. But if this was either a push to master or a tag, we want to build a release that anyone can install. I use tags to create actual releases to npm, master will generate the package but it will not be pushed anywhere.

This job is also declared in the main file that was posted as the first image. We run it on a ubuntu machine and it has a condition to not run on PullRequests. We also depend on the previous steps as we need the released artifact that we create for each platform to make a combined package.

First we make sure we have a recent node version installed. Then we create a folder to put all the releases in.

Then for each platform we download the artifact that was created, create a separate folder inside the previously created release folder and copy the needed files into the newly created folder.

We then run a node script that creates a package.json without dependencies by reading your project package.json. It also copies your LICENSE and if they exist and copies a postrelease.js file. This script figures out what platform the package is installed on, copies the correct files in place and runs the esy generated postinstall script for the correct platform. The script that esy generate will replace placeholder paths so that they are actual paths on the consumers machine. The last step is to create a placeholder file and make it executable.

This script is interesting on it’s own as it’s both bash and batch at the same time, to learn more about it you can read this. The placeholder just echoes “You need to have postinstall enabled”. As it’s both bash and batch at the same time it’s runnable on all three platforms without modification. At first I had a js file but npm was “smart” and decided that my package should be run with node which broke the binary.

Testing the release before releasing it to npm

If the build was triggered because of a tag we want to test that our release will work on all the platforms before releasing it on npm. This is done for all the platforms.
First we make sure we have a recent version of node and then we download the the packaged release artifact. We then install the package from the tarball and run the binary, the pipeline will fail if the command exits with a non-zero exit code.

Releasing to npm

At the moment of writing this Azure Pipelines doesn’t support the release workflow in yaml which includes release gates and other powerfull features. I have a setup that I did through the UI that takes the package and basically runs npm publish on it. I will write a follow up on this post when the yaml feature is released. But getting it setup with the UI is pretty straight forward.

Disclaimer: All information on this blog is offered "as is" with no warranty. It is strongly recommended that you verify all information and validate all scripts in isolated test environments before using them in production environments.