Using node modules with njs

• Environment
• Protobufjs
• DNS-packet

Often, a developer wants to use 3rd-party code, usually available as a library of some kind. In the JavaScript world, the concept of a module is relatively new, so there was no standard until recently. Many platforms (browsers) still don't support modules, which makes code reuse harder. This article describes ways to reuse Node.js code in njs.

Examples in this article use features that appeared in njs 0.3.8

There is a number of issues that may arise when 3rd-party code is added to njs:

• Multiple files that reference each other and their dependencies
• Platform-specific APIs
• Modern standard language constructions

The good news is that such problems are not something new or specific to njs. JavaScript developers face them daily when trying to support multiple disparate platforms with very different properties. There are instruments designed to resolve the above-mentioned issues.

• Multiple files that reference each other, and their dependencies

  This can be solved by merging all the interdependent code into a single file. Tools like browserify or webpack accept an entire project and produce a single file containing your code and all the dependencies.

• Platform-specific APIs

  You can use multiple libraries that implement such APIs in a platform-agnostic manner (at the expense of performance, though). Particular features can also be implemented using the polyfill approach.

• Modern standard language constructions

  Such code can be transpiled: this means performing a number of transformations that rewrite newer language features in accordance with an older standard. For example, babel project can be used to this purpose.

In this guide, we will use two relatively large npm-hosted libraries:

• protobufjs — a library for creating and parsing protobuf messages used by the gRPC protocol
• dns-packet — a library for processing DNS protocol packets

Environment

This document mostly employs a generic approach and avoids specific best practice advices concerning Node.js and JavaScript. Make sure to consult the corresponding package's manual before following the steps suggested here.

First (assuming Node.js is installed and operational), let's create an empty project and install some dependencies; the commands below assume we are in the working directory:

$ mkdir my_project && cd my_project
$ npx license choose_your_license_here > LICENSE
$ npx gitignore node

$ cat > package.json <<EOF
{
  "name":        "foobar",
  "version":     "0.0.1",
  "description": "",
  "main":        "index.js",
  "keywords":    [],
  "author":      "somename <[email protected]> (https://example.com)",
  "license":     "some_license_here",
  "private":     true,
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  }
}
EOF
$ npm init -y
$ npm install browserify

Protobufjs

The library provides a parser for the .proto interface definitions and a code generator for message parsing and generation.

In this example, we will use the helloworld.proto file from the gRPC examples. Our goal is to create two messages: HelloRequest and HelloResponse. We will use the static mode of protobufjs instead of dynamically generating classes, because njs doesn't support adding new functions dynamically due to security considerations.

Next, the library is installed and the JavaScript code implementing message marshalling is generated from the protocol definition:

$ npm install protobufjs
$ npx pbjs -t static-module helloworld.proto > static.js

Thus, the static.js file becomes our new dependency, storing all the code we need to implement message processing. The set_buffer() function contains code that uses the library to create a buffer with the serialized HelloRequest message. The code resides in the code.js file:

var pb = require('./static.js');

// Example usage of protobuf library: prepare a buffer to send
function set_buffer(pb)
{
    // set fields of gRPC payload
    var payload = { name: "TestString" };

    // create an object
    var message = pb.helloworld.HelloRequest.create(payload);

    // serialize object to buffer
    var buffer = pb.helloworld.HelloRequest.encode(message).finish();

    var n = buffer.length;

    var frame = new Uint8Array(5 + buffer.length);

    frame[0] = 0;                        // 'compressed' flag
    frame[1] = (n & 0xFF000000) >>> 24;  // length: uint32 in network byte order
    frame[2] = (n & 0x00FF0000) >>> 16;
    frame[3] = (n & 0x0000FF00) >>>  8;
    frame[4] = (n & 0x000000FF) >>>  0;

    frame.set(buffer, 5);

    return frame;
}

var frame = set_buffer(pb);

To ensure it works, we execute the code using node:

$ node ./code.js
Uint8Array [
    0,   0,   0,   0,  12, 10,
   10,  84, 101, 115, 116, 83,
  116, 114, 105, 110, 103
]

You can see that this got us a properly encoded gRPC frame. Now let's run it with njs:

$ njs ./code.js
Thrown:
Error: Cannot find module "./static.js"
    at require (native)
    at main (native)

Modules are not supported, so we've received an exception. To overcome this issue, let's use browserify or other similar tool.

An attempt to process our existing code.js file will result in a bunch of JS code that is supposed to run in a browser, i.e. immediately upon loading. This isn't something we actually want. Instead, we want to have an exported function that can be referenced from the nginx configuration. This requires some wrapper code.

In this guide, we use njs cli in all examples for the sake of simplicity. In real life, you will be using nginx njs module to run your code.

The load.js file contains the library-loading code that stores its handle in the global namespace:

global.hello = require('./static.js');

This code will be replaced with merged content. Our code will be using the "global.hello" handle to access the library.

Next, we process it with browserify to get all dependencies into a single file:

$ npx browserify load.js -o bundle.js -d

The result is a huge file that contains all our dependencies:

(function(){function......
...
...
},{"protobufjs/minimal":9}]},{},[1])
//# sourceMappingURL..............

To get final "njs_bundle.js" file we concatenate "bundle.js" and the following code:

// Example usage of protobuf library: prepare a buffer to send
function set_buffer(pb)
{
    // set fields of gRPC payload
    var payload = { name: "TestString" };

    // create an object
    var message = pb.helloworld.HelloRequest.create(payload);

    // serialize object to buffer
    var buffer = pb.helloworld.HelloRequest.encode(message).finish();

    var n = buffer.length;

    var frame = new Uint8Array(5 + buffer.length);

    frame[0] = 0;                        // 'compressed' flag
    frame[1] = (n & 0xFF000000) >>> 24;  // length: uint32 in network byte order
    frame[2] = (n & 0x00FF0000) >>> 16;
    frame[3] = (n & 0x0000FF00) >>>  8;
    frame[4] = (n & 0x000000FF) >>>  0;

    frame.set(buffer, 5);

    return frame;
}

// functions to be called from outside
function setbuf()
{
    return set_buffer(global.hello);
}

// call the code
var frame = setbuf();
console.log(frame);

Let's run the file using node to make sure things still work:

$ node ./njs_bundle.js
Uint8Array [
    0,   0,   0,   0,  12, 10,
   10,  84, 101, 115, 116, 83,
  116, 114, 105, 110, 103
]

Now let's proceed further with njs:

$ njs ./njs_bundle.js
Uint8Array [0,0,0,0,12,10,10,84,101,115,116,83,116,114,105,110,103]

The last thing will be to use njs-specific API to convert array into byte string, so it could be usable by nginx module. We can add the following snippet before the line return frame; }:

if (global.njs) {
    return String.bytesFrom(frame)
}

Finally, we got it working:

$ njs ./njs_bundle.js |hexdump -C
00000000  00 00 00 00 0c 0a 0a 54  65 73 74 53 74 72 69 6e  |.......TestStrin|
00000010  67 0a                                             |g.|
00000012

This is the intended result. Response parsing can be implemented similarly:

function parse_msg(pb, msg)
{
    // convert byte string into integer array
    var bytes = msg.split('').map(v=>v.charCodeAt(0));

    if (bytes.length < 5) {
        throw 'message too short';
    }

    // first 5 bytes is gRPC frame (compression + length)
    var head = bytes.splice(0, 5);

    // ensure we have proper message length
    var len = (head[1] << 24)
              + (head[2] << 16)
              + (head[3] << 8)
              + head[4];

    if (len != bytes.length) {
        throw 'header length mismatch';
    }

    // invoke protobufjs to decode message
    var response = pb.helloworld.HelloReply.decode(bytes);

    console.log('Reply is:' + response.message);
}

DNS-packet

This example uses a library for generation and parsing of DNS packets. This a case worth considering because the library and its dependencies use modern language constructions not yet supported by njs. In turn, this requires from us an extra step: transpiling the source code.

Additional node packages are needed:

$ npm install @babel/core @babel/cli @babel/preset-env babel-loader
$ npm install webpack webpack-cli
$ npm install buffer
$ npm install dns-packet

The configuration file, webpack.config.js:

const path = require('path');

module.exports = {
    entry: './load.js',
    mode: 'production',
    output: {
        filename: 'wp_out.js',
        path: path.resolve(__dirname, 'dist'),
    },
    optimization: {
        minimize: false
    },
    node: {
        global: true,
    },
    module : {
        rules: [{
            test: /\.m?js$$/,
            exclude: /(bower_components)/,
            use: {
                loader: 'babel-loader',
                options: {
                    presets: ['@babel/preset-env']
                }
            }
        }]
    }
};

Note we are using "production" mode. In this mode webpack does not use "eval" construction not supported by njs. The referenced load.js file is our entry point:

global.dns = require('dns-packet')
global.Buffer = require('buffer/').Buffer

We start the same way, by producing a single file for the libraries:

$ npx browserify load.js -o bundle.js -d

Next, we process the file with webpack, which itself invokes babel:

$ npx webpack --config webpack.config.js

This command produces the dist/wp_out.js file, which is a transpiled version of bundle.js. We need to concatenate it with code.js that stores our code:

function set_buffer(dnsPacket)
{
    // create DNS packet bytes
    var buf = dnsPacket.encode({
        type: 'query',
        id: 1,
        flags: dnsPacket.RECURSION_DESIRED,
        questions: [{
            type: 'A',
            name: 'google.com'
        }]
    })

    return buf;
}

Note that in this example generated code is not wrapped into function and we do not need to call it explicitly. The result is in the "dist" directory:

$ cat dist/wp_out.js code.js > njs_dns_bundle.js

Let's call our code at the end of a file:

var b = set_buffer(global.dns);
console.log(b);

And execute it using node:

$ node ./njs_dns_bundle_final.js
Buffer [Uint8Array] [
    0,   1,   1, 0,  0,   1,   0,   0,
    0,   0,   0, 0,  6, 103, 111, 111,
  103, 108, 101, 3, 99, 111, 109,   0,
    0,   1,   0, 1
]

Make sure this works as expected, and then run it with njs:

$ njs ./njs_dns_bundle_final.js
Uint8Array [0,1,1,0,0,1,0,0,0,0,0,0,6,103,111,111,103,108,101,3,99,111,109,0,0,1,0,1]

The response can be parsed as follows:

function parse_response(buf)
{
    var bytes = buf.split('').map(v=>v.charCodeAt(0));

    var b = global.Buffer.from(bytes);

    var packet = dnsPacket.decode(b);

    var resolved_name = packet.answers[0].name;

    // expected name is 'google.com', according to our request above
}