Megazord is a very good fit to embed an assistant in any app: a game or an email app written in Electron, or even a console app written in NodeJS.

Because the Megazord library can run the entire platform, it means that the users of your app do not need to have the Snips platform preinstalled on their device to use your app. It also means that multiple apps running different assistants can all work at the same time.

This tutorial is going to show you how to use Megazord from the NodeJS bindings, in a simple console app, and in a graphical Electron app.

Prerequisites

The prerequisites for this project are:

• ​macOS El Capitan or later, or a Debian Linux distribution

• ​Node.js and NPM or yarn​

• ​Homebrew on macOS

• ​Electron​

Step 1: Install the Snips Megazord library

The Snips Megazord library embeds the whole Snips platform as a dynamic library which can be shipped with any app

First, add the Snips tap to Homebrew. In a terminal window, enter the following command:

brew tap snipsco/homebrew-snips

Then, install the Snips Megazord library:

brew install libsnips_megazord

Step 2: NodeJS assistant

You can get the code for this step at quickstart/vanilla​

Installing dependencies

The first step is to install the Megazord bindings:

yarn add megazord-javascript

We need to stream audio to the Megazord library. When running Electron, we can use the browser microphone, but when writing a console app, we need to use a library. A simple way to do this from node is to use the mic NPM package which uses arecord or sox to stream the audio to node:

yarn add mic

Using Megazord-javascript

The Megazord bindings provide a low-level API to the Megazord library, and a high-level SnipsPlatform wrapper to create assistants easily. This is what we will use:

main.js
const {SnipsPlatform} = require('megazord-javascript');
const mic = require('mic');
​
const assistant = new SnipsPlatform('assistant', {
    enableSnipsWatch: true,
    enableLogs: false,
});
​
var _currentSessionId = null;
assistant
    .onIntentDetected((msg, user_data) => {
        console.log('-- intent detected: ' + msg.intent.intent_name);
        console.log(' - input: ' + msg.input);
        _currentSessionId && assistant.endSession(_currentSessionId);
    })
    .onIntentNotRecognized((msg, user_data) => {
        console.log('-- intent not recognized: ' + msg.input);
        _currentSessionId && assistant.endSession(_currentSessionId);
    })
    .onSessionStarted((msg, user_data) => {
        _currentSessionId = msg.session_id;
    })
    .onSessionEnded((msg, user_data) => {
        _currentSessionId = null;
    })
    .onListeningStateChanged((isListening, user_data) => {
        console.log('-- is_listening: ' + isListening);
    })
    .onSnipsWatch((msg, user_data) => {
        console.log('-- snips_watch', msg);
    });
​
assistant.start();
​
// stream audio
​
var micInstance = mic({rate: '16000', channels: '1'});
var micInputStream = micInstance.getAudioStream();
​
micInputStream.on('data', function(data) {
    const buf = Buffer.from(data);
    assistant.appendBuffer(buf);
});
​
micInputStream.on('error', function(err) {
    console.error("Error: " + err);
});
​
micInstance.start();

Running the code

That's it! A few tens of lines of code to add a fully-functional assistant to any node app! You can unzip an assistant in the current path and run:

node main.js

Step 3: An Electron app

You can get the code for this step at quickstart/electron​

The Megazord javascript bindings make it really easy to add an assistant to any desktop app you might want to write: games, email clients, or even your own version of Siri!

Installing dependencies with the correct Node version

# remove previous `node_modules` if any
rm -rf node_modules
​
# using electron-build
yarn add --dev electron-rebuild
./node_modules/.bin/electron-rebuild
​
# or building manually
HOME=~/.electron-gyp npm install \
    --runtime=electron \
    --target=4.2.9 \
    --disturl=https://electronjs.org/headers \
    --build-from-source \
    --arch=x64 \
    --target-arch=x64

Using Megazord-javascript from an Electron app

We can use Electron to get the microphone input and send it to the assistant.

Running the code

The code is very similar to a regular NodeJS assistant, but we will be used a Recorder class to get the microphone input from the Electron browser. We will also add a way for the user to start using the assistant by either saying the wakeword or pressing a button.

The code at quickstart-megazord-javascript/electron is using React, but you can use any framework:

    componentDidMount() {
        createAudioRecorder(
            (data) => {
                this.appendBuffer(Buffer.from(data))
            },
            (recorder) => { recorder.start() }
        );
    }

You can then run yarn start to start the electron app and a small webserver which will serve the React code.

Step 4: Using entity injection

One of the powerful features of Snips is the ability to add custom entities to your assistant in real-time. A user of your app could add contact names, music artists, songs or any other entity, and then start using them!

You can look at the code at quickstart/injection​

The idea of injection is to add new values to an entity at runtime, either from "vanilla" (as though the assistant you are adding values to was freshly downloaded from the console), or incrementally, which keeps previous entities which you might have added before.

To allow the assistant to use injection, you should use enableInjection, and provide two paths: the path g2pResourcesPath to the lexical resources (on osX, it will be /usr/local/Cellar/snips-injection/0.63.3/share/snips/g2p-models), and a path userPath where the injection data will be cached

const {SnipsPlatform, SnipsConstants} = require('megazord-javascript');
​
const assistant = new SnipsPlatform('assistant', {
    enableInjection: true,
    userPath: 'user_injection',
    g2pResourcesPath: 'snips-g2p-resources',
    enableLogs: true,
});
​
assistant.start();
​
// entity injection
const lexicon = {}; // word: [pronunciations]
const operations = [{
    kind: SnipsConstants.SnipsInjectionKindEnum.ADD_FROM_VANILLA,
    values: {
        'contactName': [
            'Mom',
            'Steve',
            'Elon',
            'Bill'
        ]
    }
}];
​
assistant.requestInjection(operations, lexicon);

When running the injection request, the platform will update the ASR in the background and replace it when it is complete. Depending on the size of the assistant and the number of entities, it can take a few milliseconds to a few seconds. During this time, it is still possible to use the platform.

​