Quick Start nodeJS / Electron

Create an embedded Voice AI in an Electron app

Megazord is a very good fit to embed an assistant in any app: a game or an email app written in Electron, or even a console app written in NodeJS.

Because the Megazord library can run the entire platform, it means that the users of your app do not need to have the Snips platform preinstalled on their device to use your app. It also means that multiple apps running different assistants can all work at the same time.

This tutorial is going to show you how to use Megazord from the NodeJS bindings, in a simple console app, and in a graphical Electron app.

Prerequisites

The prerequisites for this project are:

Step 1: Install the Snips Megazord library

The Snips Megazord library embeds the whole Snips platform as a dynamic library which can be shipped with any app

First, add the Snips tap to Homebrew. In a terminal window, enter the following command:

brew tap snipsco/homebrew-snips

Then, install the Snips Megazord library:

brew install libsnips_megazord

Step 2: NodeJS assistant

You can get the code for this step at quickstart/vanilla

Installing dependencies

The first step is to install the Megazord bindings:

yarn add megazord-javascript

We need to stream audio to the Megazord library. When running Electron, we can use the browser microphone, but when writing a console app, we need to use a library. A simple way to do this from node is to use the mic NPM package which uses arecord or sox to stream the audio to node:

yarn add mic

Using Megazord-javascript

The Megazord bindings provide a low-level API to the Megazord library, and a high-level SnipsPlatform wrapper to create assistants easily. This is what we will use:

main.js
const {SnipsPlatform} = require('megazord-javascript');
const mic = require('mic');
const assistant = new SnipsPlatform('assistant', {
enableSnipsWatch: true,
enableLogs: false,
});
var _currentSessionId = null;
assistant
.onIntentDetected((msg, user_data) => {
console.log('-- intent detected: ' + msg.intent.intent_name);
console.log(' - input: ' + msg.input);
_currentSessionId && assistant.endSession(_currentSessionId);
})
.onIntentNotRecognized((msg, user_data) => {
console.log('-- intent not recognized: ' + msg.input);
_currentSessionId && assistant.endSession(_currentSessionId);
})
.onSessionStarted((msg, user_data) => {
_currentSessionId = msg.session_id;
})
.onSessionEnded((msg, user_data) => {
_currentSessionId = null;
})
.onListeningStateChanged((isListening, user_data) => {
console.log('-- is_listening: ' + isListening);
})
.onSnipsWatch((msg, user_data) => {
console.log('-- snips_watch', msg);
});
assistant.start();
// stream audio
var micInstance = mic({rate: '16000', channels: '1'});
var micInputStream = micInstance.getAudioStream();
micInputStream.on('data', function(data) {
const buf = Buffer.from(data);
assistant.appendBuffer(buf);
});
micInputStream.on('error', function(err) {
console.error("Error: " + err);
});
micInstance.start();

Running the code

That's it! A few tens of lines of code to add a fully-functional assistant to any node app! You can unzip an assistant in the current path and run:

node main.js

You might need to symlink the libsnips_megazord library in the current path or to add it to your LD_LIBRARY_PATH or DYLD_LIBRARY_PATH to make sure that it will be found.

On osX, the library can be found in /usr/local/Cellar/libsnips_megazord

The code from quickstart/vanilla allows recording the streamed audio as .wav files which is useful to ensure that the audio streaming works correctly. You need to set recordQueryAudio to true in main.js.

Step 3: An Electron app

You can get the code for this step at quickstart/electron

The Megazord javascript bindings make it really easy to add an assistant to any desktop app you might want to write: games, email clients, or even your own version of Siri!

Installing dependencies with the correct Node version

The version of the FFI lib used by the Megazord bindings and by Electron must be compatible. If there are errors mentioning the Node version, you might want to use electron-rebuild, or run the following commands (see this page) each time you install a new native package.

If there are still errors, you might want to remove ~/.node-gyp and node_modules, and build again

# remove previous `node_modules` if any
rm -rf node_modules
# using electron-build
yarn add --dev electron-rebuild
./node_modules/.bin/electron-rebuild
# or building manually
HOME=~/.electron-gyp npm install \
--runtime=electron \
--target=4.2.9 \
--disturl=https://electronjs.org/headers \
--build-from-source \
--arch=x64 \
--target-arch=x64

If there are still errors, you might want to remove ~/.node-gyp and the node_modules, and build again

Using Megazord-javascript from an Electron app

We can use Electron to get the microphone input and send it to the assistant.

Running the code

You might need to symlink the libsnips_megazord library in the current path or to add it to your LD_LIBRARY_PATH or DYLD_LIBRARY_PATH to make sure that it will be found.

On osX, the library can be found in /usr/local/Cellar/libsnips_megazord

The code is very similar to a regular NodeJS assistant, but we will be used a Recorder class to get the microphone input from the Electron browser. We will also add a way for the user to start using the assistant by either saying the wakeword or pressing a button.

The code at quickstart-megazord-javascript/electron is using React, but you can use any framework:

componentDidMount() {
createAudioRecorder(
(data) => {
this.appendBuffer(Buffer.from(data))
},
(recorder) => { recorder.start() }
);
}

You can then run yarn start to start the electron app and a small webserver which will serve the React code.

Step 4: Using entity injection

One of the powerful features of Snips is the ability to add custom entities to your assistant in real-time. A user of your app could add contact names, music artists, songs or any other entity, and then start using them!

You can look at the code at quickstart/injection

The idea of injection is to add new values to an entity at runtime, either from "vanilla" (as though the assistant you are adding values to was freshly downloaded from the console), or incrementally, which keeps previous entities which you might have added before.

To use the entity injection, you need to install lexical resources

brew install snips-injection

To allow the assistant to use injection, you should use enableInjection, and provide two paths: the path g2pResourcesPath to the lexical resources (on osX, it will be /usr/local/Cellar/snips-injection/0.63.3/share/snips/g2p-models), and a path userPath where the injection data will be cached

const {SnipsPlatform, SnipsConstants} = require('megazord-javascript');
const assistant = new SnipsPlatform('assistant', {
enableInjection: true,
userPath: 'user_injection',
g2pResourcesPath: 'snips-g2p-resources',
enableLogs: true,
});
assistant.start();
// entity injection
const lexicon = {}; // word: [pronunciations]
const operations = [{
kind: SnipsConstants.SnipsInjectionKindEnum.ADD_FROM_VANILLA,
values: {
'contactName': [
'Mom',
'Steve',
'Elon',
'Bill'
]
}
}];
assistant.requestInjection(operations, lexicon);

When running the injection request, the platform will update the ASR in the background and replace it when it is complete. Depending on the size of the assistant and the number of entities, it can take a few milliseconds to a few seconds. During this time, it is still possible to use the platform.

If the platform is stopping after doing the injection, you might want to use enableLogs to see where the error is coming from. This could happen if you chose a slot name which does not exist in the assistant.