Megazord is a very good fit to embed an assistant in any app: a game or an email app written in Electron, or even a console app written in NodeJS.
Because the Megazord library can run the entire platform, it means that the users of your app do not need to have the Snips platform preinstalled on their device to use your app. It also means that multiple apps running different assistants can all work at the same time.
This tutorial is going to show you how to use Megazord from the NodeJS bindings, in a simple console app, and in a graphical Electron app.
The prerequisites for this project are:
The Snips Megazord library embeds the whole Snips platform as a dynamic library which can be shipped with any app
First, add the Snips tap to Homebrew. In a terminal window, enter the following command:
brew tap snipsco/homebrew-snips
Then, install the Snips Megazord library:
brew install libsnips_megazord
You can get the code for this step at quickstart/vanilla
The first step is to install the Megazord bindings:
yarn add megazord-javascript
We need to stream audio to the Megazord library. When running Electron, we can use the browser microphone, but when writing a console app, we need to use a library. A simple way to do this from node is to use the mic NPM package which uses arecord
or sox
to stream the audio to node:
yarn add mic
The Megazord bindings provide a low-level API to the Megazord library, and a high-level SnipsPlatform
wrapper to create assistants easily. This is what we will use:
main.jsconst {SnipsPlatform} = require('megazord-javascript');const mic = require('mic');const assistant = new SnipsPlatform('assistant', {enableSnipsWatch: true,enableLogs: false,});var _currentSessionId = null;assistant.onIntentDetected((msg, user_data) => {console.log('-- intent detected: ' + msg.intent.intent_name);console.log(' - input: ' + msg.input);_currentSessionId && assistant.endSession(_currentSessionId);}).onIntentNotRecognized((msg, user_data) => {console.log('-- intent not recognized: ' + msg.input);_currentSessionId && assistant.endSession(_currentSessionId);}).onSessionStarted((msg, user_data) => {_currentSessionId = msg.session_id;}).onSessionEnded((msg, user_data) => {_currentSessionId = null;}).onListeningStateChanged((isListening, user_data) => {console.log('-- is_listening: ' + isListening);}).onSnipsWatch((msg, user_data) => {console.log('-- snips_watch', msg);});assistant.start();// stream audiovar micInstance = mic({rate: '16000', channels: '1'});var micInputStream = micInstance.getAudioStream();micInputStream.on('data', function(data) {const buf = Buffer.from(data);assistant.appendBuffer(buf);});micInputStream.on('error', function(err) {console.error("Error: " + err);});micInstance.start();
That's it! A few tens of lines of code to add a fully-functional assistant to any node app! You can unzip an assistant in the current path and run:
node main.js
You might need to symlink the libsnips_megazord
library in the current path or to add it to your LD_LIBRARY_PATH
or DYLD_LIBRARY_PATH
to make sure that it will be found.
On osX, the library can be found in /usr/local/Cellar/libsnips_megazord
The code from quickstart/vanilla allows recording the streamed audio as .wav files which is useful to ensure that the audio streaming works correctly. You need to set recordQueryAudio
to true
in main.js
.
You can get the code for this step at quickstart/electron
The Megazord javascript bindings make it really easy to add an assistant to any desktop app you might want to write: games, email clients, or even your own version of Siri!
The version of the FFI lib used by the Megazord bindings and by Electron must be compatible. If there are errors mentioning the Node version, you might want to use electron-rebuild, or run the following commands (see this page) each time you install a new native package.
If there are still errors, you might want to remove ~/.node-gyp
and node_modules
, and build again
# remove previous `node_modules` if anyrm -rf node_modules# using electron-buildyarn add --dev electron-rebuild./node_modules/.bin/electron-rebuild# or building manuallyHOME=~/.electron-gyp npm install \--runtime=electron \--target=4.2.9 \--disturl=https://electronjs.org/headers \--build-from-source \--arch=x64 \--target-arch=x64
If there are still errors, you might want to remove ~/.node-gyp
and the node_modules
, and build again
We can use Electron to get the microphone input and send it to the assistant.
You might need to symlink the libsnips_megazord
library in the current path or to add it to your LD_LIBRARY_PATH
or DYLD_LIBRARY_PATH
to make sure that it will be found.
On osX, the library can be found in /usr/local/Cellar/libsnips_megazord
The code is very similar to a regular NodeJS assistant, but we will be used a Recorder
class to get the microphone input from the Electron browser. We will also add a way for the user to start using the assistant by either saying the wakeword or pressing a button.
The code at quickstart-megazord-javascript/electron
is using React, but you can use any framework:
componentDidMount() {createAudioRecorder((data) => {this.appendBuffer(Buffer.from(data))},(recorder) => { recorder.start() });}
You can then run yarn start
to start the electron app and a small webserver which will serve the React code.
One of the powerful features of Snips is the ability to add custom entities to your assistant in real-time. A user of your app could add contact names, music artists, songs or any other entity, and then start using them!
You can look at the code at quickstart/injection
The idea of injection is to add new values to an entity at runtime, either from "vanilla" (as though the assistant you are adding values to was freshly downloaded from the console), or incrementally, which keeps previous entities which you might have added before.
To use the entity injection, you need to install lexical resources
brew install snips-injection
To allow the assistant to use injection, you should use enableInjection
, and provide two paths: the path g2pResourcesPath
to the lexical resources (on osX, it will be /usr/local/Cellar/snips-injection/0.63.3/share/snips/g2p-models
), and a path userPath
where the injection data will be cached
const {SnipsPlatform, SnipsConstants} = require('megazord-javascript');const assistant = new SnipsPlatform('assistant', {enableInjection: true,userPath: 'user_injection',g2pResourcesPath: 'snips-g2p-resources',enableLogs: true,});assistant.start();// entity injectionconst lexicon = {}; // word: [pronunciations]const operations = [{kind: SnipsConstants.SnipsInjectionKindEnum.ADD_FROM_VANILLA,values: {'contactName': ['Mom','Steve','Elon','Bill']}}];assistant.requestInjection(operations, lexicon);
When running the injection request, the platform will update the ASR in the background and replace it when it is complete. Depending on the size of the assistant and the number of entities, it can take a few milliseconds to a few seconds. During this time, it is still possible to use the platform.
If the platform is stopping after doing the injection, you might want to use enableLogs
to see where the error is coming from. This could happen if you chose a slot name which does not exist in the assistant.