Quick Start Jetson TX2

For this tutorial, we assume you have a Jetson TX2 with a recent version of Jetpack installed (at least 3.0).

Installation

Step 1: Installing the Snips Platform

From your Jetson TX2 terminal, update the APT package directory, adding the Snips repository:

sudo apt-get update
sudo apt-get install -y dirmngr
sudo bash -c 'echo "deb https://raspbian.snips.ai/nvidiajetsontx2_xenial stable main" > /etc/apt/sources.list.d/snips.list'
sudo apt-key adv --fetch-keys https://raspbian.snips.ai/531DD1A7B702B14D.pub
sudo apt-get update

We are ready to install the Snips Platform:

sudo apt-get install -y snips-platform-voice
sudo apt-get install -y snips-template snips-skill-server

Configuration

Configuring audio

This section shows you how to configure the audio input and output on your Jetson TX2. Before you begin, plug your USB microphone and speaker to the device.

Everything happens from the Jetson TX2 command line:

Step 1: List your devices

Audio output

To list all playback devices detected on the system, run the following command:

aplay -l

This will produce an output similar to the following:

**** List of PLAYBACK Hardware Devices ****
card 0: tegrahda [tegra-hda], device 3: HDMI 0 [HDMI 0]
Subdevices: 1/1
Subdevice #0: subdevice #0
card 0: tegrahda [tegra-hda], device 7: HDMI 0 [HDMI 0]
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 0: ADMAIF1 CIF ADMAIF1-0 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 1: ADMAIF2 CIF ADMAIF2-1 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 2: ADMAIF3 CIF ADMAIF3-2 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 3: ADMAIF4 CIF ADMAIF4-3 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 4: ADMAIF5 CIF ADMAIF5-4 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 5: ADMAIF6 CIF ADMAIF6-5 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 6: ADMAIF7 CIF ADMAIF7-6 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 7: ADMAIF8 CIF ADMAIF8-7 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 8: ADMAIF9 CIF ADMAIF9-8 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 9: ADMAIF10 CIF ADMAIF10-9 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 10: ADMAIF11 CIF ADMAIF11-10 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 11: ADMAIF12 CIF ADMAIF12-11 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 12: ADMAIF13 CIF ADMAIF13-12 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 13: ADMAIF14 CIF ADMAIF14-13 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 14: ADMAIF15 CIF ADMAIF15-14 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 15: ADMAIF16 CIF ADMAIF16-15 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 16: ADMAIF17 CIF ADMAIF17-16 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 17: ADMAIF18 CIF ADMAIF18-17 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 18: ADMAIF19 CIF ADMAIF19-18 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 19: ADMAIF20 CIF ADMAIF20-19 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 60: ADSP PCM1 ADSP-FE1-60 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 61: ADSP PCM2 ADSP-FE2-61 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 2: USB [Jabra SPEAK 510 USB], device 0: USB Audio [USB Audio]
Subdevices: 1/1
Subdevice #0: subdevice #0

The last device is what we are looking for: card 2 / device 0 (hw:0,0). The other device (hw:1,n) is the tegra audio output, which we won't use here.

Audio input

To list all audio capture devices, run the following command:

arecord -l

We should see something like this:

**** List of CAPTURE Hardware Devices ****
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 0: ADMAIF1 CIF ADMAIF1-0 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 1: ADMAIF2 CIF ADMAIF2-1 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 2: ADMAIF3 CIF ADMAIF3-2 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 3: ADMAIF4 CIF ADMAIF4-3 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 4: ADMAIF5 CIF ADMAIF5-4 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 5: ADMAIF6 CIF ADMAIF6-5 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 6: ADMAIF7 CIF ADMAIF7-6 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 7: ADMAIF8 CIF ADMAIF8-7 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 8: ADMAIF9 CIF ADMAIF9-8 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 9: ADMAIF10 CIF ADMAIF10-9 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 10: ADMAIF11 CIF ADMAIF11-10 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 11: ADMAIF12 CIF ADMAIF12-11 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 12: ADMAIF13 CIF ADMAIF13-12 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 13: ADMAIF14 CIF ADMAIF14-13 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 14: ADMAIF15 CIF ADMAIF15-14 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 15: ADMAIF16 CIF ADMAIF16-15 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 16: ADMAIF17 CIF ADMAIF17-16 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 17: ADMAIF18 CIF ADMAIF18-17 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 18: ADMAIF19 CIF ADMAIF19-18 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 19: ADMAIF20 CIF ADMAIF20-19 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 60: ADSP PCM1 ADSP-FE1-60 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: tegrasndt186ref [tegra-snd-t186ref-mobile-rt565x], device 61: ADSP PCM2 ADSP-FE2-61 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 2: USB [Jabra SPEAK 510 USB], device 0: USB Audio [USB Audio]
Subdevices: 0/1
Subdevice #0: subdevice #0

Audio capture device here is card 2 / device 0 (hw:2,0).

Step 2: Select an input and an output

Specifying the appropriate input and output devices is done on the/etc/asound.conf configuration file. Open it with your favorite command line editor, e.g.:

sudo vi /etc/asound.conf

Add the following section, and make sure that the hw: references match with the output of aplay and arecord noted above:

pcm.!default {
type asym
playback.pcm {
type plug
slave.pcm "hw:2,0"
}
capture.pcm {
type plug
slave.pcm "hw:2,0"
}
}

Step 3: Adjust input and output volume

In order to configure the sound card, open alsamixer:

alsamixer

The sound card that controls the input volume is the one of the microphone. Use the F6 key to select this card and the F4 key to play with the decibel gain.

If you can't change settings of your microphone it may due to the fact some microphones have no support mixer (that's the case for instance for the PlayStation Eye microphone).

Troubleshoot your audio setup (optional)

To run this test, you first need to stop the snips-audio-server service, as it takes hold of the microphone:

sudo systemctl stop snips-audio-server

Execute the following command, speak into your microphone, and hit CTRL-C when you are done speaking.

arecord -f cd out.wav

This will save a file, out.wav, containing what the microphone has recorded. To check that the microphone is properly working as well as the speaker, run the following:

$ aplay out.wav

You should hear the recorded sound. If you don't, you might want to play again with the decibel gain using alsamixer.

When you are satisfied that everything works, restart the Snips audio server:

$ sudo systemctl start snips-audio-server

Configuring the Automatic Speech Recognition (ASR) engine

Using the Snips native ASR module

If you use Snips on-device ASR, you're all set! This comes free and does not require a subscription.

We are continually adding languages to the Snips ASR. If, however, you wish to use a language that is not yet supported, you can use the Google Cloud Speed-to-Text service instead. Note though that this solution is cloud-based, and thus requires an Internet connection.

Using Google Cloud Speech-to-Text

Start by setting up your Google Cloud Speech-to-Text account and credentials on the Google API Console. From the Google Console, do the following:

  1. Enable the Google Speech API

  2. Download the credentials in JSON format

Next, copy the credentials file to your Jetson TX2 device, for example using scp. From your computer, run the following command from the directory where your googlecredentials.json file resides:

scp credentials.json \
nvidia@<ip>:/home/nvidia/googlecredentials.json

Connect to the Jetson TX2:

ssh nvidia@<ip>

Using admin permissions, move the credentials file to /usr/share/snips:

sudo mv /home/nvidia/googlecredentials.json /usr/share/snips/

Finally, tell Snips to use the Google ASR service rather than the native one:

sudo apt-get remove snips-asr
sudo apt-get install snips-asr-google

Installing a demo assistant

The Snips Platform is installed as a set of standard services. The services are monitored by systemdand they are automatically launched after installation or when you boot the system. They are also relaunched if they crash.

If you want to try out a working instance, you can try out the demo assistant, which is a simple weather assistant. It can be installed as follows:

sudo apt-get install snips-platform-demo

The assistant is installed, intents can be detected, but it still needs some skills to perform an action, such as answering when an intent related to the weather assistant is detected.

To do so, proceed to the skill repository and the assistant manual installation documentation.

Once installed, you will have a fully working setup, you will be able to start asking questions to your device, such as:

Hey Snips, what's the weather going to be tomorrow in Chicago?

Please note that when an intent is detected, all the dialog sessions not explicitely answered to with an end_session or a continue_session will time out and terminated by snips-dialogue service.

[Dialogue] session with id '9b799dde-ed26-408a-af3e-bb491df8569b' was ended on site default. The session was ended because one of the component didn't respond in a timely manner

Installing your assistant

If you haven't already, make sure to go through the following guide to create an assistant on the Snips Console:

Once your assistant is ready, download it to your computer. Copy it to your device, for instance using scp:

scp assistant.zip nvidia@<ip>:/home/nvidia

Establish a terminal connection to your device:

ssh nvidia@<ip>

Remove previously installed assistants and unzip the assistant to /usr/share/snips:

sudo rm -rf /usr/share/snips/assistant/
sudo mkdir /usr/share/snips
sudo unzip /home/nvidia/assistant.zip -d /usr/share/snips/

The assistant is installed, and we can restart the Snips Platform as follows:

sudo systemctl restart 'snips-*'

Monitoring components

Except for the sound hardware configuration, which mostly happens in /etc/asound.conf, the main configuration for the Snips Platform is located in the /etc/snips.toml file. For more details, check out the Platform Configuration article.

Systemctl

The daemons are running by a system user called _snips. If you need to stop, start and check the status of the different services, you can use systemctl. Here are a few examples.

sudo systemctl stop snips-dialogue
sudo systemctl start snips-dialogue
sudo systemctl status "snips-*

Snips Watch

Another handy tool is Snips Watch, which displays platform logs in real time. You can install it as follows:

sudo apt-get install snips-watch

Run it using the command

snips-watch

For more, or less logs, add the -v, -vv, -vvv flags, e.g.:

snips-watch -vvv

For more info, check out the complete Snips Watch guide:

System logs

All the components also send logs to /var/log/syslog, so you can monitor the live debug output by running:

sudo tail -f /var/log/syslog

Updating the platform

To update the platform (and the Raspbian operating system), simply run the following:

sudo apt-get update
sudo apt-get upgrade

Configuring individual components

If you want further information about how the platform works, please head to the on-device processing overview. In a nutshell, each component subscribes to events on the bus, and publishes events based on their results, that are taken over by other components, and so on. If you want to install specific components or replace one or several component, like the audio server or the Wake Word detector, you can install the packages separately. Here is the list of the packages of the platform:

  • mosquitto

  • snips-analytics

  • snips-asr

  • snips-audio-server

  • snips-dialogue

  • snips-hotword

  • snips-queries

  • snips-tts