Hermes Reference

Messages on the Snips Platform are passed along as MQTT messages. On Android and iOS, these messages are passed as callbacks. In this document, we specify the format of the messages, and how they flow across the various components of the Snips Platform.

To illustrate the MQTT API below, we use the Eclipse Mosquitto Client for publishing messages and subscribing to topics over MQTT.

Wake Word

The Wake Word component is in charge of detecting when a wake word has been detected, and where it comes from (in case of a multi-satellite setup).

Activating the Wake Word component

This will activate the Wake Word component of the Snips Platform.

MQTT
Android
iOS

Topic (publish)

hermes/hotword/toggleOn

Payload

Key

Type

Description

siteId

String

The id of the site where the wake word detector should be turned on

Example

mosquitto_pub -h <HOSTNAME> -t 'hermes/hotword/toggleOn' \
-m '{"siteId": "default"}'

This feature is not available yet on Android.

This feature is not available yet on iOS.

Deactivating the Wake Word component

This will deactivate the Wake Word component of the Snips Platform. Consequently, nothing will be triggered when a wake word is pronounced.

MQTT
Android
iOS

Topic (publish)

hermes/hotword/toggleOff

Payload

Key

Type

Description

siteId

String

The id of the site where the wake word detector should be turned on

sessionId

Optional String

The id of the session, if there is an active session

Example

mosquitto_pub -h <HOSTNAME> -t 'hermes/hotword/toggleOff' \
-m '{"siteId": "default"}'

This feature is not available yet on Android.

This feature is not available yet on iOS.

Detecting a wake word

This message will be sent by the Snips Platform when the Wake Word component has detected that a specific wake word has been uttered.

MQTT
Android
iOS

Topic (subscribe)

hermes/hotword/<WAKEWORD_ID>/detected

Replace <WAKEWORD_ID> with the id of your wake word if multiple wake words are present, or default if there is only one wake word.

Payload

Key

Type

Description

siteId

String

The id of the site where the wake word detector should be turned on

modelId

String

The id of the model that triggered the wake word

modelVersion

String

The version of the model

modelType

String

The type of the model. Possible values: universal or personal

currentSensitivity

Float

The sensitivity configured in the model at the time of the detection

Example

mosquitto_sub -h <HOSTNAME> -t 'hermes/hotword/default/detected'

This feature is not available yet on Android.

The event will trigger the SnipsPlatform.onHotwordDetected closure.

Example

let snips = SnipsPlatform(...)
snips.onHotwordDetected = { [weak self] in
// Handler code
}

Automatic Speech Recognition (ASR)

The ASR component receives raw audio input and transcribes it into text.

Activating the ASR component

This will activate the ASR component, subsequently enabling to start listening for voice (using the startListening call described below).

MQTT
Android
iOS

Topic (publish)

hermes/asr/toggleOn

Example

mosquitto_pub -h <HOSTNAME> -t 'hermes/asr/toggleOn'

This feature is not available yet on Android.

This feature is not available yet on iOS.

Deactivating the ASR component

This will deactivate the ASR component.

MQTT
Android
iOS

Topic (publish)

hermes/asr/toggleOff

Example

mosquitto_pub -h <HOSTNAME> -t 'hermes/asr/toggleOn'

This feature is not available yet on Android.

This feature is not available yet on iOS.

Telling the ASR component to start listening

This will explicitly tell the ASR component to start listening for voice input.

MQTT
Android
iOS

Topic (publish)

hermes/asr/startListening

Payload

Key

Type

Description

siteId

String

The id of the site where the ASR should start listening

sessionId

Optional String

The id of the session, if there is an active session

Example

mosquitto_pub -h <HOSTNAME> -t 'hermes/asr/startListening' \
-m '{"siteId": "default"}'

This feature is not available yet on Android.

This feature is not available yet on iOS.

Telling the ASR component to stop listening

This will explicitly tell the ASR component to stop listening for voice input.

MQTT
Android
iOS

Topic (publish)

hermes/asr/stopListening

Payload

Key

Type

Description

siteId

String

The id of the site where the ASR should stop listening

sessionId

Optional String

The id of the session, if there is an active session

Example

mosquitto_pub -h <HOSTNAME> -t 'hermes/asr/stopListening' \
-m '{"siteId": "default"}'

This feature is not available yet on Android.

This feature is not available yet on iOS.

Obtaining intermediate ASR transcription results

When the ASR is listening, it transcribes voice to text in real time. This process stops when a longer period of silence is detected. Before it stops however, intermediate transcription results can be obtained, as described here.

MQTT
Android
iOS

In order to enable partial ASR transcriptions, you will need to start the platform with the partial flag set to true in the [snips-asr] section of /etc/snips.toml:

[snips-asr] partial=true

For more information, see Platform Configuration.

Topic (subscribe)

hermes/asr/partialTextCaptured

Payload

Key

Type

Description

text

String

The text captured

likelihood

Float

The likelihood of the capture

seconds

Float

The duration it took to do the processing

siteId

String

The id of the site where the text was captured

sessionId

Optional String

The id of the session, if there is an active session

Example

mosquitto_sub -h <HOSTNAME> -t 'hermes/asr/partialTextCaptured'

This feature is not available yet on Android.

This feature is not available yet on iOS.

Obtaining full ASR transcription results

When the ASR is listening, it transcribes voice to text in real time. This process stops when a longer period of silence is detected, after which the transcription results are posted, as described here.

MQTT
Android
iOS

Topic (subscribe)

hermes/asr/textCaptured

Payload

Key

Type

Description

text

String

The text captured

likelihood

Float

The likelihood of the capture

seconds

Float

The duration it took to do the processing

siteId

String

The id of the site where the text was captured

sessionId

Optional String

The id of the session, if there is an active session

Example

mosquitto_sub -h <HOSTNAME> -t 'hermes/asr/textCaptured'

This feature is not available yet on Android.

This feature is not available yet on iOS.

Natural Language Understanding (NLU)

Sending text to the NLU component

In order to extract an intent and slots from a piece of text, send it directly to the NLU component as follows. The NLU component will subsequently publish a message to the hermes/nlu/intentParsed topic, described below.

MQTT
Android
iOS

Topic (publish)

hermes/nlu/query

Payload

Key

Type

Description

input

String

The text to send to the NLU component

intentFilter

Optional Array of String

A list of intent names to restrict the NLU resolution on

id

Optional String

A request identifier. If provided, it will be passed back in the response on hermes/nlu/intentParsed or hermes/nlu/intentNotRecognized

sessionId

String

The id of the session, if there is an active session

Example

mosquitto_pub -h <HOSTNAME> -t 'hermes/nlu/query' \
-m '{"input": "What is the weather going to be in Paris tomorrow", "intentFilter": "myGetWeatherForecastIntent"}'

This feature is not available yet on Android.

This feature is not available yet on iOS.

Sending text to the NLU component for slot detection

In some cases, for instance if a required slot is missing from an intent, a follow up query can be performed to extract a specific slot given an intent. This is done as follows.

MQTT
Android
iOS

Topic (publish)

hermes/nlu/partialQuery

Payload

Key

Type

Description

input

String

The text to run the slot detection on

intentName

String

The intent to use when doing the slot detection

slotName

String

The slot to search for

id

Optional String

A request identifier. If provided, it will be passed back in the response on hermes/nlu/slotParsed

sessionId

String

The id of the session, if there is an active session

Example

mosquitto_pub -h <HOSTNAME> -t 'hermes/nlu/partialQuery' \
-m '{"input": "In Paris", "intentName": "myGetWeatherForecastIntent", "slotName": "location"}'

This feature is not available yet on Android.

This feature is not available yet on iOS.

Obtaining the result of an NLU parsing (low-level API)

When text has been sent to the NLU (see Sending text to the NLU component), the result of the intent resolution is sent back as follows.

MQTT
Android
iOS

Topic (subscribe)

hermes/nlu/intentParsed

Note that this is a low-level API, and it is not recommended to be used of production. In particular, this API method does not guarantee that all slots of the intent have been properly parsed. To detect an intent parsed by the NLU component, it is recommended to subscribe to the following topic instead:

hermes/intent/<INTENT_NAME>

where INTENT_NAME is the name of the intent that has been parsed. See below.

Payload

Key

Type

Description

id

Optional String

Request identifier for the request passed from hermes/nlu/query

input

String

The input that was processed

intent

JSON Object

Structured description of the intent classification

slots

Optional Array of JSON Objects

Structured description of the slots for the detected intent, if any

sessionId

String

The id of the session, if there is an active session

Example

mosquitto_sub -h <HOSTNAME> -t 'hermes/nlu/intentParsed'

This feature is not available yet on Android.

This feature is not available yet on iOS.

Obtaining the result of an NLU slot detection

When text has been sent to the NLU for slot parsing (see Sending text to the NLU component for slot detection), the result of the slot detection is sent back as follows.

MQTT

Topic (subscribe)

hermes/nlu/slotParsed

Payload

Key

Type

Description

id

Optional String

Request identifier for the request passed from hermes/nlu/partialQuery

input

String

The input that was processed

intentName

String

The intent used to find the slot

slot

Optional JSON Object

The resulting slot, if found

sessionId

String

The id of the session, if there is an active session

Example

mosquitto_sub -h <HOSTNAME> -t 'hermes/nlu/slotParsed'

Being notified when an intent was not recognised

When the NLU was unable to parse a chunk of text, it publishes a message telling so.

MQTT
Android
iOS

Topic (subscribe)

hermes/nlu/intentNotRecognized

Payload

Key

Type

Description

id

Optional String

Request identifier for the request passed from hermes/nlu/query

input

String

The input that was processed, but didn't match any intent

sessionId

String

The id of the session, if there is an active session

Example

mosquitto_sub -h <HOSTNAME> -t 'hermes/nlu/intentNotRecognized'

This feature is not available yet on Android.

This feature is not available yet on iOS.

Being notified when an error has occurred

When an error has occurred in the NLU component, it publishes a message telling so.

MQTT
Android
iOS

Topic (publish)

hermes/error/nlu

Payload

Key

Type

Description

sessionId

Optional String

The id of the session, if there is an active session

error

String

A description of the error that occurred

context

Optional String

Additional information on the context in which the error occurred

Example

mosquitto_sub -h <HOSTNAME> -t 'hermes/error/nlu'

This feature is not available yet on Android.

This feature is not available yet on iOS.

Dialogue Manager

See Dialogue Manager Reference.

Text-to-Speech (TTS)

Sending text to be spoken by the TTS component (low-level API)

The Snips Platform comes with a Text-to-Speech component that allows text to be spoken.

MQTT
Android
iOS

Topic (publish)

hermes/tts/say

Note that this is a low-level API, and it is not recommended to be used of production. You should use the following topics instead, based on the dialogue manager:

  • hermes/dialogueManager/startSession

  • hermes/dialogueManager/continueSession

See the Dialogue API Reference for further explanations.

Payload

Key

Type

Description

text

String

The text to be spoken

lang

Optional String

The language code to use when saying the text. If nothing is provided, en_GB will be used

id

Optional String

A request identifier. If provided, it will be passed back in the response on hermes/tts/sayFinished

siteId

String

The id of the site where the text should be spoken

sessionId

Optional String

The id of the session, if there is an active session

Example

mosquitto_pub -h <HOSTNAME> -t 'hermes/tts/say' \
-m '{"text": "Bonjour!", "lang": "fr_FR"}'

This feature is not available yet on Android.

This feature is not available yet on iOS.

Being notified when TTS has finished speaking some text

When TTS has finished speaking some text, it will publish a message as follows.

MQTT
Android
iOS

Topic (subscribe)

hermes/tts/sayFinished

Payload

Key

Type

Description

id

Optional String

Request identifier for the request passed from hermes/tts/say

sessionId

Optional String

The id of the session, if there is an active session

Example

mosquitto_sub -h <HOSTNAME> -t 'hermes/tts/sayFinished'

This feature is not available yet on Android.

This feature is not available yet on iOS.

Audio Server

The Snips Platform comes with an audio server to easily handle playing sound on different sites.

Playing a WAV sound

You may send a WAV sound to be played on a specific site as follows.

MQTT
Android
iOS

Topic (publish)

hermes/audioServer/<SITE_ID>/playBytes/<REQUEST_ID>

Replace <SITE_ID> with the site on which to play the sound (e.g. default), and <REQUEST_ID> with an id to be passed back on hermes/audioServer/<SITE_ID>/playFinished (see below).

Binary Payload

The WAV file to play.

Example

mosquitto_pub -h <HOSTNAME> \
-t 'hermes/audioServer/default/playBytes/8ewnjksdf093jb42' \
-f sound.wav

This feature is not available yet on Android.

This feature is not available yet on iOS.

Being notified when sound has finished playing

When the audio service has finished playing a sound, it publishes a message as follows.

MQTT
Android
iOS

Topic (subscribe)

hermes/audioServer/<SITE_ID>/playFinished

Replace <SITE_ID> with the site on which the sound was played.

Payload

Key

Type

Description

id

Optional String

Request identifier for the request passed from

hermes/audioServer/<SITE_ID>/playBytes/<REQUEST_ID>

siteId

String

The id of the site on which the sound was played

sessionId

Optional String

The id of the session, if there is an active session

mosquitto_sub -h <HOSTNAME> -t 'hermes/audioServer/default/playFinished'

This feature is not available yet on Android.

This feature is not available yet on iOS.

Being notified when a sound frame is played

Every time the platform plays an audio frame, it publishes a message as follows.

MQTT
Android
iOS

Topic (subscribe)

hermes/audioServer/<SITE_ID>/audioFrame/<OPTIONAL_REQUEST_ID>

Replace <SITE_ID> with the site on which the sound frame is played, and <REQUEST_ID> with an id of the request passed from hermes/audioServer/<SITE_ID>/playBytes/<REQUEST_ID> (see above).

Binay Payload

The WAV of the frame.

Example

mosquitto_sub -h <HOSTNAME> -t 'hermes/audioServer/default/playFinished'

This feature is not available yet on Android.

This feature is not available yet on iOS.

Feedback

The Snips Platform offers built-in functionality for handling feedback, such as notification sounds. These can be explicitly turned on or off, as described here.

Turning on notification sounds

Notification sounds are used for instance when a wake word is detected. They can be turned on as follows.

MQTT
Android
iOS

Topic (publish)

hermes/feedback/sound/toggleOn

Payload

Key

Type

Description

siteId

String

The id of the site on which sound feedback should be turned on

Example

mosquitto_pub -h <HOSTNAME> \
-t 'hermes/feedback/sound/toggleOn' \
-m '{"siteId": "bedroom"}'

This feature is not available yet on Android.

This feature is not available yet on iOS.

Turning off notification sounds

Notification sounds are used for instance when a wake word is detected. They can be turned off as follows.

MQTT

Topic (publish)

hermes/feedback/sound/toggleOff

Payload

Key

Type

Description

siteId

String

The id of the site on which sound feedback should be turned off

Example

mosquitto_pub -h <HOSTNAME> \
-t 'hermes/feedback/sound/toggleOff' \
-m '{"siteId": "bedroom"}'