Hermes Reference
Messages on the Snips Platform are passed along as MQTT messages. On Android and iOS, these messages are passed as callbacks. In this document, we specify the format of the messages, and how they flow across the various components of the Snips Platform.
To illustrate the MQTT API below, we use the Eclipse Mosquitto Client for publishing messages and subscribing to topics over MQTT.

Wake Word

The Wake Word component is in charge of detecting when a wake word has been detected, and where it comes from (in case of a multi-satellite setup).

Activating the Wake Word component

This will activate the Wake Word component of the Snips Platform.
MQTT
Android
iOS
Topic (publish)
hermes/hotword/toggleOn
Payload
Key
Type
Description
siteId
String
The id of the site where the wake word detector should be turned on
Example
1
mosquitto_pub -h <HOSTNAME> -t 'hermes/hotword/toggleOn' \
2
-m '{"siteId": "default"}'
Copied!
This feature is not available yet on Android.
This feature is not available yet on iOS.

Deactivating the Wake Word component

This will deactivate the Wake Word component of the Snips Platform. Consequently, nothing will be triggered when a wake word is pronounced.
MQTT
Android
iOS
Topic (publish)
hermes/hotword/toggleOff
Payload
Key
Type
Description
siteId
String
The id of the site where the wake word detector should be turned on
sessionId
Optional String
The id of the session, if there is an active session
Example
1
mosquitto_pub -h <HOSTNAME> -t 'hermes/hotword/toggleOff' \
2
-m '{"siteId": "default"}'
Copied!
This feature is not available yet on Android.
This feature is not available yet on iOS.

Detecting a wake word

This message will be sent by the Snips Platform when the Wake Word component has detected that a specific wake word has been uttered.
MQTT
Android
iOS
Topic (subscribe)
hermes/hotword/<WAKEWORD_ID>/detected
Replace <WAKEWORD_ID> with the id of your wake word if multiple wake words are present, or default if there is only one wake word.
Payload
Key
Type
Description
siteId
String
The id of the site where the wake word detector should be turned on
modelId
String
The id of the model that triggered the wake word
modelVersion
String
The version of the model
modelType
String
The type of the model. Possible values: universal or personal
currentSensitivity
Float
The sensitivity configured in the model at the time of the detection
Example
1
mosquitto_sub -h <HOSTNAME> -t 'hermes/hotword/default/detected'
Copied!
This feature is not available yet on Android.
The event will trigger the SnipsPlatform.onHotwordDetected closure.
Example
1
let snips = SnipsPlatform(...)
2
snips.onHotwordDetected = { [weak self] in
3
// Handler code
4
}
Copied!

Automatic Speech Recognition (ASR)

The ASR component receives raw audio input and transcribes it into text.

Activating the ASR component

This will activate the ASR component, subsequently enabling to start listening for voice (using the startListening call described below).
MQTT
Android
iOS
Topic (publish)
hermes/asr/toggleOn
Example
1
mosquitto_pub -h <HOSTNAME> -t 'hermes/asr/toggleOn'
Copied!
This feature is not available yet on Android.
This feature is not available yet on iOS.

Deactivating the ASR component

This will deactivate the ASR component.
MQTT
Android
iOS
Topic (publish)
hermes/asr/toggleOff
Example
1
mosquitto_pub -h <HOSTNAME> -t 'hermes/asr/toggleOff'
Copied!
This feature is not available yet on Android.
This feature is not available yet on iOS.

Telling the ASR component to start listening

This will explicitly tell the ASR component to start listening for voice input.
MQTT
Android
iOS
Topic (publish)
hermes/asr/startListening
Payload
Key
Type
Description
siteId
String
The id of the site where the ASR should start listening
sessionId
Optional String
The id of the session, if there is an active session
Example
1
mosquitto_pub -h <HOSTNAME> -t 'hermes/asr/startListening' \
2
-m '{"siteId": "default"}'
Copied!
This feature is not available yet on Android.
This feature is not available yet on iOS.

Telling the ASR component to stop listening

This will explicitly tell the ASR component to stop listening for voice input.
MQTT
Android
iOS
Topic (publish)
hermes/asr/stopListening
Payload
Key
Type
Description
siteId
String
The id of the site where the ASR should stop listening
sessionId
Optional String
The id of the session, if there is an active session
Example
1
mosquitto_pub -h <HOSTNAME> -t 'hermes/asr/stopListening' \
2
-m '{"siteId": "default"}'
Copied!
This feature is not available yet on Android.
This feature is not available yet on iOS.

Obtaining intermediate ASR transcription results

When the ASR is listening, it transcribes voice to text in real time. This process stops when a longer period of silence is detected. Before it stops however, intermediate transcription results can be obtained, as described here.
MQTT
Android
iOS
In order to enable partial ASR transcriptions, you will need to start the platform with the partial flag set to true in the [snips-asr] section of /etc/snips.toml:
[snips-asr] partial=true
For more information, see Platform Configuration.
Topic (subscribe)
hermes/asr/partialTextCaptured
Payload
Key
Type
Description
text
String
The text captured
likelihood
Float
The likelihood of the capture
seconds
Float
The duration it took to do the processing
siteId
String
The id of the site where the text was captured
sessionId
Optional String
The id of the session, if there is an active session
Example
1
mosquitto_sub -h <HOSTNAME> -t 'hermes/asr/partialTextCaptured'
Copied!
This feature is not available yet on Android.
This feature is not available yet on iOS.

Obtaining full ASR transcription results

When the ASR is listening, it transcribes voice to text in real time. This process stops when a longer period of silence is detected, after which the transcription results are posted, as described here.
MQTT
Android
iOS
Topic (subscribe)
hermes/asr/textCaptured
Payload
Key
Type
Description
text
String
The text captured
likelihood
Float
The likelihood of the capture
seconds
Float
The duration it took to do the processing
siteId
String
The id of the site where the text was captured
sessionId
Optional String
The id of the session, if there is an active session
Example
1
mosquitto_sub -h <HOSTNAME> -t 'hermes/asr/textCaptured'
Copied!
This feature is not available yet on Android.
This feature is not available yet on iOS.

Natural Language Understanding (NLU)

Sending text to the NLU component

In order to extract an intent and slots from a piece of text, send it directly to the NLU component as follows. The NLU component will subsequently publish a message to the hermes/nlu/intentParsed topic, described below.
MQTT
Android
iOS
Topic (publish)
hermes/nlu/query
Payload
Key
Type
Description
input
String
The text to send to the NLU component
intentFilter
Optional Array of String
A list of intent names to restrict the NLU resolution on
id
Optional String
A request identifier. If provided, it will be passed back in the response on hermes/nlu/intentParsed or hermes/nlu/intentNotRecognized
sessionId
String
The id of the session, if there is an active session
Example
1
mosquitto_pub -h <HOSTNAME> -t 'hermes/nlu/query' \
2
-m '{"input": "What is the weather going to be in Paris tomorrow", "intentFilter": "myGetWeatherForecastIntent"}'
Copied!
This feature is not available yet on Android.
This feature is not available yet on iOS.

Sending text to the NLU component for slot detection

In some cases, for instance if a required slot is missing from an intent, a follow up query can be performed to extract a specific slot given an intent. This is done as follows.
MQTT
Android
iOS
Topic (publish)
hermes/nlu/partialQuery
Payload
Key
Type
Description
input
String
The text to run the slot detection on
intentName
String
The intent to use when doing the slot detection
slotName
String
The slot to search for
id
Optional String
A request identifier. If provided, it will be passed back in the response on hermes/nlu/slotParsed
sessionId
String
The id of the session, if there is an active session
Example
1
mosquitto_pub -h <HOSTNAME> -t 'hermes/nlu/partialQuery' \
2
-m '{"input": "In Paris", "intentName": "myGetWeatherForecastIntent", "slotName": "location"}'
Copied!
This feature is not available yet on Android.
This feature is not available yet on iOS.

Obtaining the result of an NLU parsing (low-level API)

When text has been sent to the NLU (see Sending text to the NLU component), the result of the intent resolution is sent back as follows.
MQTT
Android
iOS
Topic (subscribe)
hermes/nlu/intentParsed
Note that this is a low-level API, and it is not recommended to be used of production. In particular, this API method does not guarantee that all slots of the intent have been properly parsed. To detect an intent parsed by the NLU component, it is recommended to subscribe to the following topic instead:
hermes/intent/<INTENT_NAME>
where INTENT_NAME is the name of the intent that has been parsed. See below.
Payload
Key
Type
Description
id
Optional String
Request identifier for the request passed from hermes/nlu/query
input
String
The input that was processed
intent
JSON Object
Structured description of the intent classification
slots
Optional Array of JSON Objects
Structured description of the slots for the detected intent, if any
sessionId
String
The id of the session, if there is an active session
Example
1
mosquitto_sub -h <HOSTNAME> -t 'hermes/nlu/intentParsed'
Copied!
This feature is not available yet on Android.
This feature is not available yet on iOS.

Obtaining the result of an NLU slot detection

When text has been sent to the NLU for slot parsing (see Sending text to the NLU component for slot detection), the result of the slot detection is sent back as follows.
MQTT
Topic (subscribe)
hermes/nlu/slotParsed
Payload
Key
Type
Description
id
Optional String
Request identifier for the request passed from hermes/nlu/partialQuery
input
String
The input that was processed
intentName
String
The intent used to find the slot
slot
Optional JSON Object
The resulting slot, if found
sessionId
String
The id of the session, if there is an active session
Example
1
mosquitto_sub -h <HOSTNAME> -t 'hermes/nlu/slotParsed'
Copied!

Being notified when an intent was not recognised

When the NLU was unable to parse a chunk of text, it publishes a message telling so.
MQTT
Android
iOS
Topic (subscribe)
hermes/nlu/intentNotRecognized
Payload
Key
Type
Description
id
Optional String
Request identifier for the request passed from hermes/nlu/query
input
String
The input that was processed, but didn't match any intent
sessionId
String
The id of the session, if there is an active session
Example
1
mosquitto_sub -h <HOSTNAME> -t 'hermes/nlu/intentNotRecognized'
Copied!
This feature is not available yet on Android.
This feature is not available yet on iOS.

Being notified when an error has occurred

When an error has occurred in the NLU component, it publishes a message telling so.
MQTT
Android
iOS
Topic (publish)
hermes/error/nlu
Payload
Key
Type
Description
sessionId
Optional String
The id of the session, if there is an active session
error
String
A description of the error that occurred
context
Optional String
Additional information on the context in which the error occurred
Example
1
mosquitto_sub -h <HOSTNAME> -t 'hermes/error/nlu'
Copied!
This feature is not available yet on Android.
This feature is not available yet on iOS.

Dialogue Manager

Text-to-Speech (TTS)

Sending text to be spoken by the TTS component (low-level API)

The Snips Platform comes with a Text-to-Speech component that allows text to be spoken.
MQTT
Android
iOS
Topic (publish)
hermes/tts/say
Note that this is a low-level API, and it is not recommended to be used of production. You should use the following topics instead, based on the dialogue manager:
    hermes/dialogueManager/startSession
    hermes/dialogueManager/continueSession
See the Dialogue API Reference for further explanations.
Payload
Key
Type
Description
text
String
The text to be spoken
lang
Optional String
The language code to use when saying the text. If nothing is provided, en_GB will be used
id
Optional String
A request identifier. If provided, it will be passed back in the response on hermes/tts/sayFinished
siteId
String
The id of the site where the text should be spoken
sessionId
Optional String
The id of the session, if there is an active session
Example
1
mosquitto_pub -h <HOSTNAME> -t 'hermes/tts/say' \
2
-m '{"text": "Bonjour!", "lang": "fr_FR"}'
Copied!
This feature is not available yet on Android.
This feature is not available yet on iOS.

Being notified when TTS has finished speaking some text

When TTS has finished speaking some text, it will publish a message as follows.
MQTT
Android
iOS
Topic (subscribe)
hermes/tts/sayFinished
Payload
Key
Type
Description
id
Optional String
Request identifier for the request passed from hermes/tts/say
sessionId
Optional String
The id of the session, if there is an active session
Example
1
mosquitto_sub -h <HOSTNAME> -t 'hermes/tts/sayFinished'
Copied!
This feature is not available yet on Android.
This feature is not available yet on iOS.

Audio Server

The Snips Platform comes with an audio server to easily handle playing sound on different sites.

Playing a WAV sound

You may send a WAV sound to be played on a specific site as follows.
MQTT
Android
iOS
Topic (publish)
hermes/audioServer/<SITE_ID>/playBytes/<REQUEST_ID>
Replace <SITE_ID> with the site on which to play the sound (e.g. default), and <REQUEST_ID> with an id to be passed back on hermes/audioServer/<SITE_ID>/playFinished (see below).
Binary Payload
The WAV file to play.
Example
1
mosquitto_pub -h <HOSTNAME> \
2
-t 'hermes/audioServer/default/playBytes/8ewnjksdf093jb42' \
3
-f sound.wav
Copied!
This feature is not available yet on Android.
This feature is not available yet on iOS.

Being notified when sound has finished playing

When the audio service has finished playing a sound, it publishes a message as follows.
MQTT
Android
iOS
Topic (subscribe)
hermes/audioServer/<SITE_ID>/playFinished
Replace <SITE_ID> with the site on which the sound was played.
Payload
Key
Type
Description
id
Optional String
Request identifier for the request passed from
hermes/audioServer/<SITE_ID>/playBytes/<REQUEST_ID>
sessionId
Optional String
The id of the session, if there is an active session
1
mosquitto_sub -h <HOSTNAME> -t 'hermes/audioServer/default/playFinished'
Copied!
This feature is not available yet on Android.
This feature is not available yet on iOS.

Streaming a sound

You can also stream the sound you want to play instead of sending it all on one go. This is for example used by the TTS service to start playing the generated voice before the end of its generation.
MQTT
Android
iOS
Topic (publish)
hermes/audioServer/<SITE_ID>/playBytesStreaming/<REQUEST_ID>/<CHUNK_NUMBER>/<IS_LAST_CHUNK>
Replace <SITE_ID> with the site on which to play the sound (e.g. default), and <REQUEST_ID> with an id to be passed back on hermes/audioServer/<SITE_ID>/streamFinished (see below). <CHUNK_NUMBER> is the number of the send chunk (starting at 0). <IS_LAST_CHUNK> should be 0, except for the last chunk where it should be 1.
Binary Payload
The chunk, encoded as a WAV.
Example
1
mosquitto_pub -h <HOSTNAME> \
2
-t 'hermes/audioServer/default/playBytesStreaming/8ewnjksdf093jb42/0/0' \
3
-f sound0.wav && \
4
mosquitto_pub -h <HOSTNAME> \
5
-t 'hermes/audioServer/default/playBytesStreaming/8ewnjksdf093jb42/1/0' \
6
-f sound1.wav && \
7
...
8
mosquitto_pub -h <HOSTNAME> \
9
-t 'hermes/audioServer/default/playBytesStreaming/8ewnjksdf093jb42/123/1' \
10
-f sound123.wav
Copied!
This feature is not available yet on Android.
This feature is not available yet on iOS.

Being notified when stream has finished

When the audio service has finished streaming a sound, it publishes a message as follows.
MQTT
Android
iOS
Topic (subscribe)
hermes/audioServer/<SITE_ID>/streamFinished
Replace <SITE_ID> with the site on which the sound was streamed.
Payload
Key
Type
Description
id
String
The id of stream that finished
siteId
String
The id of the site on which the sound was streamed
1
mosquitto_sub -h <HOSTNAME> -t 'hermes/audioServer/default/streamFinished'
Copied!
This feature is not available yet on Android.
This feature is not available yet on iOS.

Being notified when a sound frame is captured

Every time the platform captures an audio frame, it publishes a message as follows.
MQTT
Android
iOS
Topic (subscribe)
hermes/audioServer/<SITE_ID>/audioFrame
Replace <SITE_ID> with the site on which the sound frame was captured
Binay Payload
The WAV of the frame.
Example
1
mosquitto_sub -h <HOSTNAME> -t 'hermes/audioServer/default/playFinished'
Copied!
This feature is not available yet on Android.
This feature is not available yet on iOS.

Feedback

The Snips Platform offers built-in functionality for handling feedback, such as notification sounds. These can be explicitly turned on or off, as described here.

Turning on notification sounds

Notification sounds are used for instance when a wake word is detected. They can be turned on as follows.
MQTT
Android
iOS
Topic (publish)
hermes/feedback/sound/toggleOn
Payload
Key
Type
Description
siteId
String
The id of the site on which sound feedback should be turned on
Example
1
mosquitto_pub -h <HOSTNAME> \
2
-t 'hermes/feedback/sound/toggleOn' \
3
-m '{"siteId": "bedroom"}'
Copied!
This feature is not available yet on Android.
This feature is not available yet on iOS.

Turning off notification sounds

Notification sounds are used for instance when a wake word is detected. They can be turned off as follows.
MQTT
Topic (publish)
hermes/feedback/sound/toggleOff
Payload
Key
Type
Description
siteId
String
The id of the site on which sound feedback should be turned off
Example
1
mosquitto_pub -h <HOSTNAME> \
2
-t 'hermes/feedback/sound/toggleOff' \
3
-m '{"siteId": "bedroom"}'
Copied!

Entities injection

Entities injection allows you to update both the ASR and the NLU models directly on the device.

Request injection

Each intent within an assistant may contain some slots, and each slot has a specific type that we call an entity. If you have a contact_name entity that contains a list of contacts in an address book, Entities Injection lets you add new contact names to this list.
For a more in-depth explanation of how injection works, check the documentation

Available operations

Two injection operations are currently supported: add and addFromVanilla. Other operations are under development.
    add adds the list of values that you provide to the existing entity values.
    addFromVanilla removes all the previously injected values to the entity, and then, adds the list of values provided. Note that the entity values coming from the console will be kept.
Let's illustrate this with an entity having two values: one and two. Here is how the entity values will be affected after performing some injection operations:
OperationKind
Values to inject
Supported entity values
one, two
add
three
one, two, three
add
four
one, two, three, four
addFromVanilla
five
one, two, five
add
six
one, two, five, six
The entity values to inject are specified in a JSON file which must respect the following format:
    A key operations mapping to a list of operations
    Each operation is a tuple (OperationKind, OperationData)
    OperationKind is the type of injections to perform. See the Available operations section for the allowed operations.
    OperationData is a dictionary mapping an entity name to a list of values (strings).
For instance, if you are willing to add "The Wolf of Wall Street" to your list of films, just write the following file:
MQTT
Android
iOS
Topic (publish)
hermes/injection/perform
Payload
Key
Type
Description
id
Optional String
Request identifier for the request
crossLanguage
Optional String
Language for cross-language G2P
lexicon
Optional Array of (Value, Array of Pronunciation)
List of pre-computed prononciations to add in a model
operations
Array of (InjectionKind, Array of (Entity, Array of EntityValue))
List of pre-computed prononciations to add in a model
Example
1
{
2
"operations": [
3
[
4
"add",
5
{
6
"films" : [
7
"The Wolf of Wall Street",
8
"The Lord of the Rings"
9
]
10
}
11
]
12
]
13
}
Copied!
1
mosquitto_pub -h <HOSTNAME> \
2
-t 'hermes/injection/perform'
3
-f operations.json
Copied!
You should send a InjectionRequestMessage message using requestInjection on SnipsPlatformClient.
1
fun SnipsPlatformClient.requestInjection(injectionRequestMessage: InjectionRequestMessage)
2
3
// Usage
4
val snipsPlatformClient = SnipsPlatformClient.Builder(...)
5
val operations = listOf(InjectionOperation(InjectionKind.AddFromVanilla, mutableMapOf(Pair("locality", listOf("wonderland")))))
6
val lexicon = mutableMapOf<String, List<String>>()
7
val request = InjectionRequestMessage(operations, lexicon, null, null)
8
9
snipsPlatformClient.requestInjection(request)
Copied!
1
func requestInjection(with message: InjectionRequestMessage) throws
2
3
// Usage
4
let snips = SnipsPlatform(...)
5
let operation = InjectionRequestOperation(entities: ["locality": ["wonderland"], "region": ["wonderland"]], kind: .add)
6
try! snips.requestInjection(with: InjectionRequestMessage(operations: [operation]))
Copied!

Injection complete

Once the injection request has been processed, the ASR and NLU are reloaded. Once reloaded, Snips Platform will post on this route.
Topic (subscribe)
hermes/injection/complete
Payload
Key
Type
Description
requestId
Optional String
The id of the InjectionRequestMessage
Example
1
mosquitto_sub -h <HOSTNAME> -t 'hermes/injection/complete'
Copied!
You should register a listener using setOnInjectionComplete on SnipsPlatformClient.
You can write your closure in the onInjectionComplete property on SnipsPlatform client.
1
let snips = SnipsPlatform(...)
2
snips.onInjectionComplete = { message in
3
// Your code here
4
}
Copied!

Injection Reset

Injection reset will delete previously injected entities and reboot the ASR & NLU. You don't need to relaunch the platform. Once the injection is complete, it will post a message on hermes.
Topic (publish)
hermes/injection/reset/perform
Payload
Key
Type
Description
requestId
Optional String
The id of the request
Example
1
mosquitto_pub -h <HOSTNAME> -t 'hermes/injection/reset/perform'
Copied!
You should send a InjectionResetRequestMessage message using requestInjectionReset on SnipsPlatformClient.
1
fun SnipsPlatformClient.requestInjectionReset(injectionResetRequestMessage: InjectionResetRequestMessage)
Copied!
1
func requestInjectionReset(with message: InjectionResetRequestMessage = InjectionResetRequestMessage()) throws
2
3
// Usage
4
let snips = SnipsPlatform(...)
5
snips.requestInjectionReset()
Copied!

Injection Reset Complete

Once injection reset is finished, it will reboot both ASR & NLU services and post on this route.
Topic (subscribe)
hermes/injection/reset/complete
Payload
Key
Type
Description
requestId
Optional String
The id of the request
Example
1
mosquitto_sub -h <HOSTNAME> -t 'hermes/injection/reset/complete'
Copied!
You should register a listener using setOnInjectionResetComplete on SnipsPlatformClient.
You can write your closure in the onInjectionResetComplete property on SnipsPlatform client.
1
let snips = SnipsPlatform(...)
2
snips.onInjectionResetComplete = { request in
3
// Your code here
4
}
Copied!
Last modified 2yr ago