Code actions

Last updated 2 months ago

Foreword

When a user talks to a voice assistant, actions are the last step of the interaction. Once the voice assistant has processed and understood the intention of the user, the action code part will react to the query and execute the action. The action code is critical in providing a great user experience!

In the app store, you can find Apps that already have actions associated to them. They are marked with a little fork, on the top left corner:

A Weather app that already has actions in it

In the absence of this icon, you will need to write your own action.

An action can be written in many programming languages (Javascript, Python, Rust, C...). The only prerequisite is to be able to connect to MQTT. The MQTT protocol provides a lightweight method of carrying out messaging using a publish/subscribe model. This makes it suitable for Internet of Things messaging between low power sensors or mobile devices such as phones, embedded computers or microcontrollers.

You can imagine MQTT as a chat room. As a user, you connect to a chat room created, you can subscribe to specific topics to filter out messages. You can send and receive messages from other users on specific topics.

In our case, users are Snips Platform's components, like the Hotword, Automatic Speech Recognition or Text To Speech These components are sending and receiving messages to each other on predefined topics. We call it the Hermes Protocol. You can learn more by reading the Hermes Protocol documentation.

Digging deeper with an example

Let's say you want to control your Philips Hue at home. You create an assistant on the console, and the Smart Lights app available on the app store. The Smart Lights app has already been trained to understand simple light intents, for exampleHey Snips! Turn on the lights in the kitchen

Once the Snips Platform has processed the audio and recognized the intent, it will send the intent on Hermes, precisely on the topic hermes/intent/lightSet The message is sent in a JSON format and looks like this:

{
"sessionId": "78ca555b-9027-4dcd-b3c1-70d38d62624c",
"customData": null,
"siteId": "default",
"input": "turn on the lights in the kitchen",
"intent": {
"intentName": "lightsSet",
"probability": 0.9413356
},
"slots": [
{
"rawValue": "kitchen",
"value": {
"kind": "Custom",
"value": "kitchen"
},
"range": {
"start": 26, "end": 33
},
"entity": "house_room",
"slotName": "house_room"
}
]
}

You can find more details on this JSON response in the Hermes protocol section.

The goal for you, action developer, is to do something with this JSON. In this example, you want to connect to a Philips Hue bridge, and effectively turn the kitchen lights on.

Writing your own actions

When editing an app, within the action tab, the console gives you four options to build your action:

  • None: if there is no action, nothing will be happen after the intent is detected.

  • Code Snippets: you can write Python code snippets, directly from the console. It is meant for quick testing, or simple interactions. See here for more information.

  • ****Your own action code, published on Github: to build rich interactions, you can take full control of the action code. See this section to get the basics on how to connect to MQTT directly, or how to use our helpers: the Snips apps server and Hermes Python. If you publish your code on Github, you can point to it from the app in the console, so anyone using your app can easily leverage your work.

  • Home Assistant: there are numerous components available on Home Assistant. You can use this option to link your app to one of them. You will need to write code snippets that will call Home Assistant components with Snips' intents and slot values. **See here for more information.