Dynamic Vocabulary using Entities Injection

Presentation

The vocabulary supported by the ASR and the NLU is limited to the words it has seen at training time (i.e. words that were present in your assistant when it was downloaded). For various reasons, you may want to inject new values to your assistant without having to go through the console. Sensitive information, such as contact names, or other personal details that you want your assistant to understand, are better kept private. It can also be tedious to download again an assistant each time you want to update its vocabulary.

For these reasons, we are introducing a new feature, Entities Injection, which allows you to update both the ASR and the NLU models directly on the device.

Each intent within an assistant may contain some slots, and each slot has a specific type that we call an entity. If you have a contact_name entity that contains a list of contacts in an address book, Entities Injection lets you add new contact names to this list.

Installation

On a Raspberry Pi

The entities injection binary has to be downloaded in addition to the Snips Platform. To do so, run the following command from your device:

sudo apt-get install -y snips-injection

You can check that the binary is running by using the command:

ps aux | grep snips-injection

On Android

You need to enable the injection when building your SnipsPlatfromClient. This can be done with:

new SnipsPlatformClient.Builder(assistantDir)
// add other options
// enable injection, disabled by default
.enableInjection(true)
.build();

Inject entity values

Available operations

Two injection operations are currently supported: add and addFromVanilla. Other operations are under development.

  • add adds the list of values that you provide to the existing entity values.

  • addFromVanilla removes all the previously injected values to the entity, and then, adds the list of values provided. Note that the entity values coming from the console will be kept.

Let's illustrate this with an entity having two values: one and two. Here is how the entity values will be affected after performing some injection operations:

OperationKind

Values to inject

Supported entity values

one, two

add

three

one, two, three

add

four

one, two, three, four

addFromVanilla

five

one, two, five

add

six

one, two, five, six

On a Raspberry Pi

The entity values to inject are specified in a JSON file which must respect the following format:

  • A key operations mapping to a list of operations

  • Each operation is a tuple (OperationKind, OperationData)

  • OperationKind is the type of injections to perform. See the Available operations section for the allowed operations.

  • OperationData is a dictionary mapping an entity name to a list of values (strings).

For instance, if you are willing to add "The Wolf of Wall Street" to your list of films, just write the following file:

{
"operations": [
[
"add",
{
"films" : [
"The Wolf of Wall Street",
"The Lord of the Rings"
]
}
]
]
}

Note the difference between slot name and entity name. In the injections file, you must specify the names of the entities you used in the console. Else, it won't work.

Then, the file can be sent on the MQTT bus that will trigger the injections. For example, (if injections.json is your JSON file):

$ mosquitto_pub -t hermes/injection/perform -f injections.json

When dealing with a big assistant, the injection process can take some time. This is a first version. A quicker version is under development.

The injections are only supported with the latest version of the ASR and NLU models downloaded from the console. If you get some MissingFileError, it would probably mean that you have to download a new assistant

On Android

Entity injection is performed thanks to the requestInjection method of the SnipsPlatformClient. The object you need to pass follows the same structure as the JSON sent on MQTT:

// inject new values in the "house_room" entity
HashMap<String, List<String>> values = new HashMap<>();
values.put("house_room", Arrays.asList("bunker", "batcave"));
client.requestInjection(new InjectionRequestMessage(
Collections.singletonList(new InjectionOperation(InjectionKind.Add, values)),
new HashMap<String, List<String>>()));

On iOS

First, you'll need to add the injection g2p data to your project. Download it from https://resources.snips.ai/injection/g2p-0.60.11-28675e9.zip.

Unzip the file and drag and drop it inside your Xcode project like this:

Copy items if needed and add the folder references to your project.

You'll then need to start the platform with the enableInjection set to true and the path to the g2p folder:

let g2pResources = Bundle.main.url(forResource:
"snips-g2p-resources", withExtension: nil)!
let assistantURL = Bundle.main.url(forResource:
"assistant", withExtension: nil)!
let snips = try SnipsPlatform(assistantURL: assistantURL,
hotwordSensitivity: 0.5,
enableHtml: false,
enableLogs: true,
enableInjection: true,
userURL: nil,
g2pResources: g2pResources)

When injecting new entity values, make sure you reference an existing entity, otherwise it won't work.

For exemple, let's add a new value to the locality entity in the Snips Weather app:

var newEntities: [String: [String]] = [:]
newEntities["locality"] = ["wonderland"]
let operation = InjectionRequestOperation(entities: newEntities, kind: .add)
do {
try self?.snips?.requestInjection(with: InjectionRequestMessage(operations: [operation]))
} catch let error {
print("Injection failed, reason: \(error.localizedDescription)")
}

At the end of the injection process, the ASR and NLU models will be reloaded. If everything went well, you will be able to speak to your assistant and it should understand the new values you just added.

Now, the Snips platform will be able to understand properly the following input:

What's the weather like in wonderland?