iOS/Unity AR Game

An example of how we can implement Snips with Unity using iOS ARKit

Introduction

We don't have a plugin for Unity yet, so we are going to use the SnipsPlatform framework from iOS and communicate between Unity and iOS.

The app is actually composed of two entities:

  • The game, built on Unity

  • The Snips part built on iOS

We are going to build an augmented reality character that will jump by voice on an iOS device.

Prerequisites

  • Unity 2018.3.9.f1 or later

  • Xcode 10.2.1 or later

Create a project in Unity

  • Create a project in Unity

  • Change the build platform to iOS

  • In Player Settings, change the following:

    • Change the bundle identifier

    • Change the build number

    • Tick "Requires ARKit support"

    • Under "Camera usage description" put: "Required for AR support"

    • Under "Microphone usage description" put: "Required for Snips support"

    • Change the target minimum version to: 11.0

    • Modify the Architecture to: Arm64

Make it AR compatible

  • Open the package manager (Window > Package Manager)

  • We need to display preview packages (Advanced > Show Preview Packages)

  • Search and install: AR Foundation and ARKit XR Plugin

Import our packages in Unity

First, download our Unity package:

Then, after importing the package you will see two prefabs in Assets > SnipsARDemo > Prefabs :

  • ObjectToPlace : This is the object that will be displayed in augmented reality, it is composed of a collider that act as a floor, a character

  • Snips_AR : This is the object that handles the augmented reality part. It also has the Snips indicator UI for wake word detection and intent recognition indication.

Please drag and drop Snips_AR into the Scene and place it at position (0,0,0). Our scene should look like below :

The scene is composed of a Light and Snips_AR GameObject

A script is attached to ObjetToPlace > RPG-Character named GameController.cs. This script has a public method called char_jump. Later we will use a Snips action code to trigger this method from iOS.

Build it

Now you can build for iOS. At this point we don't have anything related to the Snips Platform. If everything goes well, you can use your iOS device to scan a plane surface. As soon as the device detects a plan, it will pop a UI indicator with an arrow. At this point you can touch the display to display our augmented reality character.

Let's build our assistant

Please head on to the Snips Console and let's build our very simple assistant.

  1. Create a new assistant, name it whatever you want, and select a language

  2. Add an app and create a new app. We are going to name it for example : Char_control

  3. Edit the app and create an intent called character_jump (this name will be used later in the code).

  4. Populate the training examples (for example "jump")

  5. Save it and deploy the assistant (download it and unzip it)

Please note that when you create an intent name, the intent name that will be used is displayed as username:intentName

Install SnipsPlatform in your Xcode Unity project

Now we are going to install and bridge the SnipsPlatform. Please import the SnipsPlatform project by using the submodule method of the platform

  1. Add the snips-platform-swift repository as a submodule of your application’s repository. https://github.com/snipsco/snips-platform-swift

  2. Drag and drop SnipsPlatform.xcodeproj into your application’s Xcode project or workspace.

  3. On the “General” tab of your application target’s settings, add SnipsPlatform.framework to the “Embedded Binaries” section.

Importing SnipsPlatform as a submodule

Make sure that you are adding the one for iOS

Import the assistant in the project

  1. Copy the assistant folder into your Xcode project folder

  2. Drag and drop the assistant folder into your Xcode project

Add the folder and ensure that "Create folder references" is checked
The project should look like this

Configuration of the build

  • On the target of the main project, Unity-iPhone , we need to change the build settings. Set Enable bitcode to No.

  • Add a new configuration under SnipsPlatform.xcodeproj > Project > Info > Configurations by duplicating the "Release" Configuration and name it ReleaseForRunning

Create bridging files and adding a custom controller

  • Create a new Swift file named AppControllerV.swift in the Classes folder

  • Create the associated bridging header Unity-iPhone-Bridging-Header.h

  • Create a header file named UnityUtils.h in the Classes folder

  • Add the following:

Unity-iPhone-Bridging-Header.h
#import <UIKit/UIKit.h>
#import "UnityAppController.h"
#import "UnityInterface.h"
#import "UnityUtils.h"
UnityUtils.h
#ifndef UnityUtils_h
#define UnityUtils_h
void unity_init(int argc, char* argv[]);
void UnityPostMessage(NSString* gameObject, NSString* methodName, NSString* message);
#endif /* UnityUtils_h */
AppControllerV.swift
import Foundation
import AVFoundation
import SnipsPlatform
import AudioToolbox
@objc(AppControllerV) class AppControllerV:UnityAppController{
var snips: SnipsPlatform? = nil
fileprivate var audioEngine: AVAudioEngine? = nil
fileprivate lazy var logView = UITextView()
let vc = UIViewController()
let url = Bundle.main.url(forResource: "assistant", withExtension: nil)!
var sessionId:String;
var messageId:String;
var intentName:String;
override func startUnity(_ application: UIApplication!) {
super.startUnity(application)
var frame = self.window.bounds
vc.view.frame = frame
logView.font = UIFont.systemFont(ofSize: 16)
logView.frame = vc.view.frame
logView.isUserInteractionEnabled = false
snips = try! SnipsPlatform(assistantURL: url)
audioEngine = try! AppControllerV.createAudioEngine(with: snips!)
setupHandlers()
do {
NSLog("***START SNIPS****")
try snips?.start()
try audioEngine?.start()
NSLog("snips and audioengine loaded")
} catch let e as SnipsPlatformError {
print("Snips error: \(e)")
} catch {
print("Unexpected error: \(error)")
}
}
override init() {
self.sessionId = ""
self.messageId = ""
self.intentName = ""
super.init()
}
required init?(coder aDecoder: NSCoder) {
fatalError("init(coder:) has not been implemented")
}
fileprivate class func createAudioEngine(with snips: SnipsPlatform) throws -> AVAudioEngine {
print("creating audio engine")
let audioEngine = AVAudioEngine()
let audioSession = AVAudioSession.sharedInstance()
try audioSession.setCategory(.playAndRecord, mode: .measurement, options: [.mixWithOthers, .allowBluetoothA2DP, .allowBluetooth])
try audioSession.setPreferredSampleRate(16_000)
try audioSession.setActive(true, options: .notifyOthersOnDeactivation)
let recordingFormat = AVAudioFormat(commonFormat: .pcmFormatInt16, sampleRate: 16_000, channels: 1, interleaved: true)
let input = audioEngine.inputNode
let downMixer = AVAudioMixerNode()
audioEngine.attach(downMixer)
audioEngine.connect(input, to: downMixer, format: nil)
downMixer.installTap(onBus: 0, bufferSize: 1024, format: recordingFormat) { (buffer, time) in
try! snips.appendBuffer(buffer)
}
audioEngine.prepare()
try audioSession.overrideOutputAudioPort(AVAudioSession.PortOverride.speaker)
return audioEngine
}
fileprivate func setupHandlers() {
snips?.onIntentDetected = { [weak self] intent in
DispatchQueue.main.sync {
self?.logView.text = String(format:
"Query: %@\n" +
"Intent: %@\n" +
"Probability: %.3f\n" +
"Slots:\n\t%@",
intent.input,
intent.intent.intentName ,
intent.intent.confidenceScore ,
intent.slots.map { "\($0.slotName): \($0.value)" }.joined(separator: "\n\t")
)
UnitySendMessage("RPG-Character", "intentRecognised", "");
if intent.slots.count > 0 {
// Example: handle intent with slots (example)
/*
if(intent.intent.intentName == "YourUsername:jump"){
if let value = intent.slots[0].value.value as? String, value == "forward" {
UnitySendMessage("ObjectToPlace", "jump", "forward");
}else if let value = intent.slots[0].value.value as? String, value == "backward" {
UnitySendMessage("ObjectToPlace", "jump", "backward");
}
try! self?.snips?.endSession(sessionId:self!.sessionId)
}
*/
} else {
if intent.intent.intentName == "username:character_jump"{
UnitySendMessage("RPG-Character", "char_jump", "");
try! self?.snips?.endSession(sessionId:self!.sessionId)
}
}
}
}
snips?.onHotwordDetected = { [weak self] in
DispatchQueue.main.sync {
UnitySendMessage("RPG-Character", "hotwordDetection", "true");
}
}
snips?.onSessionStartedHandler = {message in
print("message \(message.sessionId)");
self.sessionId = message.sessionId;
}
snips?.onIntentNotRecognizedHandler = {message in
DispatchQueue.main.sync {
}
}
snips?.onSessionEndedHandler = {message in
DispatchQueue.main.sync {
if(message.sessionTermination.terminationType == .intentNotRecognized){
UnitySendMessage("RPG-Character", "intentNotRecognised", "");
}
UnitySendMessage("RPG-Character", "hotwordListening", "true");
}
}
snips?.onListeningStateChanged = { [weak self] listening in
DispatchQueue.main.sync {
}
}
snips?.speechHandler = { message in
DispatchQueue.main.sync {
}
}
snips?.snipsWatchHandler = { message in
DispatchQueue.main.sync {
}
}
}
}
extension SlotValue {
var value: Any? {
get {
switch self {
case .custom(let value): return value
case .number(let value): return value
case .ordinal(let value): return value
case .instantTime(let value): return value.value
case .timeInterval(let value):
guard let fromString = value.from,
let toString = value.to,
let fromDate = CustomTimeInterval.dateFormatterISO8601.date(from: fromString),
let toDate = CustomTimeInterval.dateFormatterISO8601.date(from: toString)
else { return nil }
return CustomTimeInterval(from: fromDate, to: toDate)
case .amountOfMoney(let value): return value.value
case .temperature(let value): return Temperature(value: Int(value.value), unit: Temperature.parseUnit(value.unit))
case .duration(let value):
return Duration(years: value.years, quarters: value.quarters, months: value.months, weeks: value.weeks, days: value.days, hours: value.hours, minutes: value.minutes, seconds: value.seconds)
case .percentage(let value): return String(format:"%f",value)
case .musicAlbum(let value): return value
case .musicArtist(let value): return value
case .musicTrack(let value): return value
}
}
}
}
class CustomTimeInterval {
fileprivate static var dateParserFormatter: DateFormatter = {
let formatter = DateFormatter()
formatter.dateFormat = "yyyy-MM-dd HH:mm:ss"
return formatter
}()
fileprivate static var datePrinterFormatter: DateFormatter = {
let formatter = DateFormatter()
formatter.dateStyle = .short
formatter.timeStyle = .short
return formatter
}()
static var dateFormatterISO8601 = ISO8601DateFormatter()
let from: Date
let to: Date
init(from: Date, to: Date) {
self.from = from
self.to = to
}
}
extension CustomTimeInterval: CustomStringConvertible {
var description: String {
return String(format: "%@ - %@", CustomTimeInterval.datePrinterFormatter.string(from: from), CustomTimeInterval.datePrinterFormatter.string(from: to))
}
}
extension CustomTimeInterval {
static func parse(_ slotValue: [String: Any]) -> CustomTimeInterval? {
guard let kind = slotValue["kind"] as? String, kind.lowercased() == "timeinterval",
let from = parse(date: slotValue["from"]),
let to = parse(date: slotValue["to"])
else { return nil }
return CustomTimeInterval(from: from, to: to)
}
fileprivate static func parse(date: Any?) -> Date? {
guard let date = date as? String else { return nil }
var components = date.components(separatedBy: " ")
components.removeLast()
let dateString = components.joined(separator: " ")
return CustomTimeInterval.dateParserFormatter.date(from: dateString)
}
}
enum TemperatureUnit {
case celcius, fahrenheit, unspecified
}
class Temperature {
let value: Int
let unit: TemperatureUnit
init(value: Int = 0, unit: TemperatureUnit = .unspecified) {
self.value = value
self.unit = unit
}
init(value: Double = 0, unit: TemperatureUnit = .unspecified) {
self.value = Int(value)
self.unit = unit
}
}
extension Temperature: CustomStringConvertible {
var description: String {
let unitString = (unit == .celcius) ? "C" : ((unit == .fahrenheit) ? "F" : "")
return String(format: "%d°%@", value, unitString)
}
}
extension Temperature {
static func parse(_ slotValue: [String: Any]) -> Temperature? {
guard let kind = slotValue["kind"] as? String, kind.lowercased() == "temperature" else { return nil }
return Temperature(
value: (slotValue["value"] as? Int) ?? Int((slotValue["value"] as? Double) ?? 0),
unit: parseUnit((slotValue["unit"] as? String))
)
}
static func parseUnit(_ value: String?) -> TemperatureUnit {
guard let value = value else { return TemperatureUnit.unspecified }
let normalized = value.lowercased()
if normalized == "celsius" { return TemperatureUnit.celcius }
if normalized == "fahrenheit" { return TemperatureUnit.fahrenheit }
return TemperatureUnit.unspecified
}
}
enum DurationPrecision {
case exact
}
class Duration {
let years: Int
let quarters: Int
let months: Int
let weeks: Int
let days: Int
let hours: Int
let minutes: Int
let seconds: Int
let precision: DurationPrecision
init(years: Int = 0, quarters: Int = 0, months: Int = 0, weeks: Int = 0, days: Int = 0, hours: Int = 0, minutes: Int = 0, seconds: Int = 0, precision: DurationPrecision = .exact) {
self.years = years
self.quarters = quarters
self.months = months
self.weeks = weeks
self.days = days
self.hours = hours
self.minutes = minutes
self.seconds = seconds
self.precision = precision
}
}
extension Duration: CustomStringConvertible {
var description: String {
if years == 0 && quarters == 0 && months == 0 && weeks == 0 {
if days == 0 {
if hours == 0 {
return String(format: "%02d:%02d", minutes, seconds)
} else {
return String(format: "%02d:%02d:%02d", hours, minutes, seconds)
}
} else {
return String(format: "%02dd%02d:%02d:%02d", days, hours, minutes, seconds)
}
}
return String(format: "%02dy%02dm%02dw%02dd%02d:%02d:%02d", years, months, weeks, days, hours, minutes, seconds)
}
}
extension Duration {
static func parse(_ slotValue: [String: Any]) -> Duration? {
guard let kind = slotValue["kind"] as? String, kind == "Duration" else { return nil }
return Duration(
years: (slotValue["years"] as? Int) ?? 0,
quarters: (slotValue["quarters"] as? Int) ?? 0,
months: (slotValue["months"] as? Int) ?? 0,
weeks: (slotValue["weeks"] as? Int) ?? 0,
days: (slotValue["days"] as? Int) ?? 0,
hours: (slotValue["hours"] as? Int) ?? 0,
minutes: (slotValue["minutes"] as? Int) ?? 0,
seconds: (slotValue["seconds"] as? Int) ?? 0,
precision: parsePrecision((slotValue["precision"] as? String) ?? "exact")
)
}
static func parsePrecision(_ value: String) -> DurationPrecision {
let normalized = value.lowercased()
if normalized == "exact" { return DurationPrecision.exact }
return DurationPrecision.exact}
}
extension Duration {
func toSeconds() -> Double {
let yearsToMonths: Double = Double(years) * 31536000
+ Double(quarters) * 7776000
+ Double(months) * 2592000
let weeksToHours: Double = Double(weeks) * 604800
+ Double(days) * 86400
+ Double(hours) * 3600
let minutesToSeconds: Double = Double(minutes) * 60
+ Double(seconds)
return yearsToMonths + weeksToHours + minutesToSeconds
}
}

Line 12: assistant should match the name of your assistant's folder.

Line 102: Change this line to match your intentName username:intentName.

UnitySendMessage looks for the named GameObject with the named method and a string parameter to pass. Here we can see that char_jump matches the method which was in the GameController.csscript which was attached to a GameObject named RPG-Character.

The string parameter can be handy to pass intent slots to Unity.

Last step

We need to change one last thing in main.mm

On line 13, we need to change UnityAppController to AppControllerV

That's it ! Enjoy

Now you can build it for your device and after popping the character, you should be able to make him jump.

With the wake word triggered, you can see how the top right blue dot reacts.

  • It starts to grow when the wake word is detected

  • It's breathing when it is listening

  • If the intent is recognised, it becomes green

  • If the intent is not recognised, it becomes red

You won't need to go through each steps when you rebuild from Unity.

We are really excited to see what you are going to build!