This document outlines the Unity sample which shows how to receive Open SDK data via OSC in a third-party Unity Application.

StretchSense Plugin for Unity Samples

The StretchSense Plugin for Unity 3.2.0-RC3 package contains the following samples:

Common

• Sample containing dependencies for all of the below samples. Import this one first.

Open SDK OSC

• Receiving and applying bone rotation data from the animation/rotation OSC address to a VRM model imported into the project using UniVRM. The data is retargeted by Hand Engine using the RAYNOS-chan-1.0.4 retargeting model. This data overrides any animation already applied to the hands of the character.

• Receiving and visualizing glove battery status data.

• Receiving and visualizing dongle status data.

• Receiving and visualizing Hand Engine control rig slider data.

• Retargeting FBX animation onto a VRM model using UniVRM.

• Visualizing the performance of the Unity application frame rate, and graphing received OSC messages over time to help the developer identify performance spikes during testing due to network, hardware or Unity application software performance issues.

Requires the Open Sound Control Core v2.0.0 fork by Virtual Cast, Inc. to receive OSC messages from Hand Engine.

Optionally requires the UniVRM UPM package to be added to your Unity project to load the included VRoid Hub VRM sample models. If you are not using these models, you can safely delete these and their prefabs from the sample. If using URP in your project and using the VRM samples models, we recommend using the lilToon shaders instead of the UniVRM ones.

VIVE-Wave

• Receiving and applying bone rotation data from the animation/rotationWithMetacarpals OSC address to the hand rig from the VIVE-Wave SDK.

• Includes a prefab which loads the left and right hand models and is pre-configured to work with the Open SDK.

Sony Mocopi

• Contains the source retargeting profile data and the JSON files for RAYNOS-chan-1.0.4 retargeting model for use with Hand Engine.

Hand Engine Setup

Install Hand Engine 3.1.2 Open SDK Beta or higher and open Edit → Settings and set the following. Ensure the Enabled switch is toggled ON so Open SDK OSC messages are sent out from Hand Engine to third party applications running on the same PC.

1. Use the following IP addresses for sending OSC messages to your application:

   1. 127.0.0.1 - localhost loopback to the same PC that is running your Unity project either in the Unity Editor or as a build of your Unity project (EXE, APK etc).

   2. 192.168.X.X - a specific IP address on your network for another device running your Unity app.

   3. 192.168.X.255 - the broadcast address on your network. Both the PC running Hand Engine and your Unity app must be on the same subnet and IP range. Most domestic WiFi routers and many network firewalls will block or throttle these so we don’t recommend this method unless you have full control over your network configuration.

2. Add port 9401 if you wish to debug the outgoing Open SDK OSC messages from within Hand Engine using Tools → Open SDK Test Console to see the order of OSC attributes in each message. Note that the type of each OSC attribute isn’t shown in the test console, so please refer to the Open SDK API Documentation for details.

   Open

   

    

3. In order to receive animation/rotation and animation/position OSC messages, you must have your gloves connected to Hand Engine and assigned to at least one performer. A default performer is already set up for you in Hand Engine after installation. Connect the gloves and select them from the dropdowns on the left- and right-hand side:

   Open

   

    

4. Perform a glove calibration for each glove by clicking Calibrate and follow the on-screen instructions to complete the calibration. Check the hands move on screen.

5. After this, when accessing Tools → Open SDK Test Console you will see OSC messages being sent out of Hand Engine for every OSC address that has been toggled ON in Edit → Settings:

    

6. If you don’t see any messages listed on this screen, you may have another app or service on your PC using the Open SDK Test Console port; try changing it to another value.

Running the Open SDK OSC Sample

Install Sample Dependencies

Add the following packages via UPM:

Open Sound Control Core / OscCore

Open Sound Control Core v2.0.0 fork by Virtual Cast, Inc. This fork corrects issues with receiving string OSC argument types.

1. Add this URL https://github.com/virtual-cast/OscCore.git via Window -> Package Manager -> Add package from Git URL....

2. Optionally install the Basic Examples package for OscCore from the Samples tab in the Unity Package Manager.

UniVRM

Install this package if you are planning to use VRM models in your Unity project from the Open SDK OSC sample. We have provided sample VRM models from VRoid Hub for convenience so the sample scene will work out of the box with no modification.

In the Hand Engine settings the default retargeting preset will output bone rotations retargeted for a VRM model. The original retargeting preset is based on Sony’s RAYNOS-chan avatar which is compatible with UniVRM.

1. Install the latest release of UniVRM by following the installation instructions. We recommend adding the listed packages in their instructions via their UPM URLs using Window -> Package Manager -> Add package from Git URL....

2. After installation, any VRM models from the Models folder in this sample will automatically be converted into Unity prefabs which you can place in your scene. They will have their shaders and other supporting scripts added. This sample has the following VRM 1.0 models pre-configured in the StretchSense Open SDK Studio Demo scene:

   1. AvatarSample_A

   2. AvatarSample_B

   3. AvatarSample_C
      These models are already set up with weight maps which are compatible with UniVRM's VRM Spring Bone component added to the secondary GameObject in each VRM model prefab. You can adjust these or enable/disable them depending on which body parts or clothing you want physics to be enabled on. Optionally you can add the Blinker script from the UniVRM package to get automatic blinking for the eye blend shapes on your VRM characters. VRM 0.x models are also supported by UniVRM.

3. If you are planning on using the VRM model as an avatar that is part of a VR application, you can adjust the Renderers settings in the VRM First Person component to hide the Face, Body and Hair in first person mode, so these do not appear in your view. See more about the VRMFirstPerson feature here.

Sample Animations

The Open SDK sample comes with an AI generated looping animation created using Unity Muse Animate which you are free to use in your projects. Alternatively, you can retarget any FBX animation onto a VRM character imported by UniVRM using the following process:

Retargeting FBX Animations on VRM Characters

This allows you to combine an existing pre-recorded animation with StretchSense glove data on a VRM character while testing. Not all animation controllers and animations are directly compatible with VRM models imported via UniVRM, so this provides a workaround.

1. Change the import settings for your FBX animation to Humanoid:

2. Drag the FBX that contains this animation into your scene to create a GameObject of it.

3. Add a Human Pose Transfer component to the GameObject in your scene.

4. Add the VRM character prefab to your scene. We have example ones available in the Models/VRM folder of this sample (AvatarSample_A, AvatarSample_B, AvatarSample_C) that have already been imported using UniVRM.

5. Add a second Human Pose Transfer component to your VRM character in the scene.

6. Set the following properties:

   1. SourceType: HumanPoseTransfer

   2. Source: (reference the first Human Pose Transfer component you added to the GameObject containing your FBX animation)

7. When the scene starts during play mode, every frame the animation data will be copied from the FBX GameObject onto your VRM character. The character’s world position will jump to match the FBX animation, so be sure to change the material types on the FBX animation so they are not visible. For the Human Pose Transfer component to work there must be at least one SkinnedMeshRenderer enabled in your FBX animation GameObject. The StretchSense glove hand animation data will be applied on top of the FBX animation if Hand Engine is configured as per this guide and it is running with StretchSense gloves connected.

Unity Input System

Required if using the performance graph prefab from the StretchSense Open SDK Studio Demo sample scene. The performance graph will automatically enable the action provided for FPSGraph.cycleActionRef on scene start.

1. Open Window -> Package Manager -> Add package by name... and enter com.unity.inputsystem.

Importing the Open SDK OSC Sample

1. Import the sample from the Unity Package Manager window by finding StretchSense Plugin for Unity in your Unity Package Manager list.

2. Switch to the Samples tab.

3. Import the Common and Open SDK OSC samples to your project. If you have already imported a sample before it will give you the option to Reimport. If upgrading from a previous version of the StretchSense Studio Plugin for Unity, it may prompt you to delete the previous sample if importing the updated sample, ensure you have backed up any changes you have made to the samples in your project prior to doing this.

Running the Open SDK OSC Sample

1. Ensure the Hand Engine Setup is complete, your gloves are calibrated, and the Open SDK is toggled ON.

2. The sample scene showing usage of the Open SDK with StretchSense Studio Gloves will be located in your project at Samples/StretchSense Studio Plugin/3.2.0-RC3/Open SDK OSC/Scenes/StretchSense Open SDK Studio Demo.unity.

3. Open this scene and click the Play button in the Unity editor.

Importing the VIVE-Wave Sample

1. Download the VIVE-Wave SDK for All-in-one / standalone HTC headsets.

2. Import the downloaded unitypackage to your project.

3. Use the Wave Installer to install the full SDK into your project.

Running the VIVE-Wave Sample

1. Ensure the Hand Engine Setup is complete, your gloves are calibrated, and the Open SDK is toggled ON.

2. For both hands, select the HTC-VIVE-Wave-Hands retargeting models from the dropdowns.

3. The sample scene showing usage of the Open SDK with the VIVE-Wave hand rig will be located in your project at Assets/Samples/StretchSense Studio Plugin/3.2.0-RC3/VIVE-Wave/Scenes/StretchSense Open SDK for VIVE-Wave Demo.

4. Open this scene and click the Play button in the Unity editor.

Adapting the VIVE-Wave Sample

1. Use the VIVE-Wave SDK sample content and configure the VIVE-Wave hands in the same way as our example scene.

2. We recommend purchasing and using HTC Vive Ultimate trackers with dedicated wrist mounts.

3. Configure Hand Engine to use the IP address of your VIVE Focus 3 or VIVE XR Elite headset on your network. While broadcast addresses are supported for convenience (e.g. 192.168.86.255) many network router and firewall configurations will block or throttle broadcast UDP packets. We recommend finding and setting the headset’s IP address directly:

4. Turn off un-needed OSC addresses. This will improve performance over the network for your headset when receiving animation data. Make sure animation/rotationWithMetacarpals is turned on.

5. Build and deploy your app to the headset.

6. Run the app on your headset. If a connection is successful, the VIVE-Wave hand rig’s fingers will animate.

Debugging OSC Messages

When using OscCore, you need to know the type of OSC message attributes relative to their index in the OSC message in order to correctly parse them. We have limited OSC attribute types to floats, integers and strings to maximize compatibility with various OSC libraries and programming languages.

You can use the following tools to help with this:

• OscCore - The OscCore Unity package has a built-in message debugger accessible from Window -> OscCore -> Monitor. This works for the OSC server created by your app if you are using the OscCore library.

• osc-debugger - Node.js command line tool to connect OSC client and receive messages without knowing the address or message format.

• extOSC - Unity plugin to debug OSC messages including various features to send/receive OSC messages.

• Hand Engine - A debugger which shows the outgoing OSC addresses and messages is viewable by opening Tools → Open SDK Test Console. You must add port 9401 in order to see messages in this console. This port is not setup by default for end users as there is a small performance impact sending messages to this console.