OSC In Hand Engine

Hand Engine 3.1.2-Open-SDK-Beta or higher supports the retargeting of Hand Engine data into various character rig formats and ships with VRM model retargeting built in. Additional retargeting models for character rig types are loaded from JSON files located in the end-user’s local application data folder. Hand Engine creates an OSC client and dispatches OSC messages which can be received by OSC servers listening on specified ports (e.g. OSC servers built in Unity).

To perform duplex data transfer between to applications in OSC, both applications must implement an OSC server for listening to incoming OSC messages, and an OSC client for dispatching OSC messages. Use a different port for each app you wish to receive Open SDK data from Hand Engine. These ports can be adjusted in the Hand Engine Settings menu under Open SDK.

OSC Arguments

Hand Engine currently implements the signed integer (i), 32-bit float (f) and string (s) OSC argument types which are compatible with all OSC libraries.

Open SDK Core

The following functionality is intended to service both non-VR and VR integrations. It assumes that first time setup of gloves and performers has been configured either from within the Hand Engine application or via other OSC addresses.

OSC does not support enums natively. Instead, enums are transmitted as integer values in OSC messages. Developers are expected to implement their own typecasting of the OSC argument value to a structured enum that works with their chosen application framework or language. We recommend this approach as it will make applications easier to adapt for future API changes in future releases based on beta feedback.

Open

Animation Data

The following case-sensitive OSC addresses have been implemented as part of the Open SDK Core. These are often referred to these by their short name in documentation and the short name is used as a label in the Hand Engine settings screen. Animation data on these addresses has been retargeted to a particular character rig format, as described in the Retargeting section of this document:

Where:

• performerId - the integer ID of the performer in Hand Engine you wish to get data for. The default performerId is 0 and subsequent performers added will be sequential. E.g. 1, 2, 3, 4, 5 etc.

• handedness - an integer representation of the hand you wish to get data for. We suggest setting up an enum in your application to represent the below possible values for handedness:

  • 0 = Unknown (not used for getting data but can be used internally by your app if the hand state is not known.

  • 1 = Left Hand

  • 2 = Right Hand

• finger - a string representation of the finger you wish to get bone data for. Note that these are case sensitive and start with a capital letter:

  • Thumb = bones belonging to the thumb

  • Index = bones belonging to the index finger

  • Middle = bones belonging to the middle finger

  • Ring = bones belonging to the ring finger

  • Little = bones belonging to the little finger (also known as the “pinky” finger in some animation or interaction systems)

animation/rotation OSC Message Structure

In order to avoid issues with gimbal lock that commonly occur when using Euler angles for animation, this message structure provides quaternion values, suitable for character and hand rigs with 3 bones in the fingers and 2 bones in the thumb. Alternately proximal and intermediate bones can also be assigned to rigs with only 2 bones per finger.

The following Hand Engine remapping target profiles can be used with the animation/rotation OSC message data. Selecting the appropriate one of these are required for the quaternion data to be correctly retargeted to the hand rig in the application consuming the data:

• RAYNOS-chan_1.0.4-Studio.json / RAYNOS-chan_1.0.4-Fidelity.json - Compatible with VRM based models from VRoid Hub or the Sony Mocopi SDK (found under RAYNOS-chan\Retargeting\Hand Engine Retargeting Models in the Sony Mocopi sample in StretchSense Plugin for Unity 3.2.0-RC3 or higher).

animation/rotation OSC Message Argument Order

Each message has 13 arguments that are structured with the following indexes when accessing data for the 3-bone animation/rotation OSC address. You can each arguments data by calculating the correct offset using the index below. Most OSC libraries will do this calculation, so you will just need to provide the index and argument type when reading the data.

animation/rotationWithMetacarpals OSC Message Structure

These are suitable for character and hand rigs with 4 bones per finger and 3 bones in the thumb, commonly used in high-end animation or OpenXR. Unreal Engine 5 also uses 4 bones in its Manny, Quinn and MetaHuman character rigs.

The following Hand Engine remapping target profiles can be used with the animation/rotationWithMetacarpals OSC message data. Selecting the appropriate one of these are required for the quaternion data to be correctly retargeted to the hand rig in the application consuming the data:

• HTC-VIVE-Wave-Hands-Studio.json / HTC-VIVE-Wave-Hands-Fidelity.json - Compatible with the VIVE-Wave SDK’s OpenXR hand model (found under Retargeting\Hand Engine Retargeting Models in the VIVE-Wave sample in StretchSense Plugin for Unity 3.2.0-RC3 or higher).

Animation RotationWithMetacarpals Data OSC Message Formats

Each message has 17 arguments that are structured with the following indexes when accessing data for the 4-bone animation/rotationWithMetacarpals OSC address. You can each arguments data by calculating the correct offset using the index below. Most OSC libraries will do this calculation, so you will just need to provide the index and argument type when reading the data.

Animation Rotation Data OSC Arguments

The below descriptions are for arguments in the animation/rotation and animation/rotationWithMetacarpals OSC addresses.

Hand diagrams below are from Unity’s XR Hands package documentation Hand data model | XR Hands | 1.5.0-pre.1 (unity3d.com):

Metacarpal Rot. Open	Timecode	Metacarpal Rotation Quaternion X	Metacarpal Rotation Quaternion Y	Metacarpal Rotation Quaternion Z	Metacarpal Rotation Quaternion W

Metacarpal Rot.	Timecode	Metacarpal Rotation Quaternion X	Metacarpal Rotation Quaternion Y	Metacarpal Rotation Quaternion Z	Metacarpal Rotation Quaternion W
OSC type	integer	float	float	float	float
Language Type	int32 signed	float32 signed	float32 signed	float32 signed	float32 signed
Format	hhmmssfff	x float value to 17 d.p	y float value to 17 d.p	z float value to 17 d.p	w float value to 17 d.p
Example	134255043	-0.1692586	-0.1232476	-0.06889943	0.9754032
Validation	Min: 0, Max: 2147483647	Min: -1, Max: 1	Min: -1, Max: 1	Min: -1, Max: 1	Min: -1, Max: 1
Notes	fff in the format represents a 3-digit frame number for the given second.	Quaternion x value of the Proximal bone of the finger.	Quaternion y value of the Proximal bone of the finger.	Quaternion z value of the Proximal bone of the finger.	Quaternion w value of the Proximal bone of the finger.

animation/position OSC message structure

This message structure provides rotation values, suitable for character and hand rigs with 3 bones. The proximal and intermediate bones can also be assigned to rigs with only 2 bones per finger.

It is primarily used for determining the bone positioning for applications and animation tools that are based on fixed points in space for every tracked body part, in relation to the root bone of your character rig’s hands (e.g. wrist).

For most use-cases we recommend using data from the animation/rotation or animation/rotationWithMetacarpals OSC addresses instead.

Animation Position Data OSC Message Format

Animation Slider All

The current glove state is expressed by Hand Engine as float values ranging from -1 to 1 for the bend of the bone and splay of the fingers of the end-user’s hand. It is possible to derive basic hand poses from this data in your receiving application by testing these against minimum and maximum ranges for each slider.

Animation Slider Types

The following terms and features are used when working with animation sliders:

The GloveRevision argument of the Animation Slider All OSC Message contains the string value of the glove revision which can be checked to determine whether or not to read the additional control rig slider values supported by that glove.

For glove revisions that don’t support Bend3 sliders on certain fingers, such as Studio Gloves, the OSC arguments for these will be sent as an integer value -2147483648 (the minimum supported value for int32 data type). These values should be interpreted as being null and discarded or ignored by your application.

Storing Ignored Slider Values

Since a value of 0 is considered a valid value when determining the state of a bone, your application may opt to store ignored bone values as NaN. Many strongly typed languages require integers to have a value (they can't be null) but this can cause issues if you try and perform arithmetic or logical operations on a NaN value. Programming languages have options for testing for NaN like the C# Double.IsNaN(Double) method which could be used when testing control rig slider data for a specific hand gesture.

StretchSense Studio Glove Sliders

The Studio Glove supports up to 12 control rig sliders but does not include values for the Distal bone bend or individual finger splay. Aggregate splay for the entire hand is available by reading the GlobalSplay OSC argument and ThumbSplay is also available. See Animation Slider Types.

StretchSense Pro Fidelity Glove Sliders

The Fidelity Glove supports the full 21 control rig sliders, including Distal bone and per-finger splay.
See Animation Slider Types.

Animation Slider All OSC Message Format

Performer Information

The following device OSC addresses have been implemented as part of Open SDK Core which contain device OSC messages for all performers configured in Hand Engine:

• /v1/performer/all

Performer OSC Message Format

Device Information

Each glove and dongle’s status can also be received to show important information like the BatteryLevel, Handedness and if both gloves are connected to the dongle.

The following device OSC addresses have been implemented as part of Open SDK Core which contain device OSC messages for all devices connected to Hand Engine:

• /v1/device/dongle/status - Status for each USB dongle connected to Hand Engine.

• /v1/performer/glove/status - Status for each glove that is connected to a performer in Hand Engine.

Glove Revision Types

The GloveRevision OSC Argument contains these possible values as string OSC types:

Device Dongle Status

/v1/device/dongle/status returns all the dongles connected to Hand Engine.

Device Dongle Status OSC Message Format

Performer Glove Status

/v1/performer/glove/status returns all gloves connected to staged performers. In later versions of the Open SDK an additional route will be added to get all gloves connected to Hand Engine, including gloves that are not currently assigned to a performer.

Performer Glove Status OSC Message Format

Retargeting

The diagram shown in the Open SDK Core API Diagram section shows accepted OSC message arguments for each OSC address when receiving animation data for one or more calibrated performers.

Retargeted animation data is sent via OSC as each glove data packet comes in from the hardware.

Retarget From JSON During Live Animation

The Animation Bone OSC address will apply a retargeting model based on a currently selected retargeting model of the respective hand. retargeting models are imported by Hand Engine as JSON files from C:\Users\{YOUR_USER}\AppData\Local\Programs\HandEngine-3.0.11-PRO\assets\retargeting where YOUR_USER is the name of your Windows user.

A user can select which retargeting model they want using a dropdown beside each hand in the Hand Engine main screen.

Open SDK Core API Diagram

These are the core OSC addresses needed for an application to receive glove data and provide a good user experience.

OSC API Message Order of Operations

OSC messages may be received out of order, so all /v1/animation/* messages include a timecode OSC argument. This allows you to timecode sync the resulting messages if you are processing them as part of a larger motion capture system or need accuracy. For most live performance use-cases and for XR the messages should be processable by your application in a timely manner and any delay will be imperceivable by the end user, as long as messages are processed on the background thread.

Open SDK Implementation Guide

Preparing Hand Engine

Note that some Open SDK toggle settings may be unavailable in your version of Hand Engine, depending on your licence entitlements.

1. Open Hand Engine 3.1.2-Open-SDK-Beta or later.

2. Open the Hand Engine Settings Menu.

3. Under Open SDK, toggle on Enabled.

4. Close the settings menu.

5. Connect the USB dongle to your PC.

6. Turn on your gloves by pressing the power button on each glove once.

7. Wait for the gloves to start blinking blue at regular intervals.

8. For the left glove, select it from the dropdown menu.

9. If using a VRM character rig in your application, ensure the Retargeting Model is setdefault-retargeting-revg for Studio Gloves or default-retargeting-revf for Fidelity gloves.

10. Click Calibrate under the left glove.

11. Perform the calibration routine for each pose as prompted:

12. Repeat steps 8-11 for the right-hand glove.

Preparing Your Application

1. Choose an OSC library that supports multi-threading for your desired programming language (see the OSC Libraries list below). In some procedural programming languages like C for embedded systems and microcontrollers, or plugins for 3D animation software only have a single thread, so real-time performance may be reduced.

2. Follow your OSC library instructions to setup a server listening on 0.0.0.0 (all addresses), port 9400.

Receiving Animation Rotation Data

Receiving Animation Rotation Data for 3-Bone Hands

1. Set your Retargeting Model for the left hand that matches the 3-bone character rig you will be applying the rotation data to (e.g. RAYNOS-chan.1.0.4). It will automatically show you the options available for your connected glove.

2. Bind to the server to listen for the /v1/animation/{performerId}/{handedness}/{fingerName}/rotation OSC address, substituting the following variables in the address:

   1. performerId: 0 (the default/first performer)

   2. handedness: 1 (left hand)

   3. fingerName: Index

3. Alternately, if using an OpenXR compatible hand rig, bind to the server to listen for the /v1/animation/{performerId}/{handedness}/{fingerName}/rotationWithMetacarpals OSC address, substituting variables as above and using the HTC-VIVE-Wave-Hands retargeting model.

Storing Animation Rotation Data

If your OSC library supports processing incoming OSC messages on the background thread, setup handler functions to take these messages and store them in a way that is accessible by your main thread.

Animation Rotation Data Class List Recommended

This is the preferred approach.

1. Create an immutable data class instance representing the Animation Rotation Data Message Format.

2. Set the class properties to represent the serialized data, including the performerId (used to later match the data back to the performer assigned to your animated character when the data is processed on the main thread). If you are planning to sync the data with other full body tracked data sources that use a timecode, you must also store the timecode OSC argument from the OSC message.

3. Create an instance of this class every time a message comes in and read each OSC argument according to the Animation Rotation Data Message Format table, storing the argument in the data class.

4. Store this data class instance using the handedness, fingerName and bone name as the key for later reference e.g. LeftThumbProximal.

Lookup table

This is an approach that works with functional programming languages or plugin scripting languages with simpler data types.

1. Create a struct, dictionary or similar multidimensional array, with the handedness and fingerName as the key for later reference.

2. Store the full data in an array with the data in the same order as the OSC message arguments, making sure to include the performerId and timecode.

Applying Animation Rotation Data

1. Ensure you have a method of mapping the character or hand rig bones to your received bone data. Store this as a lookup table/dictionary/struct to avoid searching your game or animation engine object hierarchy.

2. Every frame in your game engine or animation software’s animation loop, read the latest values from the stored animation data.

3. (optional) filter the data so it matches the current timecode if you are wanting the animation to sync with another timecode source.

4. For each bone in your character or hand rig, read the list of data classes containing each bones information and find the matching data class instance for the current bone.

5. Apply the data to the bone transform in the game or animation engine.

Timecode Syncing Animation

1. Use the same approach for Applying Animation Rotation Data, but on every frame only get data that matches the current timecode for each bone.

2. Apply all bone data to your character or hand rig, along with any extra motion capture data such as body or face tracking in the same frame to ensure everything syncs together.

Receiving Animation Position Data

The same approach to receiving animation rotation data can be applied to the position data for each bone. For some animation systems they rely on positions of each bone instead of rotations in order to calculate frame animation for a character rig.

1. Use /v1/animation/{performerId}/{handedness}/{fingerName}/position instead of the animation rotation OSC address.

2. Process, store and apply the data using the same approach as the animation rotation guide. You may opt to store positional data in the same class, or postprocess the data into a final immutable object once both position and rotation data is received.

Receiving Control Rig Data

Hand Engine sends additional data about the current state of the hand in a simple format that can be used for adjusting the application’s state for the character or hand rig.

Gesture Recognition Using the Control Rig

There are a number of ways to use animation control rig data in your application or plugin:

XR Gestures

In XR applications, the control rig slider values have parity with gesture systems offered by other AR/VR frameworks and can drive the same functionality if a translation layer is created to map these to inputs. For example, the Unity XR Interaction System (XRI) v3.0 or higher supports Input Readers which allow the manual manipulation of the XRI state for selecting or activating objects (e.g. picking up or dropping XRGrabInteractables). While it is possible to integrate with XRI v2.5.x it requires several workarounds due to its reliance on Unity Input System Actions to drive application state, with no easy mechanism for simulating Input System Actions programmatically.

For a basic gesture recognition system within your application, it’s recommended to only detect up to 3 gestures on a single hand. We recommend the following single-handed gestures:

• Open hand - all sliders close to zero or a level with a relaxed hand pose

• Closed hand - Bend1 sliders at a threshold indicating all fingers are bent. Optionally also check the sliders for the thumb, but you will need to tweak these so it is comfortable for the user and so the gesture can be held reliably, especially when using the gesture to hold objects in the user's hand.

• Index point - Bend1 slider for index finger close to zero at a comfortable level. Other finger sliders for Bend1 should be similar to the Closed hand gesture.

To accommodate different users hand range of motion we recommend setting generous slider thresholds when detecting gestures as there may be some variance depending on the user’s calibration or the fit of their glove.

You will also need to debounce rapid changes in gestures in your application if gestures are too similar and wherever possible, give the detected gestures context by using a different gesture set while holding specific objects (e.g. a trigger pull gesture driven by Bend1 of the index finger that animates and activates a trigger while holding a pistol).

Multi-handed gestures can also be implemented as per the above method, but by checking both hands at once. This allows for a larger possible gesture space for driving behavior in your XR app.

Desk Capture for Animation

In animation programs, a plugin could use the control rig sliders to automatically select a hand gesture from a library to add as a keyframe, allowing gestures to be animated quickly outside of a mocap stage in post-production and having the consistency and precision, no matter which artist is producing the animation.

Driving Animation Controllers or Visual Scripting Node Graphs

Unity and other animation software supports driving animations using a node graph-based approach. This allows technical artists and animations to have greater control over animation blending based on incoming data. Each control rig slider exposed from animation/slider/all can be mapped to a finger bone rotation to drive the overall blending of an animation.

Aggregating multiple Bend sliders together can be used to animate the bend of a finger, with the animation graph providing final adjustment to avoid issues with mesh penetration or variable bone lengths. This also allows the gloves to drive characters with less than 5 fingers.

Some examples of this include:

• Controlling the movements of anthropomorphic humanoid characters ears, tails etc.

• Animating characters with more than two arms or tentacles, where the arm bend of extra appendages is partially driven by the pinky/little and ring fingers.

• Driving walk cycles of quadrupedal animal characters.

• Animating feet containing 3, 4 of 5 phalanges.

Receiving Other Hand Engine Data

Other data can be received and handled from Hand Engine in a similar way to animation rotation data. This includes animation positional data (for bone lengths), dongle/glove data and performer data. Take the same approach to storing this data in data objects or a lookup table in the background thread, then apply this in your main thread.

Here are some ideas for how to use the data. We show an example of this in our Open SDK OSC sample in the StretchSense Plugin for Unity:

• Create a list of all the connected gloves in Hand Engine and visualize the handedness and current battery level. This can help a user see how much charge is left in each glove without needing to look at the Hand Engine application.

• If you’re receiving data for multiple performers, visualize a list of the performers from Hand Engine and show which gloves and dongles are connected to each. Show the glove and dongle connection status and battery level as above.

• Use the data from /v1/animation/slider/all to make a gesture recognition system for your application. Show an effect when the user makes a peace gesture or a thumbs up.

Open SDK Performance Considerations

• Running more than 1 performer in Hand Engine with all the Open SDK OSC addresses toggled on may negatively impact the performance of your app. Turn off the OSC addresses your application doesn’t need. In a future Open SDK update we will allow these to be toggled programatically so your application can configure itself on start.

• The most expensive OSC addresses to listen to are as follows due to the high rate of messages transmitted based on the StretchSense glove model used:

  • animation/rotation (60-120Hz per hand, with 5 OSC messages per hand)

  • animation/rotationWithMetacarpals (60-120Hz per hand, with 5 OSC messages per hand)

  • animation/position (60-120Hz per hand, with 5 OSC messages per hand)

  • animation/slider/all (60-120Hz per hand, with 1 OSC message per hand)

• StretchSense glove models transmit at different rates, also dependent on your PCs CPU load:

  • Pro Fidelity: ~120Hz

  • Studio: ~60Hz

OSC Libraries

• TypeScript - node-osc - OSC client/server implementation in NodeJS / TypeScript recommended

• Python - python-osc - OSC client/server implementation in Python

• Unity C#

  • virtual-cast/OscCore: A performance-oriented OSC library for Unity (github.com). This library avoids garbage collection related issues that other Unity C# libraries have. Recommended

  • ValdemarOrn/SharpOSC: Full implementation of Open Sound Control in C# .NET 3.5 (github.com) This is a lower-level library that supports more types and implements the full OSC spec but requires extra boilerplate.