Open SDK OSC API Reference
In an upcoming release of the Open SDK, we aim to support auto configuration of Hand Engine using a “command” OSC address which is listening on the Command Listening Port. Third party applications can send OSC messages to this address to toggle on/off OSC addresses they will receive data from. This allows their application to be setup without a user manually configuring the Hand Engine settings for the Open SDK.
OSC In Hand Engine
Hand Engine 3.1.2-Open-SDK-Beta or higher supports the retargeting of Hand Engine data into various character rig formats and ships with VRM model retargeting built in. Additional retargeting models for character rig types are loaded from JSON files located in the end-user’s local application data folder. Hand Engine creates an OSC client and dispatches OSC messages which can be received by OSC servers listening on specified ports (e.g. OSC servers built in Unity).
To perform duplex data transfer between to applications in OSC, both applications must implement an OSC server for listening to incoming OSC messages, and an OSC client for dispatching OSC messages. Use a different port for each app you wish to receive Open SDK data from Hand Engine. These ports can be adjusted in the Hand Engine Settings menu under Open SDK.
OSC Arguments
Hand Engine currently implements the signed integer (i), 32-bit float (f) and string (s) OSC argument types which are compatible with all OSC libraries.
Null values are represented by -2147483648, so you can ignore these values as they indicate that the particular OSC argument is not to be used by your application.
Open SDK Core
The following functionality is intended to service both non-VR and VR integrations. It assumes that first time setup of gloves and performers has been configured either from within the Hand Engine application or via other OSC addresses.
OSC does not support enums natively. Instead, enums are transmitted as integer values in OSC messages. Developers are expected to implement their own typecasting of the OSC argument value to a structured enum that works with their chosen application framework or language. We recommend this approach as it will make applications easier to adapt for future API changes in future releases based on beta feedback.
Non-animation OSC messages are dispatched by Hand Engine every second, while animation OSC messages are dispatched as soon as new data is available.
Animation Data
The following case-sensitive OSC addresses have been implemented as part of the Open SDK Core. These are often referred to these by their short name in documentation and the short name is used as a label in the Hand Engine settings screen. Animation data on these addresses has been retargeted to a particular character rig format, as described in the Retargeting section of this document:
OSC Address | Short | Number of Bones Fingers | Number of Bones Thumb | Includes | OpenXR |
|---|---|---|---|---|---|
|
| 3 | 2 | ||
|
| 3 | 2 | ||
|
| 4 | 3 | ||
|
| 4 | 3 |
Where:
performerId- the integer ID of the performer in Hand Engine you wish to get data for. The defaultperformerIdis0and subsequent performers added will be sequential. E.g.1,2,3,4,5etc.handedness- an integer representation of the hand you wish to get data for. We suggest setting up an enum in your application to represent the below possible values forhandedness:0= Unknown (not used for getting data but can be used internally by your app if the hand state is not known.1= Left Hand2= Right Hand
finger- a string representation of the finger you wish to get bone data for. Note that these are case sensitive and start with a capital letter:Thumb= bones belonging to the thumbIndex= bones belonging to the index fingerMiddle= bones belonging to the middle fingerRing= bones belonging to the ring fingerLittle= bones belonging to the little finger (also known as the “pinky” finger in some animation or interaction systems)
animation/rotation OSC Message Structure
In order to avoid issues with gimbal lock that commonly occur when using Euler angles for animation, this message structure provides quaternion values, suitable for character and hand rigs with 3 bones in the fingers and 2 bones in the thumb. Alternately proximal and intermediate bones can also be assigned to rigs with only 2 bones per finger.
The following Hand Engine remapping target profiles can be used with the animation/rotation OSC message data. Selecting the appropriate one of these are required for the quaternion data to be correctly retargeted to the hand rig in the application consuming the data:
RAYNOS-chan_1.0.4-Studio.json/RAYNOS-chan_1.0.4-Fidelity.json- Compatible with VRM based models from VRoid Hub or the Sony Mocopi SDK (found underRAYNOS-chan\Retargeting\Hand Engine Retargeting Modelsin the Sony Mocopi sample in StretchSense Plugin for Unity3.2.0-RC3or higher).
animation/rotation OSC Message Argument Order
Each message has 13 arguments that are structured with the following indexes when accessing data for the 3-bone animation/rotation OSC address. You can each arguments data by calculating the correct offset using the index below. Most OSC libraries will do this calculation, so you will just need to provide the index and argument type when reading the data.
OSC Argument Index | OSC Argument Name | OSC Type |
|---|---|---|
0 | Timecode | integer |
1 | Proximal Rotation Quaternion X | float |
2 | Proximal Rotation Quaternion Y | float |
3 | Proximal Rotation Quaternion Z | float |
4 | Proximal Rotation Quaternion W | float |
5 | Intermediate Rotation Quaternion X | float |
6 | Intermediate Rotation Quaternion Y | float |
7 | Intermediate Rotation Quaternion Z | float |
8 | Intermediate Rotation Quaternion W | float |
9 | Distal Rotation Quaternion X | float |
10 | Distal Rotation Quaternion Y | float |
11 | Distal Rotation Quaternion Z | float |
12 | Distal Rotation Quaternion W | float |
animation/rotationWithMetacarpals OSC Message Structure
These are suitable for character and hand rigs with 4 bones per finger and 3 bones in the thumb, commonly used in high-end animation or OpenXR. Unreal Engine 5 also uses 4 bones in its Manny, Quinn and MetaHuman character rigs.
The following Hand Engine remapping target profiles can be used with the animation/rotationWithMetacarpals OSC message data. Selecting the appropriate one of these are required for the quaternion data to be correctly retargeted to the hand rig in the application consuming the data:
HTC-VIVE-Wave-Hands-Studio.json/HTC-VIVE-Wave-Hands-Fidelity.json- Compatible with the VIVE-Wave SDK’s OpenXR hand model (found underRetargeting\Hand Engine Retargeting Modelsin the VIVE-Wave sample in StretchSense Plugin for Unity3.2.0-RC3or higher).
Animation RotationWithMetacarpals Data OSC Message Formats
Each message has 17 arguments that are structured with the following indexes when accessing data for the 4-bone animation/rotationWithMetacarpals OSC address. You can each arguments data by calculating the correct offset using the index below. Most OSC libraries will do this calculation, so you will just need to provide the index and argument type when reading the data.
OSC Argument Index | OSC Argument Name | OSC Type |
|---|---|---|
0 | Timecode | integer |
1 | Metacarpal Rotation Quaternion X | float |
2 | Metacarpal Rotation Quaternion Y | float |
3 | Metacarpal Rotation Quaternion Z | float |
4 | Metacarpal Rotation Quaternion W | float |
5 | Proximal Rotation Quaternion X | float |
6 | Proximal Rotation Quaternion Y | float |
7 | Proximal Rotation Quaternion Z | float |
8 | Proximal Rotation Quaternion W | float |
9 | Intermediate Rotation Quaternion X | float |
10 | Intermediate Rotation Quaternion Y | float |
11 | Intermediate Rotation Quaternion Z | float |
12 | Intermediate Rotation Quaternion W | float |
13 | Distal Rotation Quaternion X | float |
14 | Distal Rotation Quaternion Y | float |
15 | Distal Rotation Quaternion Z | float |
16 | Distal Rotation Quaternion W | float |
Animation Rotation Data OSC Arguments
The below descriptions are for arguments in the animation/rotation and animation/rotationWithMetacarpals OSC addresses.
Hand diagrams below are from Unity’s XR Hands package documentation Hand data model | XR Hands | 1.5.0-pre.1 (unity3d.com):
Metacarpal Rot. | Timecode | Metacarpal Rotation Quaternion X | Metacarpal Rotation Quaternion Y | Metacarpal Rotation Quaternion Z | Metacarpal Rotation Quaternion W |
|---|---|---|---|---|---|
OSC type | integer | float | float | float | float |
Language Type | int32 signed | float32 signed | float32 signed | float32 signed | float32 signed |
Format | hhmmssfff | x float value to 17 d.p | y float value to 17 d.p | z float value to 17 d.p | w float value to 17 d.p |
Example | 134255043 |
|
|
|
|
Validation | Min: | Min: | Min: | Min: | Min: |
Notes | fff in the format represents a 3-digit frame number for the given second. | Quaternion x value of the Proximal bone of the finger. | Quaternion y value of the Proximal bone of the finger. | Quaternion z value of the Proximal bone of the finger. | Quaternion w value of the Proximal bone of the finger. |
Proximal Rot. | Timecode | Proximal Rotation Quaternion X | Proximal Rotation Quaternion Y | Proximal Rotation Quaternion Z | Proximal Rotation Quaternion W |
|---|---|---|---|---|---|
OSC type | integer | float | float | float | float |
Language Type | int32 signed | float32 signed | float32 signed | float32 signed | float32 signed |
Format | hhmmssfff | x float value to 17 d.p | y float value to 17 d.p | z float value to 17 d.p | w float value to 17 d.p |
Example | 134255043 |
|
|
|
|
Validation | Min: | Min: | Min: | Min: | Min: |
Notes | fff in the format represents a 3-digit frame number for the given second. | Quaternion x value of the Proximal bone of the finger. | Quaternion y value of the Proximal bone of the finger. | Quaternion z value of the Proximal bone of the finger. | Quaternion w value of the Proximal bone of the finger. |
Intermediate Rot. | Intermediate Rotation Quaternion X | Intermediate Rotation Quaternion Y | Intermediate Rotation Quaternion Z | Intermediate Rotation Quaternion W |
|
|---|---|---|---|---|---|
OSC type | float | float | float | float |
|
Language Type | float32 signed | float32 signed | float32 signed | float32 signed |
|
Format | x float value to 17 d.p | y float value to 17 d.p | z float value to 17 d.p | w float value to 17 d.p |
|
Example |
|
|
|
|
|
Validation | Min: | Min: | Min: | Min: |
|
Notes | Quaternion x value of the Intermediate bone of the finger. | Quaternion y value of the Intermediate bone of the finger. | Quaternion z value of the Intermediate bone of the finger. | Quaternion w value of the Intermediate bone of the finger. |
|
Distal Rot. | Distal Rotation Quaternion X | Distal Rotation Quaternion Y | Distal Rotation Quaternion Z | Distal Rotation Quaternion W |
|
|---|---|---|---|---|---|
OSC type | float | float | float | float |
|
Language Type | float32 signed | float32 signed | float32 signed | float32 signed |
|
Format | x float value to 17 d.p | y float value to 17 d.p | z float value to 17 d.p | w float value to 17 d.p |
|
Example |
|
|
|
|
|
Validation | Min: | Min: | Min: | Min: |
|
Notes | Quaternion x value of the Distal bone of the finger. | Quaternion y value of the Distal bone of the finger. | Quaternion z value of the Distal bone of the finger. | Quaternion w value of the Distal bone of the finger. |
|
animation/position OSC message structure
This message structure provides rotation values, suitable for character and hand rigs with 3 bones. The proximal and intermediate bones can also be assigned to rigs with only 2 bones per finger.
It is primarily used for determining the bone positioning for applications and animation tools that are based on fixed points in space for every tracked body part, in relation to the root bone of your character rig’s hands (e.g. wrist).
For most use-cases we recommend using data from the animation/rotation or animation/rotationWithMetacarpals OSC addresses instead.
Animation Position Data OSC Message Format
Proximal Position | Timecode | Proximal Position X | Proximal Position Y | Proximal Position Z |
|---|---|---|---|---|
OSC Argument Index | 0 | 1 | 2 | 3 |
OSC type | integer | float | float | float |
Language Type | int32 signed | float32 signed | float32 signed | float32 signed |
Format | hhmmssfff | x position value to 17 d.p | y position value to 17 d.p | z position value to 17 d.p |
Example | 134255043 |
|
|
|
Validation | Min: |
|
|
|
Notes | fff in the format represents a 3-digit frame number for the given second. | X position value of the Proximal bone of the finger. | Y position value of the Proximal bone of the finger. | Z position value of the Proximal bone of the finger. |
Intermediate Position | Intermediate Position X | Intermediate Position Y | Intermediate Position Z |
|
|---|---|---|---|---|
OSC Argument Index | 4 | 5 | 6 |
|
OSC type | float | float | float |
|
Language Type | float32 signed | float32 signed | float32 signed |
|
Format | x position value to 17 d.p | y position value to 17 d.p | z position value to 17 d.p |
|
Example |
|
|
|
|
Validation |
|
|
|
|
Notes | X position value of the Intermediate bone of the finger. | Y position value of the Intermediate bone of the finger. | Z position value of the Intermediate bone of the finger. |
|
Distal Position | Distal Position X | Distal Position Y | Distal Position Z |
|
|---|---|---|---|---|
OSC Argument Index | 7 | 8 | 9 |
|
OSC type | float | float | float |
|
Language Type | float32 signed | float32 signed | float32 signed |
|
Format | x position value to 17 d.p | y position value to 17 d.p | z position value to 17 d.p |
|
Example |
|
| 0 |
|
Validation |
|
|
|
|
Notes | X position value of the Distal bone of the finger. | Y position value of the Distal bone of the finger. | Z position value of the Distal bone of the finger. |
|
Animation Slider All
The current glove state is expressed by Hand Engine as float values ranging from -1 to 1 for the bend of the bone and splay of the fingers of the end-user’s hand. It is possible to derive basic hand poses from this data in your receiving application by testing these against minimum and maximum ranges for each slider.
Animation Slider Types
The following terms and features are used when working with animation sliders:
Slider Feature | Type | Description | Supports Studio? Rev g | Supports Pro Fidelity? Rev f | Animation Sliders |
|---|---|---|---|---|---|
Bend | Bend | The curl rotation of the other bones in the finger. |
|
| ThumbBend1, ThumbBend2, IndexBend1, IndexBend2, MiddleBend1, MiddleBend2, RingBend1, RingBend2, PinkyBend1, PinkyBend2 |
Distal Bend | Bend | The curl rotation of the end bone of the finger. |
|
| ThumbBend3, IndexBend3, MiddleBend3, RingBend3, PinkyBend3 |
Per-finger Splay | Splay | The horizontal movement outwards of the index, middle, ring and thumb fingers. |
|
| IndexSplay, MiddleSplay, RingSplay, PinkySplay |
Thumb Splay | Splay | The outward rotation of the thumb 90 degrees to the hand. |
|
| ThumbSplay |
Global Splay | Splay | The aggregated outward splay of all the fingers on the Studio glove. Used to detect spread fingers and in detection of an open palm. |
|
| GlobalSplay |
The GloveRevision argument of the Animation Slider All OSC Message contains the string value of the glove revision which can be checked to determine whether or not to read the additional control rig slider values supported by that glove.
For glove revisions that don’t support Bend3 sliders on certain fingers, such as Studio Gloves, the OSC arguments for these will be sent as an integer value -2147483648 (the minimum supported value for int32 data type). These values should be interpreted as being null and discarded or ignored by your application.
Storing Ignored Slider Values
Since a value of 0 is considered a valid value when determining the state of a bone, your application may opt to store ignored bone values as NaN. Many strongly typed languages require integers to have a value (they can't be null) but this can cause issues if you try and perform arithmetic or logical operations on a NaN value. Programming languages have options for testing for NaN like the C# Double.IsNaN(Double) method which could be used when testing control rig slider data for a specific hand gesture.
StretchSense Studio Glove Sliders
The Studio Glove supports up to 12 control rig sliders but does not include values for the Distal bone bend or individual finger splay. Aggregate splay for the entire hand is available by reading the GlobalSplay OSC argument and ThumbSplay is also available. See Animation Slider Types.
StretchSense Pro Fidelity Glove Sliders
The Fidelity Glove supports the full 21 control rig sliders, including Distal bone and per-finger splay.
See Animation Slider Types.
Animation Slider All OSC Message Format
Glove Metadata | Timecode | PerformerId | Handedness | GloveId | GloveRevision |
|---|---|---|---|---|---|
OSC Argument Index | 0 | 1 | 2 | 3 | 4 |
OSC type | integer | integer | integer | string | string |
Language Type | int32 signed | int32 signed | int32 signed | string | string |
Format | hhmmssfff | integer value | enum value | ModelSizeHandOther | string value |
Example | 134255043 |
|
| A2L230309 |
|
Validation | Min: | Min: | Unknown: | Rev G Max Length: | Max Length: Studio Glove: Pro Fidelity Glove: |
Notes | fff in the format represents a 3-digit frame number for the given second. Hand Engine supports up to 120fps data depending on the glove revision and your PC hardware capability. | This OSC attribute should be typecast to an Enum type in your application or programming language. Refer to the validation row to see all possible integer values for constructing the enum. | Rev G The handedness (shown in green) can be extracted from the GloveId, however we recommend using the Handedness field for this. Possible values are | ||
Rev F Pro Fidelity Glove Notes |
|
|
|
|
Thumb bones | ThumbBend1 | ThumbBend2 | ThumbBend3 | ThumbSplay |
|
|---|---|---|---|---|---|
OSC Argument Index | 5 | 6 | 7 | 8 |
|
OSC type | float | float | float | float |
|
Language Type | float32 signed | float32 signed | float32 signed or integer32 signed | float32 signed |
|
Format | float value | float value | float value or integer if not set | float value |
|
Example |
|
|
|
|
|
Validation | Min: | Min: | Min: | Rev G Min: Rev F Min: |
|
Notes |
|
|
| The splay of the thumb. A value of |
|
Rev G Studio Glove Notes | Rev G Represents the Distal thumb bone on Studio gloves. | Rev G Represents the Distal thumb bone on Studio gloves. |
|
|
|
Rev F Pro Fidelity Glove Notes | Rev F Represents the Metacarpal thumb bone on Fidelity gloves. | Rev F Represents the Proximal thumb bone on Fidelity gloves. | Rev F Represents the Distal thumb bone on Fidelity gloves.
|
|
Index bones | IndexBend1 | IndexBend2 | IndexBend3 |
|
|
|---|---|---|---|---|---|
OSC Argument Index | 9 | 10 | 11 |
|
|
OSC type | float | float | float |
|
|
Language Type | float32 signed | float32 signed | float32 signed or integer32 signed |
|
|
Format | float value to 17 d.p. | float value to 17 d.p. | float value to 17 d.p. or integer if not set |
|
|
Example |
|
|
|
|
|
Validation | Min: | Min: | Min: |
|
|
Notes | The Proximal index finger bone.
| The Intermediate index finger bone.
| The Distal index finger bone. |
|
|
Rev G Studio Glove Notes |
|
|
|
|
|
Rev F Pro Fidelity Glove Notes |
|
|
|
|
|
Middle bones | MiddleBend1 | MiddleBend2 | MiddleBend3 |
|
|
|---|---|---|---|---|---|
OSC Argument Index | 12 | 13 | 14 |
|
|
OSC type | float | float | float |
|
|
Language Type | float32 signed | float32 signed | float32 signed or integer32 signed |
|
|
Format | float value to 17 d.p. | float value to 17 d.p. | float value to 17 d.p. or integer if not set |
|
|
Example |
|
|
|
|
|
Validation | Min: | Min: | Min: |
|
|
Notes | The Proximal middle finger bone.
| The Intermediate middle finger bone.
| The Distal middle finger bone. |
|
|
Rev G Studio Glove Notes |
|
|
|
|
|
Rev F Pro Fidelity Glove Notes |
|
|
|
|
|
Ring bones | RingBend1 | RingBend2 | RingBend3 |
|
|
|---|---|---|---|---|---|
OSC Argument Index | 15 | 16 | 17 |
|
|
OSC type | float | float | float |
|
|
Language Type | float32 signed | float32 signed | float32 signed or integer32 signed |
|
|
Format | float value to 17 d.p. | float value to 17 d.p. | float value to 17 d.p. or integer if not set |
|
|
Example |
|
|
|
|
|
Validation | Min: | Min: | Min: |
|
|
Notes | The Proximal ring finger bone.
| The Intermediate ring finger bone.
| The Distal ring finger bone. |
|
|
Rev G Studio Glove Notes |
|
|
|
|
|
Rev F Pro Fidelity Glove Notes |
|
|
|
|
|
Pinky bones | PinkyBend1 | PinkyBend2 | PinkyBend3 |
|
|
|---|---|---|---|---|---|
OSC Argument Index | 15 | 16 | 17 |
|
|
OSC type | float | float | float |
|
|
Language Type | float32 signed | float32 signed | float32 signed or integer32 signed |
|
|
Format | float value to 17 d.p. | float value to 17 d.p. | float value to 17 d.p. or integer if not set |
|
|
Example |
|
|
|
|
|
Validation | Min: | Min: | Min: |
|
|
Notes | The Proximal ring finger bone.
| The Intermediate ring finger bone.
| The Distal ring finger bone. |
|
|
Rev G Studio Glove Notes |
|
|
|
|
|
Rev F Pro Fidelity Glove Notes |
|
|
|
|
|
Bone splay | GlobalSplay | IndexSplay | MiddleSplay | RingSplay | PinkySplay |
|---|---|---|---|---|---|
OSC Argument Index | 18 | 19 | 20 | 21 | 22 |
OSC type | float | float | float | float | float |
Language Type | float32 signed or integer32 signed | float32 signed or integer32 signed | float32 signed or integer32 signed | float32 signed or integer32 signed | float32 signed or integer32 signed |
Format | float value to 17 d.p. or integer if not set | float value to 17 d.p. | float value to 17 d.p. or integer if not set | float value to 17 d.p. or integer if not set | float value to 17 d.p. or integer if not set |
Example |
|
|
|
|
|
Validation | Min: | Min: | Min: | Min: | Min: |
Notes | The aggregate splay of all fingers in the hand, excluding the thumb. | The splay of the index finger in the left and right horizontal direction from the palm facing the user. | The splay of the middle finger in the left and right horizontal direction from the palm facing the user. | The splay of the ring finger in the left and right horizontal direction from the palm facing the user. | The splay of the pinky finger in the left and right horizontal direction from the palm facing the user. |
Rev G Studio Glove Notes |
|
|
|
|
|
Rev F Pro Fidelity Glove Notes |
|
|
|
|
|
Performer Information
The following device OSC addresses have been implemented as part of Open SDK Core which contain device OSC messages for all performers configured in Hand Engine:
/v1/performer/all
Performer OSC Message Format
Dongle Metadata | PerformerId | IsStaged | PerformerName |
|---|---|---|---|
OSC Argument Index | 0 | 1 | 2 |
OSC type | integer | integer | string |
Language Type | int32 signed | int32 signed | string |
Format | integer value | boolean value | string value |
Example |
|
|
|
Validation | Min: | Not staged:
| Max Length: Allowed characters:
|
Notes | This OSC attribute should be typecast to a boolean type in your application or programming language. Refer to the validation row to see all possible integer values for constructing the enum. |
Device Information
Each glove and dongle’s status can also be received to show important information like the BatteryLevel, Handedness and if both gloves are connected to the dongle.
The following device OSC addresses have been implemented as part of Open SDK Core which contain device OSC messages for all devices connected to Hand Engine:
/v1/device/dongle/status- Status for each USB dongle connected to Hand Engine./v1/performer/glove/status- Status for each glove that is connected to a performer in Hand Engine.
Glove Revision Types
The GloveRevision OSC Argument contains these possible values as string OSC types:
Name | String Value |
|---|---|
Pro Fidelity | RevF |
Studio | RevG |
Device Dongle Status
/v1/device/dongle/status returns all the dongles connected to Hand Engine.
Device Dongle Status OSC Message Format
Dongle Metadata | DongleId | GloveId | VersionStatus |
|---|---|---|---|
OSC Argument Index | 0 | 1 | 2 |
OSC type | string | string | integer |
Language Type | string | string | int32 signed |
Format | string value | ModelSizeHandOther | enum value |
Example | Rev G Rev F | A2L230309 |
|
Validation | Rev G Max Length: Rev F Max Length: | Rev G Max Length: | Unrecognized: Version Status Error: Version Status Unknown: Version Status Behind: Version Status Up To Date: Version Status Ahead:
|
Notes |
| Rev G The handedness (shown in green) can be extracted from the GloveId, however we recommend using the Handedness field for this. Possible values are | This OSC attribute should be typecast to an Enum type in your application or programming language. Refer to the validation row to see all possible integer values for constructing the enum. |
Rev F Pro Fidelity Glove Notes |
| VersionStatus |
Performer Glove Status
/v1/performer/glove/status returns all gloves connected to staged performers. In later versions of the Open SDK an additional route will be added to get all gloves connected to Hand Engine, including gloves that are not currently assigned to a performer.
Performer Glove Status OSC Message Format
Performer Glove Metadata | PerformerId | ProfileHandedness | DongleId | GloveId | GloveHandedness |
|---|---|---|---|---|---|
OSC Argument Index | 0 | 1 | 2 | 3 | 4 |
OSC type | integer | integer | string | string | integer |
Language Type | int32 signed | int32 signed | string | string | int32 signed |
Format | integer value | enum value | string value | ModelSizeHandOther | enum value |
Example |
|
| Rev G Rev F | A2L230309 |
|
Validation | Min: | Unknown: | Rev G Max Length: Rev F Max Length: | Rev G Max Length: | Unknown: |
Notes | This OSC attribute should be typecast to an Enum type in your application or programming language. Refer to the validation row to see all possible integer values for constructing the enum. |
| Rev G The handedness (shown in green) can be extracted from the GloveId, however we recommend using the Handedness field for this. Possible values are | This OSC attribute should be typecast to an Enum type in your application or programming language. Refer to the validation row to see all possible integer values for constructing the enum. | |
Rev F Pro Fidelity Glove Notes |
|
|
|
|
Glove Metadata | IsCalibrated | GloveRevision | ProductName | FirmwareVersion | VersionStatus |
|---|---|---|---|---|---|
OSC Argument Index | 5 | 6 | 7 | 8 | 9 |
OSC type | integer | string | string | string | integer |
Language Type | int32 signed | string | string | string | int32 signed |
Format | boolean value | string value | string value | string value | enum value |
Example |
|
| Studio Glove | Rev G Rev F |
|
Validation | Not calibrated:
| Max Length: Studio Glove: Pro Fidelity Glove: | Studio Glove: Pro Fidelity: | Rev G Max Length: Rev F Max Length: | Unrecognized: Version Status Error: Version Status Unknown: Version Status Behind: Version Status Up To Date: Version Status Ahead:
|
Notes | This OSC attribute should be typecast to a boolean type in your application or programming language. Refer to the validation row to see all possible integer values for constructing the enum. | This OSC attribute should be typecast to an Enum type in your application or programming language. Refer to the validation row to see all possible integer values for constructing the enum. |
Glove Hardware Metadata | BatteryLevel | IsSdCardPresent | IsJamSynced |
|
|
|---|---|---|---|---|---|
OSC Argument Index | 10 | 11 | 12 |
|
|
OSC type | float | integer | integer |
|
|
Language Type | float32 signed | int32 signed | int32 signed |
|
|
Format | float value | enum value | enum value |
|
|
Example |
|
|
|
|
|
Validation | Min:
| Min: Rev G
| Min: Rev G |
|
|
Notes | A max value of |
|
|
|
|
Rev G Studio Glove Notes |
|
|
|
Retargeting
The diagram shown in the Open SDK Core API Diagram section shows accepted OSC message arguments for each OSC address when receiving animation data for one or more calibrated performers.
Retargeted animation data is sent via OSC as each glove data packet comes in from the hardware.
Retarget From JSON During Live Animation
The Animation Bone OSC address will apply a retargeting model based on a currently selected retargeting model of the respective hand. retargeting models are imported by Hand Engine as JSON files from C:\Users\{YOUR_USER}\AppData\Local\Programs\HandEngine-3.0.11-PRO\assets\retargeting where YOUR_USER is the name of your Windows user.
A user can select which retargeting model they want using a dropdown beside each hand in the Hand Engine main screen.
Open SDK Core API Diagram
These are the core OSC addresses needed for an application to receive glove data and provide a good user experience.
OSC API Message Order of Operations
OSC messages may be received out of order, so all /v1/animation/* messages include a timecode OSC argument. This allows you to timecode sync the resulting messages if you are processing them as part of a larger motion capture system or need accuracy. For most live performance use-cases and for XR the messages should be processable by your application in a timely manner and any delay will be imperceivable by the end user, as long as messages are processed on the background thread.
Open SDK Implementation Guide
Preparing Hand Engine
Note that some Open SDK toggle settings may be unavailable in your version of Hand Engine, depending on your licence entitlements.
Open Hand Engine 3.1.2-Open-SDK-Beta or later.
Open the Hand Engine Settings Menu.
Under Open SDK, toggle on Enabled.
Close the settings menu.
Connect the USB dongle to your PC.
Turn on your gloves by pressing the power button on each glove once.
Wait for the gloves to start blinking blue at regular intervals.
For the left glove, select it from the dropdown menu.
If using a VRM character rig in your application, ensure the Retargeting Model is set
default-retargeting-revgfor Studio Gloves ordefault-retargeting-revffor Fidelity gloves.Click Calibrate under the left glove.
Perform the calibration routine for each pose as prompted:
Repeat steps 8-11 for the right-hand glove.
Preparing Your Application
Choose an OSC library that supports multi-threading for your desired programming language (see the OSC Libraries list below). In some procedural programming languages like C for embedded systems and microcontrollers, or plugins for 3D animation software only have a single thread, so real-time performance may be reduced.
Follow your OSC library instructions to setup a server listening on
0.0.0.0(all addresses), port9400.
Receiving Animation Rotation Data
Receiving Animation Rotation Data for 3-Bone Hands
Set your Retargeting Model for the left hand that matches the 3-bone character rig you will be applying the rotation data to (e.g. RAYNOS-chan.1.0.4). It will automatically show you the options available for your connected glove.
Bind to the server to listen for the
/v1/animation/{performerId}/{handedness}/{fingerName}/rotationOSC address, substituting the following variables in the address:performerId:
0(the default/first performer)handedness:
1(left hand)fingerName:
Index
Alternately, if using an OpenXR compatible hand rig, bind to the server to listen for the
/v1/animation/{performerId}/{handedness}/{fingerName}/rotationWithMetacarpalsOSC address, substituting variables as above and using the HTC-VIVE-Wave-Hands retargeting model.
Storing Animation Rotation Data
If your OSC library supports processing incoming OSC messages on the background thread, setup handler functions to take these messages and store them in a way that is accessible by your main thread.
Animation Rotation Data Class List Recommended
This is the preferred approach.
Create an immutable data class instance representing the Animation Rotation Data Message Format.
Set the class properties to represent the serialized data, including the
performerId(used to later match the data back to the performer assigned to your animated character when the data is processed on the main thread). If you are planning to sync the data with other full body tracked data sources that use a timecode, you must also store thetimecodeOSC argument from the OSC message.Create an instance of this class every time a message comes in and read each OSC argument according to the Animation Rotation Data Message Format table, storing the argument in the data class.
Store this data class instance using the
handedness,fingerNameand bone name as the key for later reference e.g.LeftThumbProximal.
Lookup table
This is an approach that works with functional programming languages or plugin scripting languages with simpler data types.
Create a struct, dictionary or similar multidimensional array, with the
handednessandfingerNameas the key for later reference.Store the full data in an array with the data in the same order as the OSC message arguments, making sure to include the
performerIdandtimecode.
Applying Animation Rotation Data
Ensure you have a method of mapping the character or hand rig bones to your received bone data. Store this as a lookup table/dictionary/struct to avoid searching your game or animation engine object hierarchy.
Every frame in your game engine or animation software’s animation loop, read the latest values from the stored animation data.
(optional) filter the data so it matches the current timecode if you are wanting the animation to sync with another timecode source.
For each bone in your character or hand rig, read the list of data classes containing each bones information and find the matching data class instance for the current bone.
Apply the data to the bone transform in the game or animation engine.
Timecode Syncing Animation
Use the same approach for Applying Animation Rotation Data, but on every frame only get data that matches the current timecode for each bone.
Apply all bone data to your character or hand rig, along with any extra motion capture data such as body or face tracking in the same frame to ensure everything syncs together.
Receiving Animation Position Data
The same approach to receiving animation rotation data can be applied to the position data for each bone. For some animation systems they rely on positions of each bone instead of rotations in order to calculate frame animation for a character rig.
Use
/v1/animation/{performerId}/{handedness}/{fingerName}/positioninstead of the animation rotation OSC address.Process, store and apply the data using the same approach as the animation rotation guide. You may opt to store positional data in the same class, or postprocess the data into a final immutable object once both position and rotation data is received.
Receiving Control Rig Data
Hand Engine sends additional data about the current state of the hand in a simple format that can be used for adjusting the application’s state for the character or hand rig.
Gesture Recognition Using the Control Rig
There are a number of ways to use animation control rig data in your application or plugin:
XR Gestures
In XR applications, the control rig slider values have parity with gesture systems offered by other AR/VR frameworks and can drive the same functionality if a translation layer is created to map these to inputs. For example, the Unity XR Interaction System (XRI) v3.0 or higher supports Input Readers which allow the manual manipulation of the XRI state for selecting or activating objects (e.g. picking up or dropping XRGrabInteractables). While it is possible to integrate with XRI v2.5.x it requires several workarounds due to its reliance on Unity Input System Actions to drive application state, with no easy mechanism for simulating Input System Actions programmatically.
For a basic gesture recognition system within your application, it’s recommended to only detect up to 3 gestures on a single hand. We recommend the following single-handed gestures:
Open hand - all sliders close to zero or a level with a relaxed hand pose
Closed hand -
Bend1sliders at a threshold indicating all fingers are bent. Optionally also check the sliders for the thumb, but you will need to tweak these so it is comfortable for the user and so the gesture can be held reliably, especially when using the gesture to hold objects in the user's hand.Index point -
Bend1slider for index finger close to zero at a comfortable level. Other finger sliders forBend1should be similar to the Closed hand gesture.
To accommodate different users hand range of motion we recommend setting generous slider thresholds when detecting gestures as there may be some variance depending on the user’s calibration or the fit of their glove.
You will also need to debounce rapid changes in gestures in your application if gestures are too similar and wherever possible, give the detected gestures context by using a different gesture set while holding specific objects (e.g. a trigger pull gesture driven by Bend1 of the index finger that animates and activates a trigger while holding a pistol).
Multi-handed gestures can also be implemented as per the above method, but by checking both hands at once. This allows for a larger possible gesture space for driving behavior in your XR app.
Desk Capture for Animation
In animation programs, a plugin could use the control rig sliders to automatically select a hand gesture from a library to add as a keyframe, allowing gestures to be animated quickly outside of a mocap stage in post-production and having the consistency and precision, no matter which artist is producing the animation.
Driving Animation Controllers or Visual Scripting Node Graphs
Unity and other animation software supports driving animations using a node graph-based approach. This allows technical artists and animations to have greater control over animation blending based on incoming data. Each control rig slider exposed from animation/slider/all can be mapped to a finger bone rotation to drive the overall blending of an animation.
Aggregating multiple Bend sliders together can be used to animate the bend of a finger, with the animation graph providing final adjustment to avoid issues with mesh penetration or variable bone lengths. This also allows the gloves to drive characters with less than 5 fingers.
Some examples of this include:
Controlling the movements of anthropomorphic humanoid characters ears, tails etc.
Animating characters with more than two arms or tentacles, where the arm bend of extra appendages is partially driven by the pinky/little and ring fingers.
Driving walk cycles of quadrupedal animal characters.
Animating feet containing 3, 4 of 5 phalanges.
Receiving Other Hand Engine Data
Other data can be received and handled from Hand Engine in a similar way to animation rotation data. This includes animation positional data (for bone lengths), dongle/glove data and performer data. Take the same approach to storing this data in data objects or a lookup table in the background thread, then apply this in your main thread.
Here are some ideas for how to use the data. We show an example of this in our Open SDK OSC sample in the StretchSense Plugin for Unity:
Create a list of all the connected gloves in Hand Engine and visualize the handedness and current battery level. This can help a user see how much charge is left in each glove without needing to look at the Hand Engine application.
If you’re receiving data for multiple performers, visualize a list of the performers from Hand Engine and show which gloves and dongles are connected to each. Show the glove and dongle connection status and battery level as above.
Use the data from
/v1/animation/slider/allto make a gesture recognition system for your application. Show an effect when the user makes a peace gesture or a thumbs up.
Open SDK Performance Considerations
Running more than 1 performer in Hand Engine with all the Open SDK OSC addresses toggled on may negatively impact the performance of your app. Turn off the OSC addresses your application doesn’t need. In a future Open SDK update we will allow these to be toggled programatically so your application can configure itself on start.
The most expensive OSC addresses to listen to are as follows due to the high rate of messages transmitted based on the StretchSense glove model used:
animation/rotation(60-120Hz per hand, with 5 OSC messages per hand)animation/rotationWithMetacarpals(60-120Hz per hand, with 5 OSC messages per hand)animation/position(60-120Hz per hand, with 5 OSC messages per hand)animation/slider/all(60-120Hz per hand, with 1 OSC message per hand)
StretchSense glove models transmit at different rates, also dependent on your PCs CPU load:
Pro Fidelity: ~120Hz
Studio: ~60Hz
OSC Libraries
TypeScript - node-osc - OSC client/server implementation in NodeJS / TypeScript recommended
Python - python-osc - OSC client/server implementation in Python
Unity C#
virtual-cast/OscCore: A performance-oriented OSC library for Unity (github.com). This library avoids garbage collection related issues that other Unity C# libraries have. Recommended
ValdemarOrn/SharpOSC: Full implementation of Open Sound Control in C# .NET 3.5 (github.com) This is a lower-level library that supports more types and implements the full OSC spec but requires extra boilerplate.
Changelog | Version | Publish Date (YYYY/MM/DD) |
|---|---|---|
Initial version. | v1.0.0 | 2024/05/14 |