Develop

Sensation Editor tutorial: #2 Using the reference code 

1136 views 31/07/2018 Nicci 2

Tip: Follow our tutorial to learn how to use the Sensation Editor tool

Using the Sensation Editor reference code 

You can use the Sensation Editor to export AM Ultrahaptics Sensation Package (.usp) files and integrate into your application using our reference C/C++ example code. This is a small, minimal build, console application that uses the same core code used by the Sensation Editor.

The Sensation Editor reference code builds a small application called FPGenApp that takes a .usp sensation package filename as its only argument and plays it back in the same way as the Sensation Player.

Building the Sensation Editor reference code 

You can download the reference code and unzip the package to view the files: 

Reference code

Open the Readme.txt file and follow the instructions to build the code. Please note the required pre-requisites: 

For details on building code using CMake and the correct version of the Leap Motion® SDK, please read our tutorial on building our Ultrahaptics SDK C++ examples.

Once built, the executable FPGenApp can be found under your build directory inside src/FPGenApp/. 

Run the code from a console or a DOS prompt by typing 

FPGenApp <sensation package filename>

You can integrate sensation playback directly into your application by calling the command above or by customising the reference code directly. 

Reference code architecture 

The reference code package contains a documentation suite that can be accessed by opening doc/html/index.html. This contains documentation for all of the code entities contained in the package. 

The reference code architecture and its relationship with the Ultrahaptics API can be seen in the diagram below:

Reference code architecture

Modules are abstracted into separate libraries with responsibility for hand tracking, source interface (sensation package file) and sink interface (the Ultrahaptics device). At the heart of the system is the Focal Point Generation module, written in C and with interfaces to sensation package and sink device objects. This decodes the sensation package for a time step, interpolates between time steps, applies tracking and writes the output to the sink. 

In main.cpp, the application instantiates an instance of the source class:

Ultrahaptics::FPGen::SensationPkgToFPGenAdapter

and a sink:

FPGenApp::AppSinkImplementation

to decode the sensation package and update the array hardware by continually calling FPGEN_evaluatePackageAtStep.

Here is the code outline of the main function (some code has been removed to ease reading):  

void main(int argc, char **argv)
{
    SensationPackage pkgData;
    const unsigned frameInterval = 10;

    loadSensationFromFile(std::string(argv[1]), pkgData); // Load a sensation file

    auto tracking = std::make_unique();
    auto alignment = std::make_unique("dragonfly.alignment.xml");
    /* Instantiation of the sensation package to FPGen adapter class. */
    Ultrahaptics::FPGen::SensationPkgToFPGenAdapter sensationImplementation(&pkgData, 
	tracking.get(), alignment.get());

    /* Instantiation of the hardware sink class. */
    FPGenApp::AppSinkImplementation sinkImplementation;

    unsigned step = 0;
    unsigned lengthInSteps = FPGEN_calculatePlaybackLength(
	sensationImplementation.getSensationPkgInterface());

    using clock = std::chrono::steady_clock;
    const auto playbackStartTime = clock::now();
    isRunning = 1;
    do
    {
        /* Example loop, not real-time. */
        FPGEN_evaluatePackageAtStep(
		sensationImplementation.getSensationPkgInterface(),
		step, sinkImplementation.getSinkInterface());

        std::this_thread::sleep_for(std::chrono::milliseconds(frameInterval));
        auto timeSinceStart = clock::now() - playbackStartTime;
        step = (unsigned)((std::chrono::duration_cast(
	timeSinceStart).count() * pkgData.stepsPerSecond) / 1000);
    } while (((step < lengthInSteps) || (lengthInSteps == 0)) && isRunning);

    /* Stop the device by clearing the sink and updating. */
    sinkImplementation.clearContext(&sinkImplementation, 0);
    sinkImplementation.updateDevice(&sinkImplementation, nullptr);
}

Example button application  

The reference code above can be easily modified by customising the contents of main.cpp to make a simple button application. First, refactor the main function to make a separate playSensation function:

void playSensation(Ultrahaptics::FPGen::SensationPkgToFPGenAdapter &clickInSensastion,
          FPGenApp::AppSinkImplementation &sinkImplementation, 
	unsigned stepsPerSecond)
{
    unsigned step = 0;
    unsigned lengthInSteps = FPGEN_calculatePlaybackLength(
                                 clickInSensastion.getSensationPkgInterface());

    using clock = std::chrono::steady_clock;
    const auto playbackStartTime = clock::now();
    do {
        /* Not a real-time loop but good enough as an example. */
        FPGEN_RTN_E eRtn = FPGEN_evaluatePackageAtStep(
	clickInSensastion.getSensationPkgInterface(),
	step, sinkImplementation.getSinkInterface());

        if (eRtn != FPGEN_RTN_OK) {
            std::cout << "Error code detected: " << eRtn << ".\n";
            break;
        }
        std::this_thread::sleep_for(std::chrono::milliseconds(10));
        auto timeSinceStart = clock::now() - playbackStartTime;
        step = (unsigned)((std::chrono::duration_cast(
	timeSinceStart).count() * stepsPerSecond) / 1000);

    } while ((step < lengthInSteps) || (lengthInSteps == 0));

    /* Stop the device by clearing the sink and updating. */
    sinkImplementation.clearContext(&sinkImplementation, 0);
    sinkImplementation.updateDevice(&sinkImplementation, nullptr);
}

We can use a state machine to model the behaviour of our button gesture:

AM exercise

  • The INIT state corresponds to the application start-up
  • When the hand appears above a certain height, here 20cm, we move to the READY state.
  • Once the hand falls below 19cm, we call playSensation with our button click sensation and move to the CLICKED state.
  • When the hand moves backs above 20cm, we play a click sensation again and return to READY.

You can even create two, separate instances of the SensationPkgToFPGenAdapter class, one for each sensation to click in and click out with, each slightly different to give a different sensation for your button click.

Again, the code fragment below shows the while loop to implement the button clicking, including a statement to indicate button status, and with two separate sensation packages.

// Load a sensation file
SensationPackage pkgDataIn, pkgDataOut;
loadSensationFromFile("../Resources/data/tapOn.usp", pkgDataIn);
loadSensationFromFile("../Resources/data/tapOff.usp", pkgDataOut);

auto tracking = std::make_unique();
/* Instantiation of the sensation package to FPGen adapter class. */
Ultrahaptics::FPGen::SensationPkgToFPGenAdapter clickInSensastion(&pkgDataIn, tracking.get());
Ultrahaptics::FPGen::SensationPkgToFPGenAdapter clickOutSensastion(&pkgDataOut, tracking.get());

/* Instantiation of the hardware sink class. */
FPGenApp::AppSinkImplementation sinkImplementation;

// Get all the hand positions from the leap and position a focal point on each.
Ultrahaptics::LeapWrapper * leapWrapper = tracking.get();
int switchState = 0; // switch travel state
GESTURE_STATE gState = GESTURE_INIT; // enumerated type for our state machine

while (1) {
    if (leapWrapper->isTrackingDataAvailable()) {
        const Ultrahaptics::Alignment alignment;
        Eigen::Vector3f palmPos;
        Eigen::Vector3f handDir;
        Eigen::Vector3f palmNormal;

        leapWrapper->getExternalTrackingData(&alignment, palmPos, handDir, palmNormal);

        switch(gState) {
        case GESTURE_INIT:
            if (palmPos.z() > 20 * Ultrahaptics::Units::cm)
                gState = GESTURE_READY; // Hand is above
            break;
        case GESTURE_READY:
            if (palmPos.z() < 19 * Ultrahaptics::Units::cm) { 
                playSensation(clickInSensastion, 
                    sinkImplementation, pkgDataIn.stepsPerSecond); 
                gState = GESTURE_CLICKED; 
            } 
            break; 
        case GESTURE_CLICKED: 
            if (palmPos.z() > 20 * Ultrahaptics::Units::cm) {
                playSensation(clickOutSensastion, sinkImplementation, 
		pkgDataOut.stepsPerSecond);
                switchState = (switchState + 1) & 0x1;
                std::cout << "State: [" << ((switchState == 0) ? "Off" : "On") << "]\n";
                gState = GESTURE_READY;
            }
            break;
        }
    }

    std::this_thread::sleep_for(std::chrono::milliseconds(10));
}

Once built you can generate as many different sensations as you wish to perfect your button haptic. You can experiment with adding audible and visual cues to reinforce the haptic effect.

Tip: Follow our tutorial to learn how to use the Sensation Editor tool

Was this helpful?