VirtualInputExample.cpp

This example shows how one can inject virtual input into a Cornerstone application. In this example we create a simple application with only one widget, namely Sparkles. The motivation for this is that Sparkles shows nicely where our simulated input is located.

Screenshot of the VirtualInput example

The key element in virtualising input is the Application's before-input-event. That is sent after input is gathered across the network but not yet processed by widgets. This is the moment when we want to inject artificial input into the input samples. This event sends sample indices as its parameters. In order for input to be properly handled we need to push our generated input to each sample.

Main function of this example is relatively simple. We create Sparkles-widget to the background by using plugin-machinery built into Cornerstone. After this we create an instance of HandMovement. It is added to listen Application's after-update event in order to update the location simulated hand.

int main(int argc, char** argv)
{
  MultiWidgets::Application app;

  if(!app.init(argc, argv))
    return 1;

  auto sparkles = MultiWidgets::createPlugin("cornerstone.sparkles");
  app.mainLayer()->addChild(sparkles);
  sparkles->setSize(app.mainLayer()->size());

  Examples::HandMovement hm;

  app.eventAddListener("after-update", [&hm] {
    hm.update();
  });

  return app.run();
}

HandMovement-class simulates a finger that makes circles at the middle of the screen. It has an instance of MultiWidgets::VirtualInput as its member. This class acts as an user friendly interface for managing virtual input. It is enough for the user to create appropriate VirtualObject's and pass them to instance of VirtualInput. The rest of the HandMovement's members are used for keeping track of the finger movements and the state of the finger.

  private:
    MultiWidgets::VirtualInput m_virtualInput;
    MultiWidgets::VirtualHandPtr m_virtualHand;

    State m_state;

    Radiant::Timer m_timer;
    float m_angle;

    enum State { FINGER_DOWN, FINGER_UP, MOVE_FINGER, WAIT};

The function that updates the location of the virtual hand is split into four different branches of execution. The choice of branch depends on the value of m_state. All of the branches have common initialization where the current position of the finger is calculated.

    void update()
    {

      MultiWidgets::Application& app = MultiWidgets::Application::instance();
      Nimble::Vector2f center = app.mainLayer()->size().toVector() * 0.5f;

      // Finger position on circle centered on middle of the screen
      Nimble::Vector2f onCircle = 200.f*Nimble::Vector2f(cosf(m_angle), sinf(m_angle));
      Nimble::Vector2f fingerPos = center + onCircle;

First call to function initializes the hand. We create virtual hand with single finger using function MultiWidgets::VirtualHand::fingerDown. After that we register our newly created hand to VirtualInput and update the internal state of the object.

      if(m_state == FINGER_DOWN) {
        if(!m_virtualHand) {
          m_virtualHand = MultiWidgets::VirtualHand::fingerDown(fingerPos);
        }
        m_virtualInput.addObject(m_virtualHand);
        m_state = MOVE_FINGER;
      }

When moving finger we specify the position of the whole hand in relation to finger we want to control. This can be done using function MultiWidgets::VirtualHand::setLocationAboutFinger. Finally we check if hand is already made a full circle and update the state if needed.

      else if(m_state == MOVE_FINGER) {
        m_virtualHand->setLocationAboutFinger(0, fingerPos);
        if(m_angle > Nimble::Math::TWO_PI) {
          m_state = FINGER_UP;
        }
      }

Lifting the simulated finger up is simple. The only thing needed to do is to unregister our VirtualHand from VirtualInput.

      else if(m_state == FINGER_UP) {
        m_virtualInput.removeObject(m_virtualHand->id());
        m_state = WAIT;
      }

The rest of synthesizeInput-function detects when it is time to continue the movements of simulated finger. Also the angle is updated and associated timer is resetted.

The full source code for the complete application is listed below:

#include <MultiWidgets/Application.hpp>
#include <MultiWidgets/Plugins.hpp>

#include <Nimble/Math.hpp>
#include <Radiant/Timer.hpp>
#include <MultiWidgets/VirtualInput.hpp>

namespace Examples
{

  class HandMovement
  {
  public:

    enum State { FINGER_DOWN, FINGER_UP, MOVE_FINGER, WAIT};

    HandMovement()
      : m_virtualHand(nullptr),
        m_state(FINGER_DOWN),
        m_angle(0)
    {
      m_timer.start();
    }

    void update()
    {

      MultiWidgets::Application& app = MultiWidgets::Application::instance();
      Nimble::Vector2f center = app.mainLayer()->size().toVector() * 0.5f;

      // Finger position on circle centered on middle of the screen
      Nimble::Vector2f onCircle = 200.f*Nimble::Vector2f(cosf(m_angle), sinf(m_angle));
      Nimble::Vector2f fingerPos = center + onCircle;

      if(m_state == FINGER_DOWN) {
        if(!m_virtualHand) {
          m_virtualHand = MultiWidgets::VirtualHand::fingerDown(fingerPos);
        }
        m_virtualInput.addObject(m_virtualHand);
        m_state = MOVE_FINGER;
      }

      else if(m_state == MOVE_FINGER) {
        m_virtualHand->setLocationAboutFinger(0, fingerPos);
        if(m_angle > Nimble::Math::TWO_PI) {
          m_state = FINGER_UP;
        }
      }

      else if(m_state == FINGER_UP) {
        m_virtualInput.removeObject(m_virtualHand->id());
        m_state = WAIT;
      }

      else if(m_angle > 2*Nimble::Math::TWO_PI) {
        m_state = FINGER_DOWN;
        m_angle = 0.f;
      }

      m_angle += Nimble::Math::PI * m_timer.time();
      m_timer.start();
    }

  private:
    MultiWidgets::VirtualInput m_virtualInput;
    MultiWidgets::VirtualHandPtr m_virtualHand;

    State m_state;

    Radiant::Timer m_timer;
    float m_angle;
  };

}

int main(int argc, char** argv)
{
  MultiWidgets::Application app;

  if(!app.init(argc, argv))
    return 1;

  auto sparkles = MultiWidgets::createPlugin("cornerstone.sparkles");
  app.mainLayer()->addChild(sparkles);
  sparkles->setSize(app.mainLayer()->size());

  Examples::HandMovement hm;

  app.eventAddListener("after-update", [&hm] {
    hm.update();
  });

  return app.run();
}