Using Third Party Software with Cornerstone

MultiTouch Cornerstone can output tracking information using different protocols, like TUIO and XML.

Using these protocols, it is possible to interface with external software.

TUIO Forwarding

Any Cornerstone application can be configured to send computer vision results as a TUIO stream over a UDP socket. The stream includes finger tips and marker information as standard TUIO messages. Additionally hands can be encoded using custom extension (see tphw_tuio_forward_hand_extension).

To enable TUIO forwarding, you must add the following block to your application config.txt file:

TUIOSender {
    /* The features that should be enabled, options are:
       "fingers" - Send finger tip locations
       "hands" - Enable the TUIO Hand Extension and send palm / finger connectivity information.
       "objects" - Send marker information
       "silent" - Do not print debug messages to console
       "verbose" - More verbose debug messages printed to console
       "objects_as_fingers" - Markers are sent over TUIO as if they were fingers
       To enable multiple features, just put multiple keywords into the field.
       The default value is "fingers objects" */
    features = "fingers objects"
    /* address: The TUIO stream receiver IP network address, default = 127.0.0.1 */
    address = "localhost"
    /* port: The TUIO stream receiver IP port, default = 3333 */
    port = "3333"
}

In addition you the block above, you need to configure your sources normally, e.g. using one or more NetBridge blocks.

TUIO Hand Extension

In addition to just finger tips, Cornerstone has the ability to track whole hands with finger connectivity information. By default, TUIO does not support this feature. However, it is possible to use a custom extension with TUIO that allows hand information to be transmitted with TUIO protocol.

The event format is as follows:

• message address "/tuio/_hand"
• "set" (string)
• id of the hand (32-bit integer)
• palm location coordinates (x/y, 32-bit floating-point numbers normalized to [0, 1] range)
• palm motion vector (x/y, 32-bit floating-point numbers normalized to [0, 1] range)
• palm movement amount (32-bit floating-point number, length of the motion vector)
• finger ids (five 32-bit integers giving the IDs of the fingers attached to the hand. If there are fewer than five fingers, the IDs are set to -1)

To enable the hand extension, you must add the hands to the features parameter of tphw_tuio_forward TUIOSender:

TUIOSender {
  features = "fingers hands"
}

TUIO Forward with Multiple Cells

It is possible to merge the output of multiple MultiTaction Cells into a single TUIO stream. Cornerstone will handle the ID conflicts for tracked objects and output a single normalized TUIO stream that can be used directly. The application using the TUIO stream does not need to know that the TUIO stream is coming from multiple sources.

To use this feature, just define multiple NetBridge blocks inside your config.txt that describe the sources with correct input transformation parameters. Then define a single TUIOSender block normally. Cornerstone will then automatically combine the input from all NetBridge blocks and merge it into a single TUIO stream. The combined resolution is automatically calculated by using the bounding box of all input sources.

An example of merging the output of two MultiTaction Cells in landscape mode into a single TUIO stream is shown below. You can use eg. MTServer with this config file.

NetBridge {
  host = "10.0.0.100"
  input-translate = "0 0"
}

NetBridge {
  host = "10.0.0.101"
  input-translate = "1920 0"
}

TUIOSender {
  address = "localhost"
  port = "3333"
}

XML Stream

MultiTouch Cornerstone can output tracking information as XML using TCP. The XML stream is meant to be easy to parse with practically any programming language. Please be aware that since XML is text-based protocol and rather verbose, parsing it may add some overhead.

Enabling XML Stream

To enable the XML stream server, define the xml-server-host and xml-server-port variables in the Globals section of your config.txt:

Globals {
 /* Hostname for the XML server socket. */
  xml-server-host = "localhost"
  /* Port number for the XML server socket. */
  xml-server-port = "5501"
}

When you start any MultiTouch Cornerstone application, the XML server is started automatically. It will listen for connections on the defined TCP port. Connecting to the port will automatically start data transmission.

Example XML Stream Data

The XML stream is human-readable and should be easy to parse. The data will look like this:

<sample>
<finger><id>1</id><age>63</age><loc><x>506.21</x><y>612.48</y></loc></finger>
<finger><id>2</id><age>62</age><loc><x>402.90</x><y>726.62</y></loc></finger>
<finger><id>3</id><age>62</age><loc><x>593.97</x><y>618.48</y></loc></finger>
<finger><id>4</id><age>62</age><loc><x>644.36</x><y>664.68</y></loc></finger>
<finger><id>5</id><age>62</age><loc><x>679.96</x><y>748.62</y></loc></finger>
<hand><id>0</id><age>63</age><palm><x>530.23</x><y>771.51</y></palm><fingerid>1</fingerid><fingerid>2</fingerid><fingerid>3</fingerid><fingerid>4</fingerid><fingerid>5</fingerid></hand>
</sample>
<sample>
<finger><id>1</id><age>64</age><loc><x>506.19</x><y>612.54</y></loc></finger>
<finger><id>2</id><age>63</age><loc><x>402.91</x><y>726.68</y></loc></finger>
<finger><id>3</id><age>63</age><loc><x>593.97</x><y>618.54</y></loc></finger>
<finger><id>4</id><age>63</age><loc><x>644.34</x><y>664.73</y></loc></finger>
<finger><id>5</id><age>63</age><loc><x>679.96</x><y>748.65</y></loc></finger>
<hand><id>0</id><age>64</age><palm><x>530.22</x><y>771.63</y></palm><fingerid>1</fingerid><fingerid>2</fingerid><fingerid>3</fingerid><fingerid>4</fingerid><fingerid>5</fingerid></hand>
</sample>