Week 7, Part 1: 30 March - 6 April [Leap Motion, Research, Testing]

-

To keep building on the focus of the project, making the application as interactive as possible, I was keen to include Leap Motion as a method of input.

When I began using Kinect, I found dedicating some time to understanding how the device functions allowed me to approach the problem with knowledge of specifically what the device itself needed and identify potential issues before beginning development

Similarly, with Leap Motion, while I understand what the peripheral does, though I’m not fully aware of how the device reads and processes data.

Leap Motion key components
Leap Motion uses three infrared LEDs to project (a hemisphere) above the device. It then uses two wide-angle-lensed cameras to detect when objects are hit by the infrared. This is the same approach the Kinect uses to map its surroundings.
From reading I found Leap Motion then applies algorithms to the sensor’s raw data to map hand movements - rather than generate a depth map as the Kinect had done.

The device then allows users to interact with an application through gestures, with a high level of accuracy.

The LEDs project infrared over an ‘interaction zone’ of 8 cubic feet above the device (2 feet above the controller, 2 feet either side).
The interaction zone is limited to this size due to how infrared propagates through space as it becomes much harder to determine the position of the hand in 3D beyond this distance.
With greater power the effective range could be increased, however the intensity of the LED lights is limited by the current that can be drawn via a USB connection.

In 2016 Leap Motion released new software, called ‘Orion’, built specifically for VR. This also increased the effective interaction space above the device by 33%.
As a team, later in the projects lifetime using VR is central to our applications design. Knowing that Leap Motion is being actively developed with VR in mind is encouraging.

-

After doing this introductory reading to learn how the equipment functions, I moved onto installing the device ready for use.

Leap Motion have made their software easily available and promote its use with unity. An asset pack is available from their website’s developer portal which can be imported directly into unity with ease.
It immediately provides several example scenes, scripts and prefabs which showcase basic functionality.

By default, hands are tracked and displayed in the scene. Displayed hands have a rigidbody component, so can interact with other scene objects using basic physics.

Having access to and being able to analyse these scripts was a huge help in understanding how to implement the functionality I wanted.

I was then able to implement some basic functionality myself. Building and testing these experiments highlighted some potential issues, as well as solutions for future development.

The first issue I encountered was the orientation and position of the device itself. Initially hands were being tracked accurately, though along the wrong axis. I attempted to adjust the prefab rotation in Unity to correct this, both in the inspector and in code, though the problem persisted. After searching forums I soon realised that the device orientation can be specified (horizontal/vertical) within Unity, depending on whether the device is being used standalone (horizontal) or attached to a headset (vertical).
I then found the leap can calibrate its Z-axis based on initial input in a session (so if flat on a desk it does not matter which way forward is).
Through testing, which I then confirmed through reading the Leap Motion documentation, the device cannot be placed facing the user as the algorithm also tracks forearms to inform where hands are expected to appear. If the device faces the player in vertical orientation, hand tracking is very unreliable.

Smudges on the device can also hinder accuracy of detection and tracking, although the device provides notifications warning of smudges if one is detected. These notifications have so far always been reliable (has always been a smudge when waned of one) and seem to be sent to the user very rapidly once the smudge has occurred.

Leap Motion produces a grayscale image as it tracks in near-infrared:
Leap Motion produced greyscale image (skeletal hand respresentations are overlayed)

I found that objects which reflect infrared light, can interfere with detection and tracking of hands and fingers, causing one or both hands to become untracked until the reflection onto the device is removed.

Other infra-red sources (halogens and sunlight to name two) can also negatively affect the accuracy of readings. For the best results only infra-red emitted from the Leap Motion should illuminate the hands to be tracked. ‘Ghost hands’ can appear and take priority over the user’s hands, replacing them in the scene, if additional infra-red sources are strong enough.


After these experiments, I looked into what other developers have been able to create using the device.

This open source menu system “Hover-UI-Kit” (Github/Hover-UI-Kit/wiki, 2018) shows how fluid and accurate the device is.
Using the Hovercast interface with Leap Motion hands

As shown in the above gif, using gestures to trigger the opening of an interactable menu, track individual joints and bones independently, then allow natural hand movements to interact with the menu contents seems to be the most intuitive implementation of Leap Motion I have seen. As the application will need to be accessible to a range of users, I will try to apply an approach in my own design, Inspired by the Hover-UI-Kit.



References
  • Colgan, A. and Colgan, A. (2018). How Does the Leap Motion Controller Work?. [online] Leap Motion Blog. Available at: http://blog.leapmotion.com/hardware-to-software-how-does-the-leap-motion-controller-work/ [Accessed 26 Mar. 2018].
  • Developer.leapmotion.com. (2018). C# SDK Documentation — Leap Motion C# SDK v2.3 documentation. [online] Available at: https://developer.leapmotion.com/documentation/v2/csharp/index.html [Accessed 27 Mar. 2018].
  • Developer.leapmotion.com. (2018). Leap Motion Unity Assets and Plugin — Leap Motion Unity SDK v2.3 documentation. [online] Available at: https://developer.leapmotion.com/documentation/v2/unity/index.html [Accessed 27 Mar. 2018].
  • En.wikipedia.org. (2018). Leap Motion. [online] Available at: https://en.wikipedia.org/wiki/Leap_Motion [Accessed 26 Mar. 2018].
  • GitHub. (2018). aestheticinteractive/Hover-UI-Kit. [online] Available at: https://github.com/aestheticinteractive/Hover-UI-Kit/wiki [Accessed 25 Mar. 2018].
  • Medium. (2018). How Does the Leap Motion Controller Work? – LeapMotion – Medium. [online] Available at: https://medium.com/@LeapMotion/how-does-the-leap-motion-controller-work-9503124bfa04 [Accessed 26 Mar. 2018].

Comments

Post a Comment

Popular posts from this blog

Week 3: 2 - 9 March [Continued AR prototype]

-
Image
I have spoken with the university’s 3D technologist, who is happy to assist with printing the airship model, but does not have the capacity to do so for several weeks. Until I can print a model I will continue to develop the prototype started last week. Instantiating model versus instantiating model with shadow shader. To add an element of realism to the instantiated airship, I added a plane below the model so the light source within the scene would be blocked by the model, casting a shadow onto the plane. The shadow had very poor definition, not appearing to look even vaguely similar the model, and often went unnoticed against grey/white backgrounds. To improve this, I added a shader to the airship model, which casts a much more well-defined shadow. I also updated the components of the plane within Unity, setting its alpha value to 0, making it transparent. This still casts a shadow onto the planes surface, while making it appear to the user that the shadow is cast direc...
Read more

Week 7, Part 2: 30 March - 6 April [Leap Motion, Prototype]

-
Image
Thinking of possible uses of the Leap Motion in the application, I ruled out a pop-up menu as this could only be used to trigger scene changes – not the level of engagement the project requires. Despite this, I was still able to draw inspiration from the ‘Hover-UI-Kit’ menu I experimented with in the previous blog entry, in Unity. Using a gesture to activate a ‘mode’ then more intuitive movements within the mode for further functionality. Using the hand to select parts of the airship to then display related information was the idea I decided to develop. This could be done with relative ease through raycasting out from the hand prefab to detect colliders on a ship model. I used the last bone in the index finger for the origin of the ray, casting the ray forward in the direction of the bones Z-axis. Once this functionality was implemented it needed to be made toggleable, triggered by a user gesture. Leap Motion provides an example script to detect predefined hand positions. Th...
Read more

Week 1: 16 - 23 Feb. [Meeting with Museum Director]

-
Image
While I was intrigued by both the energy efficient car and the museums interactive experience, the prospect of working with new technology directly related to the content of the degree, where people outside of the university would be the end users made the museum’s project far more appealing. Diss museum had reached out to the university for assistance with their upcoming exhibition, to commemorate the successful double Atlantic crossing of the R34 airship. No brief or requirements were specified in the museum’s request, so after learning of the project I contacted the museum manager, Basil Abbott, to arrange a meeting to further clarify what they hoped to include in their exhibition and available resources. From initial email exchanges it was clear that the intention was to involve the community with a hands-on experience(s) rather than a static exhibition, the implementation of which the museum has left to our discretion. To prepare for the meeting it was necess...
Read more