Joint Angle Estimation for the Index Finger

Earlier today I implemented a way to map end-effector positions of the index finger from the Leap Motion controller to a target point visualized as a unit-radius sphere in Maya. Each frame update from the Leap Motion controller is first processed for the presence of fingers. If fingers exist, the fingers are sorted by their x-coordinate values. This facilitates selecting the index finger using index position '1' (the thumb lies in position '0', the pinky in position '4', etc.). Using this indexed array, we isolate the index finger, take its tip position and subtract the Leap's detected palm position from this value. The result gives us the direction vector of the index finger. 

The direction vector is useful for a number of reason. First, it describes the way the finger is positioned relative to the palm in space. Second, its length gives us information about the bend of the finger. We consider the length of the finger with an open, flat palm (all fingers extended) to be our base length. In every frame, we can use the current length of the direction vector to determine a target position for the  sphere in Maya. The smaller the length of the direction vector, the more likely the finger is to be bent. 

With this length aspect in mind, I devised a way to get a base length of the index finger. This is the length that will serve as the means of comparing all subsequent lengths from direction vectors. The purpose of the comparison will will become clearer in a moment. Calculating the base length is simple a process that samples the first 1000 frames in which fingers are present, determines their direction vectors, computes their lengths and average all of the values for each individual finger.  After the average length is calculated, the process of relaying position updates to Maya remains the same; however, there is one additional piece of information. We now send along lengthRatio, the ratio of the finger's length in the current frame to the base length. 

In Maya, we maintain the length of the joint chain (this can be calculated at run time if need be). When we receive a position update from the Leap with the direction vector and a lengthRatio, we normalize the direction vector and multiply it by the product of joint chain length and the lengthRatio.  We now need to obtain the position of the target point relative to the base of the joint chain using this direction vector. This is accomplished by adding the position of the joint chain base to the direction vector. The result is now a target positon mapped from the Leap coordinate space to Maya coordinate space with respect to the joint chain. We apply a translation transformation to the target point sphere using the  (xform) command. This then triggers a function call to perform Cyclic Coordinate Descent for estimating the joint bend angles. 

The reasoning behind using the lengthRatio, is simply to allow us to map a wider range of motion into Maya. Applying only the joint chain length to the unit direction vector tells us nothing about how a finger is actually bent. In fact, its sole application keeps the position of the target point on the surface of a sphere with a radius equal to the joint chain length and base at the joint chain base. The lengthRatio, on the contrary, describes how the target point sphere should be positioned on the joint chain sphere's surface (ratio equal to 1) or inside of the sphere (ratio is less than 1). 

One potential issue with this method is that the sampling/averaging process is (of course) not an exact measurement. Namely, if the user loosens their hand in the initial sampling process, the average length will be smaller than it should be. When the user decides to fully extend their fingers and open their palm during normal frame updates, the ratio will be greater than 1 causing positioning of the target point to be out of reach with respect to the joint chain length. This is a minor issue- we easily handle this occurrence using a projection procedure. If the position of a target point lies outside of the reach of the joint chain, we project that point onto its respective location on the surface of a sphere that has a radius equal to the joint chain length and a center at the base of the joint chain.

Cyclic Coordinate Descent & Forcing Maya UI Updates

Previously, I described my plan to implement an algorithm called Cyclic Coordinate Descent (CCD) and integrate it into Maya. The process was cumbersome, but I am happy to report that my attempt was successful. 

The largest source of trouble originated from understanding Maya's construction of composite matrices for object transformations and the various application programming interfaces for data types like Matrices and Quaternions. Initially, I went the route of using the Leap Motion SDK data types (Leap Vectors, Matrices, etc) to calculate the rotation angles for a joint while performing CCD. However, problems arose when I attempted to translate Maya's composite matrix (as an array of double values) into Leap Matrices and then back to a Maya Matrix after applying a rotation. Artifacts would result in the Leap Matrix that I could not remedy because the Leap SDK does not permit direct access to the data backing a matrix. 

After discovering the artifacts, I delved into Maya's data types but encountered more trouble because their interface for matrices has some of the same limitations that the Leap data types have- namely, direct access to matrix values is not possible. My latest (and successful) shift was to PyMel, a Python wrapper around Maya commands that adds the functionality that I needed.

With PyMel in place, the only other feat was Gimbal Lock. Gimbal Lock is an issue that results when using Euler angles for the rotation of 3D objects in space. Essentially, successive rotations can result in two rotation axes locking in the same plane. Subsequent rotations about these locked axes causes a single rotation to applied to the object with respect to that plane. Effectively, the object loses a rotational degree of freedom.

On the left, you see how each gimbal allows rotation around 

a specific axis. On the right, you can see a set of gimbals in 

gimbal lock. The inner-most gimbal can't change in pitch 

unless the gimbals are put into another positions. 

[Source: HowStuffWorks]

There are a number of ways to avoid Gimbal Lock such changing the order of rotation about axes after detecting when a sequence of rotations is likely to cause Gimbal Lock. I decided to use axis-angle rotation pairs and convert those into quaternions. There are two benefits to using this approach. The first is obviously that quaternions avoid Gimbal Lock;  they permit rotations to occur about an arbitrary axis in space. The second benefit (though not currently in use), is that they permit 3D rotation to be animated smoothly (Spherical Linear Interpolation).

As a secondary update, I have changed the way in which the Leap Motion controller interfaces with Maya. Originally, a listener object would compute position and rotation updates then send them in Maya Embedded Language (MEL) as a string over a socket command port linked to Maya. This implementation meant that I was performing CCD from within the Python script running in my Bash. Thus, references of the rotation and positions of joints had to be queried each time a transformation on a particular joint in the chain occurred. The logic behind this rests in the fact that an arbitrary rotation on a single joint node in a chain causes the positions of all connected children joints nodes to rotate as well; this propagation is a feature of the structure of a Scene Graph. 

My currently implementation places all CCD and any operations for joint transformations within the Maya python script. The Leap controller listener is only sending coordinates of end-effector positions to Maya as a string. Queries, since they are now occurring within Maya and not over a socket connection, are much faster.

A few posts back I also described an issue in the Maya UI halting on position updates. I have now found a way to solve this problem- Maya has a command called refresh permits the input of a force boolean. When force is true, the command triggers a complete update of the camera views displaying the objects in a scene. I am currently using this to visualize intermediate stages of the CCD algorithm.

References
1) Cyclic Coordinate Descent (CCD)
2) PyMel - Python-Maya Wrapper Application Programming Interface
3) YouTube: Gimbal Lock
4) How Stuff Works (Image): Gimbal Lock
4) Wikipedia: Euler angles
5) Wikipedia: Quaternions
6) Wikipedia: Spherical Linear Interpolation (Slerp)
7) Maya Embedded Language (MEL)
8) Wikipedia: Scene Graph
9) PyMel API - Refresh Command

Alpha Review Recap

This is a brief update that I will follow-up with later on this weekend.

Alpha reviews occurred right before Spring Break last week. The feedback I received was very positive and helpful in moving forward. The steps I am currently taking  are implementing Cyclic Coordinate Descent attached to a time node in Maya. This will allow iterations of CCD to be slowly visualized in Maya to permit testing whether my current implementation works properly.

Until now, I have not been able to directly test my method's efficacy using the Leap because of the large amount of rapid frame updates that continually update position in Maya. I plan to have a working time node set up in Maya by the end of this weekend. Once this is completed I will update with more details.