Joint Angle Estimation for the Index Finger

Earlier today I implemented a way to map end-effector positions of the index finger from the Leap Motion controller to a target point visualized as a unit-radius sphere in Maya. Each frame update from the Leap Motion controller is first processed for the presence of fingers. If fingers exist, the fingers are sorted by their x-coordinate values. This facilitates selecting the index finger using index position '1' (the thumb lies in position '0', the pinky in position '4', etc.). Using this indexed array, we isolate the index finger, take its tip position and subtract the Leap's detected palm position from this value. The result gives us the direction vector of the index finger. 

The direction vector is useful for a number of reason. First, it describes the way the finger is positioned relative to the palm in space. Second, its length gives us information about the bend of the finger. We consider the length of the finger with an open, flat palm (all fingers extended) to be our base length. In every frame, we can use the current length of the direction vector to determine a target position for the  sphere in Maya. The smaller the length of the direction vector, the more likely the finger is to be bent. 

With this length aspect in mind, I devised a way to get a base length of the index finger. This is the length that will serve as the means of comparing all subsequent lengths from direction vectors. The purpose of the comparison will will become clearer in a moment. Calculating the base length is simple a process that samples the first 1000 frames in which fingers are present, determines their direction vectors, computes their lengths and average all of the values for each individual finger.  After the average length is calculated, the process of relaying position updates to Maya remains the same; however, there is one additional piece of information. We now send along lengthRatio, the ratio of the finger's length in the current frame to the base length. 

In Maya, we maintain the length of the joint chain (this can be calculated at run time if need be). When we receive a position update from the Leap with the direction vector and a lengthRatio, we normalize the direction vector and multiply it by the product of joint chain length and the lengthRatio.  We now need to obtain the position of the target point relative to the base of the joint chain using this direction vector. This is accomplished by adding the position of the joint chain base to the direction vector. The result is now a target positon mapped from the Leap coordinate space to Maya coordinate space with respect to the joint chain. We apply a translation transformation to the target point sphere using the  (xform) command. This then triggers a function call to perform Cyclic Coordinate Descent for estimating the joint bend angles. 

The reasoning behind using the lengthRatio, is simply to allow us to map a wider range of motion into Maya. Applying only the joint chain length to the unit direction vector tells us nothing about how a finger is actually bent. In fact, its sole application keeps the position of the target point on the surface of a sphere with a radius equal to the joint chain length and base at the joint chain base. The lengthRatio, on the contrary, describes how the target point sphere should be positioned on the joint chain sphere's surface (ratio equal to 1) or inside of the sphere (ratio is less than 1). 

One potential issue with this method is that the sampling/averaging process is (of course) not an exact measurement. Namely, if the user loosens their hand in the initial sampling process, the average length will be smaller than it should be. When the user decides to fully extend their fingers and open their palm during normal frame updates, the ratio will be greater than 1 causing positioning of the target point to be out of reach with respect to the joint chain length. This is a minor issue- we easily handle this occurrence using a projection procedure. If the position of a target point lies outside of the reach of the joint chain, we project that point onto its respective location on the surface of a sphere that has a radius equal to the joint chain length and a center at the base of the joint chain.

Testing the Functionality

In an effort to understand the information obtainable through the Leap Motion controller, I tested out the functionality using the Software Development Kit's (SDK) visualization program. It provides a decent interface for understanding and visually showing information the controller is receiving. Before I talk about what sort of information I am able to receive, it is probably useful to discuss some basic information about the structure of the hand and its phalanges (fingers):

Phalangeal Range of Motion

The Finger Structure

The hand has a very complex structure permitting a wide range of motion to each of its appendages. Additionally, each finger has its own motional complexity. The finger can divided into three phalanges bones that are connected by collateral ligaments. Ligaments are the tissues that connect bones to other bones. In the case of the finger, they provide the mechanism of joints. As an example, in the figure on the left, the collateral ligament between the metacarpal bone and the 1st phalanx forms what is typically called the 'knuckle'.

From the finger structure, there exist three possible joints that theoretically can be independently moved. This is the case for all of the fingers. The thumb, however, only has a knuckle joint and one additional joint connecting its two phalanges. That is not to say that the thumb is less complex. Its opposability gives it a much larger range of motion and thereby greater complexity.

Returning to the Leap- I used their visualization that depicts effectors as cylinders. The basic method displays them as vectors in 3D space, but that is not as helpful when analyzing movements relative to each finger. The cylindrical representation also makes it easier to see relative lengths of the fingers (widths, unfortunately, do not currently reflect the actual widths of the fingers). 

There is a good amount of information that can be visualized (as well as received in terms of data from the Leap). In particular, it is possible to see the palm normal of the hand. When moving the separate fingers, this information is useful because we can tell how fingers can move relative to a (presumably) stationary plane. Angle calculations, at the very least, can be made with respect to the knuckles. 

Palm Normal

Fingers Visualized as Cylinders

There is an interesting problem to solve in terms of how the direction of the effector (the finger vector) can be used to achieve the joint angles present along the other two bend angles on each of the fingers (and the one angle of the thumb). 

To begin, I will most likely assume that the joint angle move in sync. This assumption will allow me to interpolate bend angles along the finger to at least get a working prototype. From there, I will attempt to find a mechanism for achieving better estimates of all of the joint angles. This may result in the use of multiple Leap devices to capture information from different view angles.

Another bit of information that is potentially useful in detecting joint angles is the palm sphere radius. This works by estimating the radius of a spherical object that could be held by the hand. A closed hand produces a smaller sphere radius while a large one produces a larger radius. This information is useful because a smaller radius implies a closed hand, thereby the fingers must be angled downward. I'm not sure how this can be used for joint angles, but that is something I will find out.  I will end here with some goals for this week:

Open Palm

Closed Palm

1. Finish my proposal for Project PAALM.
2. Start reading through the research on hand motion detection devices and hand tracking.
3. Start working on a prototype that will at least give some information about estimated joint angles. I will work towards just visualizing and estimating angles for a single finger (probably the index).