Cyclic Coordinate Descent & Forcing Maya UI Updates

Previously, I described my plan to implement an algorithm called Cyclic Coordinate Descent (CCD) and integrate it into Maya. The process was cumbersome, but I am happy to report that my attempt was successful. 

The largest source of trouble originated from understanding Maya's construction of composite matrices for object transformations and the various application programming interfaces for data types like Matrices and Quaternions. Initially, I went the route of using the Leap Motion SDK data types (Leap Vectors, Matrices, etc) to calculate the rotation angles for a joint while performing CCD. However, problems arose when I attempted to translate Maya's composite matrix (as an array of double values) into Leap Matrices and then back to a Maya Matrix after applying a rotation. Artifacts would result in the Leap Matrix that I could not remedy because the Leap SDK does not permit direct access to the data backing a matrix. 

After discovering the artifacts, I delved into Maya's data types but encountered more trouble because their interface for matrices has some of the same limitations that the Leap data types have- namely, direct access to matrix values is not possible. My latest (and successful) shift was to PyMel, a Python wrapper around Maya commands that adds the functionality that I needed.

With PyMel in place, the only other feat was Gimbal Lock. Gimbal Lock is an issue that results when using Euler angles for the rotation of 3D objects in space. Essentially, successive rotations can result in two rotation axes locking in the same plane. Subsequent rotations about these locked axes causes a single rotation to applied to the object with respect to that plane. Effectively, the object loses a rotational degree of freedom.

On the left, you see how each gimbal allows rotation around 

a specific axis. On the right, you can see a set of gimbals in 

gimbal lock. The inner-most gimbal can't change in pitch 

unless the gimbals are put into another positions. 

[Source: HowStuffWorks]

There are a number of ways to avoid Gimbal Lock such changing the order of rotation about axes after detecting when a sequence of rotations is likely to cause Gimbal Lock. I decided to use axis-angle rotation pairs and convert those into quaternions. There are two benefits to using this approach. The first is obviously that quaternions avoid Gimbal Lock;  they permit rotations to occur about an arbitrary axis in space. The second benefit (though not currently in use), is that they permit 3D rotation to be animated smoothly (Spherical Linear Interpolation).

As a secondary update, I have changed the way in which the Leap Motion controller interfaces with Maya. Originally, a listener object would compute position and rotation updates then send them in Maya Embedded Language (MEL) as a string over a socket command port linked to Maya. This implementation meant that I was performing CCD from within the Python script running in my Bash. Thus, references of the rotation and positions of joints had to be queried each time a transformation on a particular joint in the chain occurred. The logic behind this rests in the fact that an arbitrary rotation on a single joint node in a chain causes the positions of all connected children joints nodes to rotate as well; this propagation is a feature of the structure of a Scene Graph. 

My currently implementation places all CCD and any operations for joint transformations within the Maya python script. The Leap controller listener is only sending coordinates of end-effector positions to Maya as a string. Queries, since they are now occurring within Maya and not over a socket connection, are much faster.

A few posts back I also described an issue in the Maya UI halting on position updates. I have now found a way to solve this problem- Maya has a command called refresh permits the input of a force boolean. When force is true, the command triggers a complete update of the camera views displaying the objects in a scene. I am currently using this to visualize intermediate stages of the CCD algorithm.

References
1) Cyclic Coordinate Descent (CCD)
2) PyMel - Python-Maya Wrapper Application Programming Interface
3) YouTube: Gimbal Lock
4) How Stuff Works (Image): Gimbal Lock
4) Wikipedia: Euler angles
5) Wikipedia: Quaternions
6) Wikipedia: Spherical Linear Interpolation (Slerp)
7) Maya Embedded Language (MEL)
8) Wikipedia: Scene Graph
9) PyMel API - Refresh Command

Live Position Updates in Maya through the Leap Motion Controller

I postponed updating my research last week because I was not able to make progress in my implementation. This update, however, serves to highlight the results I have obtained solely during this weekend. 

My work has been on two components-the first involves using the Python Leap SDK to filter frame data to isolate a particular finger, and then using that finger data to obtain motion information (specifically, the end tip position of an effector after a motion); the second is integrating my data into Maya to update finger positions in real time.

Leap Motion Controller Coordinate System

I managed to implement a quick filtering algorithm that does basic sorting of fingers by x-position (relative to the Leap's coordinate system). In this manner, it is possible to order the fingers on the hand such that we can easily determine each of them. I selected the index finger as my finger of focus for this preliminary implementation. After obtaining the index finger information, I used the direction vector of motion to determine the finger's positioning in relation to the base of the finger. From the Leap perspective, the palm position was used as the base position. When mapping this information into Maya, the base position is taken to be the knuckle position.

Initially, I wrote my program in the style of a Maya plug-in using the Maya Python API. The idea was that it would be portable, effective, and fully integrated. I soon uncovered that this implementation had terrible performance effects. To elaborate, the Python Leap Motion SDK features a base class that permits a Python program to listen for 'frame' events. On a new frame capture from the Leap Motion controller, a function can be called to handle the frame data (this is where I integrated my filtering and sorting algorithm). Actual capturing of the frame data runs on a thread that only terminates after user input.

When running my script in Maya, the rapid data capture coupled with the waiting for a user input process halted the user interface (including the script editor). This is the result of scripts in the Maya running on the Main thread). When I terminated program through user input, all of the captured Leap data popped off the program call stack. Each command was then processed, but the finger tip position only snapped to the finger tip position of the most recent frame. More specifically, this was not real time and was not animated.  

I approached the problem by attempting to launch multiple threads for the processes in Maya and using different means of obtaining user input. Each attempt only resulted in Maya crashing or a complete UI halt. With the help of a friend, who found this resource for opening socket connections in Maya, I was able to open a command port, and use my Python program from within a bash (Terminal in OS X). 

The features of this approach are:

1. The open-socket permits commands to be run continuously without halting the UI in Maya. 
2. There is no need to install python modules/packages
3. Any version of Python can be used from the Bash. Likewise there are no limitations to what modules you use in the Bash. This is unlike the Maya version of Python that limits the use of modules and their functionality.

To demonstrate my progress, I have included a video showing the live updating of a finger joint position in Maya. The left pane is the Maya Editor window. The top-right pane is the Maya Script Editor. The bottom right pane is the Bash window that has my Python program running. It currently prints out the effector tip position for each frame capture.

My next goal is to implement the Cyclic Coordinate Descent algorithm to perform inverse kinematics and estimate the phalangeal joint angles.