The first is the one for which this research blog is named- Nuisical (pronounced like 'musical'). This was my first idea focusing on producing a Natural User Interface for Sound Interaction and Composition. I have been pondering it for quite some time, but after finding that a great deal has been done in similar respects using the Microsoft Kinect (see the V Motion Project), I am not certain the endeavor would (1) contribute much to the field, or (2) be interesting enough to propel me through the semester-long project.
That said, I have still been doing research on the topic, exploring Application Programming Interfaces (APIS) for the Kinect and audio platforms, documentation on NUIs. I intend on attempting basic prototypes once I gain access to the Center for Human Modeling and Simulation, but that may not begin until some time next week (ID-card issues).
My second project idea is an extension of a multi-user American Sign Language gesture translator prototype, called Social Sign, that I built with 2 other individuals during the 40-hour PennApps Spring 2013 hackathon. Social Sign used the new Leap Motion controller, a new gesture-recognition technology that has unparalleled detection of nuanced hand and finger movements. There is great promise for building responsive and easy to use NUIs using the device.
The extension, for which I am currently working the details out, focuses on using the Leap Motion controller as a device to detect joint angles in hand and finger movements. To a degree, this information can be captured using developments in wearable globes but the gloves do not provide enough information and restrict hand movement. The feat of the Leap Motion device is that it is entirely free of hand obstruction and is roughly the size of two-stacked packs of gum. That is, it permits a user to move their hand freely. With information such as joint angles, rigging hand models for character animations could be as simple as waving a hand over the Leap Motion device.
I have been combing through the Leap Motion API to find out what sorts of data the device can return so that I can determine if it will be useful for obtaining joint angles. So far I have come up with the following:
- List of fingers/pointables visible - (objects are declared as tools or fingers)
- Finger: has a reference frame, id, and associated hand. The ID is unique while hand is still in view, but it is lost once the object disappears for the Leap’s visual field.
- Frame: contains references to all the hands, fingers and pointables in the scene. Also can tell you the rotation matrix/angle/axis between two frames. Scale factor, Translation vector.
- List of Hands visible
- Hands have direction, fingers, an id, a frame.
- Palm-normal, palm velocity, palm position
- Rotation angle/axis/matrix
- Sphere Center, Sphere Radius (as if hand was holding an invisible ball).
- Translation
One of the difficult tasks in dealing with this data from the Leap Motion, at least from what I understand, is that fingers are perceived as straight vectors, regardless of their bending. I need to do a bit more research into this. It may be the case that from the information provided, joint angles may be an impossibility. Tomorrow I will be meeting with one of my teammates to discuss the project further. Based on that conversation, my decision for switching may be finalized. (If that's the case, my blog will be re-locating to a more appropriate domain name).
For now, I have put together some of the materials I will be reviewing over the next week for both projects, until I have a clearer sense of which one is more feasible for the semester and more importantly, interesting to me.
Nuisical Resources
- Music & Computers - an online resource book that digs into the aspects of digital sound and computation.
- Brave NUI World - just received this book by a designer and researcher of the Microsoft Surface that describes methods for developing NUIs.
- KinectSpace - a new discovery which allows a user to train gestures into the Kinect for later recognition.
Gesture-Recognition and Translation for Sign-Language
- Leap Motion API (I would link to this- but it is currently closed to only authorized developers)
- An Approach to Glove-Based Gestured Recognition
- Dynamic Hand Gesture Recognition Using Predictive EigenTracker
- A Revealing Introduction to Hidden Markov Models
- Kalman Filters
No comments:
Post a Comment