The Virtual Reality Group now focuses on

1. Computer Vision Interface for Identifying Human Pose
   
2. Interactive Control of the Animation Character.
   

The goals of our researches are to create an interface that lets users interact with virtual environment using natural gestures and can control the animation character interactively.
Because current computer user interfaces such as keyboard and mouse do not have high dimensions or cannot intuitively interact with 3D virtual environments, we need new types of interfaces to suffice the requirements. Whereas vision based interface that identifies human pose in real-time can provide such functions. In the future we plan to integrate other types of sensors and force-feedback devices that make interface more functional and provide more realistic experiences.

Computer Vision Interface for Identifying Human Pose

It is very difficult to identify the human pose in real-time because of the high dimensionality of the human pose. If we use the tracking approach,which assumes the difference between two frames is small in a short time, we can track the human pose based on a given initialization. However, in tracking approach, we need an initialization process and the system may lose tracking because of interference.

There is another approach called example-based approach.We create a database containing many example images and each example has a corresponding pose. At run time, we compare the input image with the images in the database to find the most similar example and consider the current pose as that of it. The drawbacks of this approach are heavy computing load and insufficient precision.

In our research,we use the example-based approach to find a similar pose quickly, and use this pose as the initial conditional for a tracking-based approach to find more precise pose estimation.In this way, we can improve the precision of the example-based approach and prevent the lose tracking of the tracking approach.

To achieve fast matching, we use machine learning approaches to select features that can effectively discriminate different poses,and reduce the dimension of the image by encoding it instead of the intensity based representation.On the other hand, to improve the precision and avoid the ambiguity, we use a multi-camera vision system.
 
Interactive Control of the Animation Character

Interactive control of the animation character has a large potential especially in the video game area. However, human motion has extremely high dimensionality and traditional user interface cannot provide rich and intuitive control.Since we try to use the developing the vision interface to achieve the goals.

In our research, we preprocess the motion capture data and edit them into a connected network of many sequences,which is called Motion Graph, and we use these sequences as training data to train a motion model through a machine learning approach.On the hand, our vision interface not only identifies the pose in a single frame but inputs a sequence of poses into the trained motion model to find the similar motion in the motion capture database. The animation system will play the identified motion sequences smoothly.

In the future, we plan to integrate the system with physics simulation and more editing techniques to let the character can perform more versatile and realistic motions.

黃恩暐 ewhuang (2002- )ewhuang@sgi.csie.ntu.edu.tw
鍾哲民 Adms (2008- )adam145love@yahoo.com.tw　

許平昇 layka (2008- )lazybone0@gmail.com
鄭一中 brady(2009- )brady31027@gmail.com　

林志鴻 pcpark (2007-2009)pcpark@sgi.csie.ntu.edu.tw
黃俊棋 elestel (2004-2006)elestel@ntu.edu.tw
王劭廷 teves (2005-2007)teves@sgi.csie.ntu.edu.tw
楊嘉鳴 hearthead (2006-2008)hearthead@sgi.csie.ntu.edu.tw

International Conference
[1] Huang, E. and Fu, L. Segmented Gesture Recognition for Controlling Character Animation. ACM Symposium on Virtual Reality Software and Technology, 2008.
[2] Huang, E. and Fu, L. Gesture Stroke Recognition Using Computer Vision and Linear Accelerometer. IEEE International Conference on Automatic Face and Gesture Recognition, 2008.
[3] Huang, E. and Fu, L. Real-Time Arm Tracking System Using Example-based Matching and Local Optimization. IEEE International Conference on Systems, Man and Cybernetics, 2006.

國內碩士論文
[1] Chun-chi Huang, Li-Chen Fu, “Vision-based Interactive 3D Character Animation System”, 2006
[2] Shao-ting Wang, Li-Chen Fu, “Human Pose Recognition by Combining Static Recognition with Motion Information”, 2007
[3] Yang-chia Ming, Li-Chen Fu, “An Interactive Character Animation System with Dynamic Transition for High Degree of Freedom Input Devices, 2008