Teleoperation guided by vision and augmented reality

Abstract

In this report, the controlling of arm robot of RV-M1 using teleoperation system and supported by augmented reality is presented. The arm robot of RV-M1 has five degree of freedom. In order to control the movement of robot, the hand gestures are chosen because of its natural movement. In order to detect the hand gestures, the image processing method with the help of USB camera webcam is used. For every movement of robot, different gesture of the hand is assigned. The most natural gestures of the hand are selected for this project, so that most of the people will not have a problem to perform it. Once the camera detects the gesture made by hand, the image is acquired and processed by the Mathematics Laboratory (MatLab) software. In Matlab, the designed application would classify the gesture of the hand. Then, the software will send a command corresponding to the gesture to the robot, hence, the robot moves accordingly to the command sent. For the implementation of augmented reality system, another USB camera webcam is needed. By using this camera, user can observe and watch the frame of robot in real time. The implementation of augmented reality shows the indicator of robot movement on the image frame. Besides, the coordinate of the robot is displayed in the screen. The coordinate changes the robot moves. Another interesting feature that has been developed in this project is the virtual object aiming for testing the teleoperation of robot using hand gesture. The teleoperation of robot by gesture is quite difficult especially for a person that never used it before. Using this function, the virtual object can be picked and placed like a real object. Another interesting function created in this project Estimated Robot Moves where a desired position can be selected from the image. Once selected, the system transforms the selected position in the image into a robot coordinate. Then, the robot moves to the estimated coordinate. This reduce the time needed to teleoperate the robot to a desired position.