Detection and Tele-replication of Human Hand Motions by a Robotic Hand
Lucian Milea1, Monica Dascalu2, Eduard Franti2, 3, Suzana Cismas4, Doina Moraru1, Florin Lazo2, Elteto Zoltan2
1Solaris Consult S.R.L., Bucharest, Romania
2Centre for New Electronic Architecture, Research Institute for Artificial Intelligence, Bucharest, Romania
3Micromachined Structures, Microwave Circuits and Devices Laboratory, National Institute for Research and Development in Microtechnologies (IMT), Bucharest, Romania
4Department of Biotechnology, University of Agricultural Sciences and Veterinary Medicine, Bucharest, Romania
To cite this article:
Lucian Milea, Monica Dascalu, Eduard Franti, Suzana Cismas, Doina Moraru, Florin Lazo, Elteto Zoltan. Detection and Tele-replication of Human Hand Motions by a Robotic Hand.American Journal of Aerospace Engineering.Vol.2, No. 4, 2015, pp. 30-35. doi: 10.11648/j.ajae.20150204.11
Abstract: This paper presents a tele-operated robotic hand controlled by replication of human hand motions and is focused on the description of technical solutions for detection and tele-replication of movements, in order to control a robotic hand. The purpose of such research is justified by the need of high precision human controlled operations in special environments. The system is based on a flex sensors set with processing units and has as effectors a robotic arm and an anthropomorphic hand. The current article displays the modality of achieving an anthropomorphic robotic arm capable of efficiently handling objects of different sizes. In order to implement and test the technical and computing solutions, the authors have used a commercial product as experimental platform and improved it both in its mechanical structure and in its command and control system. For implementing the motion algorithms of the robotic arm, a method was developed for decoding arm movements performed by a human operator. To this end, bending sensors placed at the human operator’s joints (shoulder, elbow, wrist and fingers) were used. Signals collected from the sensors during the realization of these different movements by the human operator were decoded, processed and implemented in the drive system corresponding to the anthropomorphic robotic arm. In this regard, all sets of complex movements by the human arm operator were duplicated and implemented in the anthropomorphic robotic arm. The results obtained in handling various objects by means of using the anthropomorphic robotic arm have certified the effectiveness of this method.
Keywords: Robotic Hand, Robotic Arm, Flex Sensors, Motion Replication
Tele-operation (distance command and control) of automatic and robotic systems is nowadays used in various domains, from automotive applications to medical high-precision systems, from aero-spatial vehicles to games. Many industries are now based on tele-operated factory automation and/or robotic systems. In particular, tele-operated robotic arms were developed and are now available as commercial products.
Mechanical arms, remotely operated by human workers, are nowadays used in more and more industries. They provide high precision in handling operations of different size objects. A special category is that of the robotic arms used for performing various operations in hazardous or toxic environments. Manipulating items under hazardous or toxic circumstances requires the human operator to wear special protective equipment so as to ensure personal safety; such equipment often makes movement and desired precision difficult or even drastically reduced.
Anthropomorphic mechanical arms have been made with a large number of degrees of freedom, similar to those of a healthy human arm. They allow the achievement of a high number of operations in manipulating objects. Teleoperation of a robotic arm can be done using a joystick or another type of teleoperation interface, or by replication of human motion (of the operator), or tele-replication . Research and development reported in recent scientific literature show that currently, certain difficulties are faced in conceiving and developing algorithms that underlie the sets of complex movements for such anthropomorphic robotic arms.
In the case of tele-replication of human arm motion, the most common solution implies the use of an exoskeleton. The movements of the corresponding segments of the exoskeleton attached to the human arm are more easily put in correspondence with the artificial arm movement. For tele-replication of hand and finger motion, sensor-based systems like those implemented in virtual-reality gloves are preferred .
The system presented in this paper uses tele-replication of human motion as the method of remote control, using sensors for detecting arm and hand motions. The tele-operation algorithm is based on decoding the motion patterns for all the human operator’s arm and hand joints, for complex sets of operations, and subsequently implementing them into an anthropomorphic robotic arm. Through this method, the anthropomorphic robotic arm is able to perform complex sets of movements which are useful in the manipulation of objects characterized by different sizes.
Although sophisticated devices like multisensory five-finger dexterous hand DLR/HIT hands are now available on the market, low-cost and high precision solutions are still needed (a performant robotic arm nowadays costs from $100,000 to $300,000).
2. The Sensor-Based Replication of Human Arm and Hand Motions in Robotic Systems
Replication of human movement by robotic arms and hands, so that the latter can be remotely controlled by human hand and arm movements, is an essential part of tele-operation. In order to achieve it effectively, the first necessary thing is to accurately detect (in point of quality and quantity) all the movements made by the human arm and hand (or only those joints that make up the segment of interest), followed by appropriate processing of the obtained information and the command of the robotic systems , .
Using of the most suitable sensors will be necessary in order to achieve accurate movements’ detection, and the information obtained from them will be specifically processed to control the actuators of the robotic systems.
In doing so, we must consider that effector systems consisting of robotic arms and hands can have articulated structures and various drivers (not always similar to the human ones in point of cinematics and functionality), and they must be specifically ordered. Therefore, different systems often need necessary adjustment of the processing models used, or even completely new and specific approaches.
Detection of human hand movements can have several levels of complexity, depending on the monitored number of degrees of freedom and the required accuracy in various applications. In the general case, we can investigate the monitoring of all the degrees of freedom of the human hand and arm, from shoulder to fingers. Starting from the resulting information and in conjunction with the specific structure of the robotic arm to be controlled, suitable methods of processing the data and the control drives can be selected and / or designed.
In monitoring the position of each joint by its various degrees of freedom several sensors types may be used:
- strain bands (strain gauges) (of various types) for measuring the flexion of a joint
- flexion (or bending) sensors – they are the most commonly used in detecting human body movement. They are fixed on a jouint and fold with it. Typically they are resistive and the value of their electrical resistance is directly proportional to the bending angle and inversely proportional to the curvature radius . They can be uni or bi-directional and some variants can even detect the bending direction. In terms of construction, flexion sensors may appear in many building variants:
conductive ink on a flexible substrate - these are usually unipolar sensors, and, in order to perform detection in both directions, two such sensors should be used, placed in opposite directions.
piezo resistive strips - they have a huge range of angles, they are bidirectional, but they have high costs.
resistive polymer stacked between conductor layers - their resistance decreases when compressed, either directly or by bending; they have very low costs, but also a slow response, reduced accuracy and repeatability.
- optical sensors, in which the amount of light transmitted or reflected by a bent optical fiber is a measure of the bending degree thereof. They are made of plastic optical fiber, which at one end is fed from a light source, and, at the other end, has a photodetector; bending results in a loss in light transmission, which translates into a reduction in the detected voltage or an increase in resistance (for a photoresistor). Due to their operation, these sensors are unipolar.
- pairs of Hall sensors and magnets placed on the exoskeleton attached to the human hand.
- sensor networks for flexion and angle. Such an application enables the monitoring of all joints positions in the human body, so as to duplicate movements in a virtual space.
- displacement sensors placed in specific frames for measuring flexion / extension of the joints.
- 3D gyroscopes, accelerometers, magnetometers. Each can provide information about the absolute or relative position of the segments of the human body; their combined use can provide a complete and stable image on the current position of the human body, based on prior knowledge of the interconnection of the monitored segments. There are also inertial sensors for human motion tracking .
Important design decisions are: the type of sensor used, their number, their position on arm/joints. We choose the type of sensors to use – a bending sensor, and analyzing various properties of bending sensors , we decided to use FS-L-0XXX-103-ST sensors (figure 1), considering the above mentioned data and the specifics of the application. Their main advantage is that they are easy to use and reasonably priced. They are uni-directional sensors and measure the bending degree with a reasonably linearity, and, thus, just as the sensor is bent, its electrical resistance increases in linear proportion.
3. The Basic Structure of the Robotic System
The robotic system consists of a robotic arm with 5 degrees of freedom, at the end of which an anthropomorphic robot hand is fitted (five fingers, each with the same number of phalanges that mirror the human hand). This commercial product is controlled through a sophisticated software tele-operation interface that was further replaced with a customized control system.
The joints are powered by electric actuators which allow rotation between 30 and 180 degrees (depending on use and location). Of the 18 degrees of freedom of the system, only 16 are current. The actuators are pulse controlled by the PWM output of an Arduino Mega 2560 module, equipped with the ATMEL ATmega 2560 processor.
The metal fingers that originally equipped the arm had batracian fingers anatomical shape and were built from aluminum articulated modules that were operated by 5 micro actuators through a system of levers. The major drawback of the system was that it amplified the initial joints motion in proportion to the levers ratio (1: 5), making it imprecise to grip objects using the fingers. Meanwhile fingertips mobility was limited to the upper limit, making it impossible to squeeze a small object in the hand.
Because of these shortcomings in the initial system, the original fingers were replaced by modular translucent plexiglass that mimics the human fingers anatomical shape; the grabbing base ("the palm") was maintained together with the 5 micro actuators. The lever drive was replaced by a system of wires that connect to the last phalanx of each finger and are "pulled" by the 5 micro actuators, thereby arching the fingers up to the desired position.
Thus the accuracy of gripping various objects has improved. Regarding the drive system, the arm was equipped with multiple actuators to optimally drive each joint:
The shoulder - has two actuators: one for rotation around the 0Z axis (0-160 degrees) and another for lifting / lowering the arm vertically (0 - 90 degrees);
The elbow: 1 actuator 0-90 degrees vertical;
The wrist: 1 actuator, 0-160 degrees horizontally;
4. The Method of Processing, Command and Control
Based on the law of variation of the sensor output value depending on the input, and taking into account the type and range of controls specific to the actuators chosen, the solution of appropriate scaling the range of available values was chosen.
The sensors are connected into a resistive divider circuit, at the analog input of a a control module; thus, the bending variation will produce a variation in the internal resistance of the sensor, and implicitly the voltage on the analog input of the Arduino board; this voltage is converted into a numerical value in the range 0-1024.
The numerical value is scaled in accordance with the action range of the actuator on the arm corresponding to the respective joint (the map function is used = scales one range of linear values to another range or linear arithmetic values: map (value, fromLow, fromHigh, toLow, toHigh)).
5. Block Diagram of the Robotic System
The block diagram of the robotic system is presented in Figure 2. The robotic system used contains two main blocks: the block formed by the bending sensors, on the human operator’s arm, together with the block formed by the mechanical arm, equipped with pressure sensors and actuators. Each of the two blocks has one control system Arduino 2560 type with AtMega 2560 microcontroller. The figure below illustrates a block diagram of the robotic system used and the flow of signals during its operation.
6. Robotic System Implementation
In the implementation of the robotic system (figure 3), two boards Arduino mega 2560, equipped each with a microcontroller AtMega 2560, were used, as well as shoulder, elbow, wrist and fingers bending sensors placed on the human operator. For the shoulder, elbow and fingers, bending sensors 11cm long were used, and for the wrist a 5cm long sensor was used. The figure below schematically represents the robotic system implementation. In functional terms, signals from the human operator’s bending sensors on the shoulder, elbow, wrist and fingers are taken on the analog inputs of microcontroller ATmega 2560 and converted into numerical values, which will be processed and then transmitted to the control unit the mechanical arm by bidirectional serial communication. Serial communication between the command blocks is bidirectional, with parametric 115200 baud rate, 8N1.
The data frame is 24 bytes (octets) in which an octet plays a role in error correction (CRC).
The data frame transmitted by the human arm and sensors glove control unit to the mechanical arm control block is the following:
Byte 0 = 0x20; - start frame identifier
Byte 1 = 0x0A; - start frame identifier
Byte 2 = 0x18; - dataframe length (0x18 = 24 bytes)
Byte 3 = finger 1 LSB byte
Byte 4 = finger 1 MSB byte
Byte 5 = finger 2 LSB byte
Byte 6 = finger 2 MSB byte
Byte 7 = finger 3 LSB byte
Byte 8 = finger 3 MSB byte
Byte 9 = finger 4 LSB byte
Byte 10 = finger 4 MSB byte
Byte 11 = finger 5 LSB byte
Byte 12 = finger 5 MSB byte // wrist:
Byte 13 = hand wrist flexion angle LSB byte
Byte 14 = hand wrist flexion angle MSB byte
Byte 15 = hand wrist rotation LSB byte
Byte 16 = hand wrist rotation MSB byte // elbow:
Byte 17 = wrist elbow angle LSB byte
Byte 18 = wrist elbow angle MSB byte // shoulder:
Byte 19 = shoulder lifting angle LSB byte
Byte 20 = shoulder lifting angle MSB byte
Byte 21 = shoulder rotation LSB byte
Byte 22 = shoulder rotation MSB byte
Byte 23 = CRC
7. Experiments and Results
By using this system, all necessary movement algorithms were detected and recorded using bend sensors included into support bandages and glove (see figure 4), so that the mechanical arm could perform the main sets of operations in remote-controlled interventions. These arrangements represents advances from past work, previously reported , . Figure 5 is a collection of images from different experiments that involved grasping and manipulation of objects of different forms, materials and sizes.
A library of movement algorithms was thus created and organized under different categories of operations for remote-controlled interventions.
The figures below illustrate various types of moves performed using the mechanical arm and hand, which is remote-controlled by a human operator. Manipulation using a mechanical hand for different size objects is also presented in the figures below.
The robotic arm that was built enabled efficient handling of objects of different sizes. By the method presented in this paper we managed to decode the movement algorithms of the human operator’s arm and to implement them into an anthropomorphic robotic arm.
Although a commercial product was used for offering an experimental basis, the system designed is original. The main novelty of the paper is the simple solution for replication of arm movement, in terms of sensors and processing algorithms. The obtained system is more versatile.
The results obtained in manipulating objects by means of the anthropomorphic robotic arm certify the effectiveness of the method used and clear the way for new experiments for achieving more complex operations with a higher degree of dynamism. In the next stages of the project, such operations of handling objects will be conducted, within certain sequences of predefined time.
This work was supported by a grant of the Romanian National Authority for Scientific Research, Programme for research - Space Technology and Advanced Research - STAR, project number 82/2013.