Distinguish the Textures of Grasped Objects by Robotic Hand Using Artificial Neural-Network

 Feature vector was acquired from force and actuator sensors in time series response.  Arduino microcontroller and the Matlab program are integrated to acquire sensor data.  Neural-Network used as an intelligent classifier to distinguish the object softness. The object identification properties with tactile sensing are valuable in interaction with the environment for both humans and robots, and it is the core of sensing used for exploration and determining properties of objects that are inaccessible from visual perception. Object identification often involves with rigid mechanical grippers, tactile information and intelligent algorithms. This paper proposes a methodology for feature extraction techniques and discriminates objects for different softness using adaptive robotic grippers, which are equipped with force and angle sensors in each four fingers of an underactuated robot hand. Arduino microcontroller and the Matlab program are integrated to acquire sensor data and to control the gripping action. The neuralnetwork method used as an intelligent classifier to distinguish between different object softness by using feature vector acquired from the force sensor measurements and actuator positions in time series response during the grasping process using only a single closure grasping. The proposed method efficiency was validated using experimental paradigms that involving three sets of model objects and everyday life objects with various shapes, stiffness, and sizes. A R T I C L E I N F O Handling editor: Muhsin J. Jweeg


Introduction
Determining the object properties via the haptic feedback, or vision, in the area of robotic grasping are significant issues for interaction with the environment, and have received extensive attention among researchers lately. Knowing object properties is an important to provide necessary information that helps to optimize the manipulation action to minimize the amount of forces for guaranteeing stability of the grasp, in this context giving good knowledge of the object class empowers the execution of object-specific plans or strategies [1]. Although, visual feedback is an important source to provide the necessary information for grasp operation and control [2,3], however, it is not always enough or trivial to obtain and difficult task for sophisticated vision algorithms. Moreover, limitations in the workspaces of the camera and poor lighting conditions may render the methodologies based on vision are infeasible. Furthermore, other physical properties, like stiffness, texture and weight are difficult to visually ascertain. Consequently, once vision is unobtainable, or when there is a need for additional information, the tactile sensors can be utilized as a viable alternative to improve grasping performance, and perform object identification during grasping process [4]. Wherever, different methodologies with tactile sensing can recognize various object properties, such as weight, shape, size, texture and stiffness [5][6][7][8], or to distinguish an object's class [9,10], roboticists have employed tactile sensing to robotic hand that inspired by nature's versatile of the human hand. Where, the human hand has approximately 17,000 units of mechanoreceptive ability, which innervate it's skin for understanding the environment, this leads to a highly sophisticated system [11]. Humans have the ability to identifying versatile objects, and their properties, in daily life by relying on tactile information; such as: roughness, hardness, thermal conductivity, identifying an object in the dark, and sensitivity slip during object manipulation. In the field of industrial robotics, tactile sensing is used for classification of objects during grasping proses for blind grasping and bin picking within the production lines. Furthermore, in quality inspection, the tactile sensing can be detected deficiencies, in the weight or shape of a manufactured object during picked up.
Tactile sensors systems in service robotics are crucial for interaction with the environment, and they are suitable to improve correct grasping performance. Due to its importance different researchers had been used various tactile sensory principles; such as: piezo-electric vibration sensors [12], piezo-resistive sensors [13], pressure-sensitive of polymers conductive [14], and capacitive sensors which can be measured temperature [15] or sheer force [16]. An intensive review for numerous approaches used is well presented by Tegin and Wikander [17]. Other works used advanced sensor technology in the field of object classification using the advanced machine learning techniques as classifier technology, e.g. Irin et al. [18] proposed the implementation of the robotic system and piezoresistive pressure sensors to acquire the pressure data with the help of a PIC32 microcontroller, and employed the decision tree method as intelligent classifiers for grading the objects to three rigidity levels by using the variance, mean and maximum force values as features. Lam et al. [19] investigated the identify the nature of the object surface based on surface features data that is collected from robotic finger equipped with a 6axis force-torque tactile sensor via sliding the finger on the object surface. The vibration properties caused by the contact sensing fingertip are used as surface features of the object which can be used as input data for the neural-network classifier. Jeremy and Gerald [20] presented a method for texture recognition using a multimodal sensor that provides temperature, force, and vibration data. These data are obtained by exploratory movements of the fingertip on the surface of the object with a predefined trajectory and measurement of reaction forces and tactile vibrations when exploring textures were used to extract measures of textural properties and Bayesian inference was employed for classification of textures. Chu et al. [21] Introduced a biomimetic tactile sensor equipped with the PR2 humanoid robot to produce the physical properties of everyday objects like temperature, pressure and deformation of contact surfaces by directly touching objects. Where, The platform uniquely was created with multi-channel tactile sensors capable of controlled manipulation and identify the objects characterizations like compact, metallic, cool, unpleasant, and absorbent by implemented the scikit-learn as a machine learning algorithm. Drimus et al [22] presented design of a tactile-array sensor based on flexible piezoresistive materials and conductive thread electrodes mounted on both fingers of a robotic gripper. The k Nearest Neighbors classifier (kNN) was applied for classifying deformable and rigid objects by using the resulting dynamic time series from the palpation procedure during squeezing and releasing of the object. Furthermore, Tapomayukh et al. [23] used the haptic sensing techniques with a large contact region of robot forearm to identifies rigidity (soft vs. hard), and mobility compliance (fixed vs. sliding), of the object by sliding a tactile array on the object surface to generate the select features at time series and using a kNN as classifier method. Danfei et al. [24] proposed BioTac multimodal tactile sensors (temperature, force and vibration) that integrated onto fingertips of the five-fingered Shadow Dexterous Robotic Hand for identifying texture, compliance, and thermal properties of the objects by programming the robot to make exploratory movements similar to those humans make, using learning techniques process to collect data and provide measures of these perceptual properties.
This work aims to present a methodology for distinguishing between various objects during the grasping process using an underactuated robotic hand equipped with force and angle sensors. The Neural-Network with advanced machine learning techniques used as an intelligent classifier to distinguish between different object softness by using feature space acquired from the force sensor measurements and actuator positions at time series response during the grasping process.

Experimental Setup
To test the performance efficiency of the proposed method, the experimental setup that including the details of the underactuated robotic hand with embedded sensors are explained and the object properties that utilized in the experiments are also characterized.

Under-actuated Hand
The under-actuated robotic hand used in this work has four fingers that consists two phalanges per finger with a distal and a proximal pin joint. The principle of this hand mechanism is highly underactuated, which controlled via a single actuator using differentials of linkage seesaw as shown in Figure 1. The design of the robot finger integrates segments via pin joints and a tendon, which allowing it adaptable different objects , easily grasping and it can be used with wide applications. The details of the design and benefits arrangement of this hand are described in [25]. The stepper motor was used to actuate the hand gripper, since the stepper motor is an appropriate actuator for accurate positioning. To evaluate the robotic hand, an underactuated finger is equipped with a force sensor resistor and angular potentiometers to measure the grasping forces and the positions of each phalange during the grasping procedures. Also, to increase the stability and robustness of grasps during manipulation and for distribution contact force over the force sensor resistor, soft fingertip is preferred to use in the gripper of robotic hand [26] [27].

Force Sensor
To evaluate the modified robotic hand for a proposed methodology of discriminating softness objects during grasping, the grasping forces have to be measured during the grasping procedures. The striped force sensor resistor (FSR type SF10-60) with reading range (10g-5kg) was mounted on four fingered robotic gripper, as shown in Figure 2. The advantage in using striped sensors is the large active area according to the shape of phalanges ,low cost, thin size (less than 0.5mm) and also good shock resistance. This sensor is a polymer thick film device that exhibits a decrease in resistance with an increase in the applied force to the active surface of the sensor and tell if there is an object/surface touching the finger, and what amount of force is being created and give feedback information about the strength of the contact between the object/surface and the finger.

Angular Potentiometers
To measure the angular displacement of each joints of the robotic finger, an angular potentiometer with total angular travel of 270 was chosen as an angle sensor. The maximum joint angles travel limit for the proximal and the distal joint are 90, which are within the mechanical limits of the selected potentiometers. The Potentiometer can be defined as a variable resistor that is attuned from zero ohms to whatever maximum resistance that is specific to it. Thus, it can be used as voltage dividers and the position information of the joint angels was obtained by reading the potentiometer as a voltage value. The potentiometer with the circuit diagram is shown in Figure 3. The potentiometer has several advantages. First, it offers a linear characteristic and high accuracy. Second, its inexpensive, small in size, and straightforward to implement. Third, its not be affected by metal components like return springs and bearings or magnetic properties of grasped objects.

Objects
In this work, three sets of model objects with different stiffness ,shapes and sizes and everyday life objects are used to test the efficiency of the proposed methodology of discriminating softness objects. The model objects were fabricated with variation parameters, where the rigid model objects were designed using hard material (wood), while the medium softness objects were designed using silicon rubber and the soft model objects were designed using sponge. The model objects involve 27 model (three different spherical size, three different cylindrical size and three different rectangles size with three different softness for each type). The present model objects are shown in Figure 4, and the dimensions of model objects are reported in Table I. The second set contains of everyday life objects that are selected from daily life set with a diverse range of stiffness ,shape and size. These objects are depicted in Figure 5.

Data Collection
The data was generated using tactile sensors reading and real robotic hand in grasping objects at time domain response recorded from. The experiments were conducted for collection data by grasping each 27 model objects several times, in the arbitrary pose within the workspace of the gripper. For the everyday objects with deferent range of softness, shape and size, the data collection are recorded for each object in unconstrained orientations to take account of variations in surface properties. Furthermore, six empty grasps with no object present were also recorded. In each case of the trial (grasp), the actuator was commanded to move with a constant velocity and open loop control (no influence of feedback sensor). The feature space of the force sensor and the joint angle positions were reading at time series response during the grasping process.
The first instance time is recorded (t1) when any finger of the robotic hand is in contact with the surface of the object. The last instance (t2) is taken when (t =14 s) at which time the motion of actuator was reached of the steady state. Generally, the feature vector has 8 variables, which contains 4 joint angles (the distal joint angles for each finger) and 4 force sensors (see Figure. 9). The proposed methodology is obtained the raw data without need a-priori information regarding of actual joint angles or robot model and this is very important for applications of underactuated robotic hands, since the calculation of kinematics hand after contact with an object is very complicated, where passive compliance takes place. The force sensor data, plus actual positions of the joints angles are acquired as analog voltage signals, then calibrated both the force and angle sensors to converted the voltage signals to physical values which measured the force values in ( Newton) and angle value in (Degree). The feature space that acquisition from these data are stored and used later in the offline mode to develop the learning model using a neural network. An example of grasping an object to obtained the feature space is illustrated in Figure  6.
After the data collection for each objects, these objects are labeled as either soft or mid-softness or hard by considered the model soft objects (Sponge) to be soft and the medium softness objects (silicon rubber) to be medium, while rigid model objects (wood) to be rigid. Also, the everyday life objects are labeled by considering the range of softness.

Data Acquisition System
The typical data acquisition system is consisting from hardware Arduino device and application software Matlab. This configuration is especially suitable is for developing automated instrumentation systems and effectively used for data acquiring and analyzing the complete measurement system. Arduino device and Matlab program are used to interface the sensors with the planner PC and evaluate and measure the reading sensors. After the robotic fingers begin to contact with the object in grasping, the feature space was acquisition from the force sensor and the actuator positions readings at time series response during the grasping process are acquired as analog voltage signals and these data was reading from hardware devise (Arduino) to be processed by software program Matlab for analysis and data logging. Where, the Matlab software is used for buffering the main acquire sensors data and converted to vector form to prepare as the input vector of neural network that is used as intelligent classifier method. The graphical user interface "GUI" was designed in Matlab to display the results of testing objects in N.N. Figure 7 shows the block diagram of the data acquisition method with software program for classification objects.

Classifier Based on Neural Network
The neural network proposed consists of three layers: the first layer is called as "Input Layer" and the last layer is called as "Output Layer". The layers which are between the first and the last layer can be called as "Hidden Layer", the number of hidden units affects the smoothing of the decision boundary, a low number of hidden units could over generalize the model. In this work, the neural network toolbox in Matlab has been used as an intelligent classifier to identify the object softness by classifying objects to three different grads. This neural network is a feed forward back-propagation with two hidden layers, as shown in Figure 8. Both hidden layers consist of four neurons with a sigmoid transfer function and a purelin transfer function for output layer. Model of neural network technique have some of the advantages: such as, it runs fast ,high accuracy of classification, efficiently on large databases and it can handle thousands of input variables.
The data set for network training has been prepared from the results of sensors using the data collected from grasping of 27 model objects and 18 household objects and its can be described as input data. The input data of the neural network is given by x n where n in denotes the number of input nodes in the input layer.
The summation function results will be sent to the activation functions (AF). Where, the AF is a mathematical function in which the combines given-input values with it and lastly, the obtained results from this function is constituted the array for output values. If the predicted output is not the same as the desired output, the weights are adapted by used the back propagation (BP) algorithm to get the correct prediction. The training procedure of the back propagated-ANN occurs in three stages: Calculating the result output values from the activation functions after applying input values.

1)
Calculating the error by comparing the results of output values with expected values.

2)
The weights are adapted by minimizing the mean square error. 3)   The training process starts with an iteration of these three stages, and the iteration is stopped, if iteration number equals to an expected amount or gets an error under a value that is defined before. The Levenberg-Marquardt back-propagation is used to train the classifiers by minimizing the mean square error. Initially, a simple neuronal network was implemented with one hidden layer and the default transfer function (tansig, mathematically equivalent to tanh) and default values for the other training parameters. Then, different numbers of hidden layers and hidden nodes and various transfer functions have been tried to get the least error. Thus, only the appropriate configurations that can be achieved acceptable of minimizing error are reported.
The weights were tested across all samples to obtain the correct ones for training the neural network and used them to predicting the class label of an unknown sample. These weights are buffered to prepare the input vector in testing of neural network. Where, the final output of pre-training can be used as input for a standard neural network classifier.

Results and Discussion
To detect the objects softness, the model and everyday life objects with various softness, shapes and sizes are selected as samples. The array of tactile information of the force and angles displacement data of each phalanx from a palpation procedure during a single grasp of cylinder objects had been recorded as a time series of features and plotted for three different softness as shown in Figure 9. From close observations in Figure 9, it is clear that the force generated on grasping the hard object is much higher than the moderate or soft ones. Also, the increasing rate in the force contact of the moderate and soft object is gradually rising with an increase in time grasping. This is reasonable due to the additional constraints of increasing stiffness of the object. In addition, the softer object having a higher ratio of angle displacement than the harder one under the same size. The data mining software package and neural network are used for training and obtaining the model to be implemented in real time system. The time domain response from the robotic grasping of the objects are stored and used later in the offline mode to develop the learning model using neural network.
For selection the parameters of a neural network, the results trend that more neurons are not better and more hidden layers did not get better results, but the recognition rate was decreased with less hidden layers and neurons. Also, the other transfer functions and initial learning rates did not improve the recognition rate. Thus, only the appropriate configurations which can achieve acceptable recognition accuracy are selected optimized training parameters. The training performance is shown in Figure 10.
The proposed method efficiency for discrimination of objects softness was validated using experimental paradigms that involving the different everyday objects. Where, the performance of the network was evaluated under various objects for retesting. The used objects contain 24 samples of specific shape and 30 samples of irregular shape with three classes. These objects are labeled as either soft or moderate or hard related to the softness of objects measured by sensors. The test results with ANN detected data are recorded in the confusion matrices as shown in Figures 11-12, where, the abbreviations used in these matrices as follows: H = hard, M = moderate, and S = soft. The diagonal elements of the confusion matrices refer to the correctly classified trials while the non-diagonal elements refer the misclassifications. The results showed that the ANN classifier is very efficient in distinguish between the examined of the specific shape objects, with only one object was misclassified, and the present approach had a 95.83% accuracy. The most confusion was in distinguish between the softness of the examined irregular shape objects, and the current approach had a 86.66% accuracy. These misclassifications are caused by the lack of similarity softness between certain sides. Furthermore, the confusion matrices indicted the sensitivity of recognized of hard objects is more than the medium and soft ones. On the other hand, the precision of discriminating the soft objects is more than medium and hard ones.
In general, the advantages of the proposed methodology presented in this study are: work in a direct manner (no regrasping required), can deal with the asymmetric finger in contact locations ,trajectories and contact times and it can deal with fingers that pushing the object during the grasping. Furthermore, the proposed approach system does not require any prior information about the shape object to grasp from or the robot hand model. This is particularly beneficial for grippers that are difficult to be derived from kinematic and dynamic models.

Conclusions
This paper proposed methodology for discriminate between various softness objects using four-fingered underactuated robotic hand that is equipped with force and angels sensors. The feature space used consists of the force sensor and the joint angle positions readings in time series response of the open-loop grasping process. The hardware Arduino and Matlab software are integrated to acquire sensor signals. A neural network classifier was applied to distinguish between various softness objects. The experiments were concluded to validate the efficiency of the present classifier methodology by grasped a wide range of objects with various sizes, shapes and softness in arbitrary poses. The results showed that the proposed methodology is very efficient in recognition of the examined objects classes, with only a few objects was misclassified . Where, this approach has a 95.83% accuracy for identify the softness of various sets of objects with specific shapes and has a 86.66% accuracy with irregular shape. In future work, the proposed approach will be developed for discriminating the objects shape via specifying the location contact.

Author contribution
All authors contributed equally to this work.

Funding
This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Data availability statement
The data that support the findings of this study are available on request from the corresponding author.

Conflicts of interest
The authors declare that there is no conflict of interest.