An Automatic Deformation for Facial Smile using PSO Detection

: To improve the convenience of triangulation based image warping algorithm, an automatic approach for facial expression deformation is presented. In order to guarantee the expression deformation is dealt with high quality and conveniently, the approach uses two key technologies: the PSO detection and the affine transformation. The PSO algorithm is utilized to search the optimal position of rectangle containing face and eyes. The affine transformation is applied to warp related triangles to deform expression. The simulation results show that this method can produce natural smile expression, without marking the spots manually, especially works well for some shy smile expressions.


Introduction
Facial expression is an important tool in human communication.Except the inter occular distance and the ears position, most features in a face can change their position via muscle action of different expression.Especially, the lips (mouth) position change biggest.Our deformation approach is mainly focus on the location and deformation of lips.The technologies used in face image deformation can be roughly divided into two categories: One is the deformation based on scattered point interpolation.The other is the deformation based on fragments.These methods have their own limitations.In order to get good results, and at the same time divide the triangles conveniently, we propose a deformation approach for facial expression based on PSO (particle swarm optimization) and affine transformation.In general, to correctly warp one face into another expression, we need to mark numerous spots first.We have to point the eyes, nose, lips and other notable face areas.Marking the spots manually can ensure the highest quality, but it may take long time.Our algorithm is fully automated.After a face image is loaded, "algorithm of Face Location Based on PSO" can automatically recognize the image as face.Similar algorithm is executed to recognize the eyes and lips, then label the vertices used to form triangles.The PSO method is a member of the wide category of swarm intelligence methods.We introduce PSO method to help to position feature points and conveniently divide triangles.Finally, the affine transformation process starts to get a smile face.

PSO Searching
It is known that, most information is stored on the image edge.Also, face and features location is based on the edge information.After image preprocessing, a binary image of face image can be obtained.The next work is all performed on the edge of binary image.Here, the Sobel algorithm is used for generating edge image of binary image.Face, eyes regions can be located precisely using particle swarm searching on the edge image, which will improve the accuracy of expression recognition.Then, feature extraction method will be used to detect the face and eyes position.Human face is rather special because it contains more details than other parts of the body.This is fully reflected in the edge image by the fact that face area contains high density edge points.How to define the face region?We have the following assumptions.The face position detection is preceded under the assumption that the face region can be surrounded by a rectangle.The rectangle has three parameters: the abscissa (x) and ordinate (y) of the upper left corner, the length of the diagonal (r).It is commonly assumed in literature that the ratio of the length to width of the rectangle area is 1.5: 1, once the face region is located.But there is no limit on the size of rectangle since we don't know the scale of face in the image.In the same way, it is assumed that the ratio of the width to length of the rectangle area is 1.5: 1, once the eye region is located.
The criterion of feature extraction is defined by the number of edge pixels in the rectangle area to the rectangle perimeter.The ratio is large when the rectangle contains edge points at more frequent.The criterion is calculated below. where e is edge image of the original image and e(i, j) represents a pixel point with value 1 or 0. Let (x, y) denotes the coordinate of the upper left corner of the face rectangle, while (x', y') denotes the lower right corner of the face rectangle.Thus, according to the ration of length and width, the following relationship exists.(3) Similarly, Let (x, y) denotes the coordinate of the upper left corner of the eye rectangle, while (x", y") denotes the lower right corner of the eye rectangle.Thus, according to the ration of length and width, the following relationship exists.(5) Based on the above analysis, as long as the largest ratio of rectangle is found, the task of face and eyes detection is completed.The immediate question is how to search such rectangle, which is a very difficult combinatorial optimization problem.Next section, we will utilize PSO algorithm to search the optimal rectangle region.
In the PSO algorithm, each particle corresponds to a possible optimal solution.For the optimal problem of face and features detection, particle representation is relatively simple, i.e., 3-dimensional vector X i = (x i1 , x i2 , x i3 ), where elements x i1 and x i2 denotes the values of parameter x and y respectively.x i3 denotes the value of parameter r.For example, particle X 3 = (x 31 , x 32 , x 33 ) means that the rectangle represented by the third particle has three parameters: x 31 denotes the horizontal coordinate of upper left corner of the rectangle; x 32 denotes the longitudinal coordinate of upper left corner of the rectangle; x 33 denotes the diagonal length of the rectangle.Obviously, objective function f(x, y, r) could be defined as the criterion (Eq.( 1)) to measure the property of particle.The detailed process of detecting face and eyes contours using PSO will not be discussed here, due to page limitation.For more information, see our previous work.The following Fig. 1 summarizes the whole detection process.So far, we have got the feature data needed to deform expression.In the next section, a smile deformation method based on these face feature data is proposed.

Expression Deformation
From the above PSO preprocessing, we get the location of face counter, eyes and lips.The next task is to deform the face image according to these information.As we known, the affine transformation is a powerful tool to realize the image deformation.So, we use affine transformation to realize the expression deformation in this paper.According to the basic theorem of affine transformation, three pairs of corresponding points determine a unique affine transformation.Therefore, we divide the region which needed to be deformed into lots of triangular patches.
At the beginning, the vertices of triangles should be determined.Since we focus on smile deformation, the lips region is the main consideration.In order to make smile image, the lips corners should rise.So, we first generate two triangles with vertices (1, 2, 4) and (2,3,4).Then, to warp the two triangles, some other triangles should be defined outside them.There are two selections for the outside triangles, as shown in Fig. 2((a)(b)).One is triangles with 8 vertices.The other is triangles with 9 vertices.The difference between them is 9 vertices will effect more region when lips corners rising.Thus, we adapt the method of 9 vertices to divide triangles, because of its natural effect on nose without increasing too many vertices.The specific method of determining these vertices is described here.The four vertices (1, 2, 3, 4) of two triangles including lips are located respectively at the center of each border of the lips rectangle.The other five vertices (5,6,7,8,9) are located at the inner corners of the eyes rectangles and the face contour (see Fig. 2(c)).Based on the 11 triangles, we can do affine transformation to make face smile (see Fig. 2(d)).
The affine transformation changes the pixel position within the range surrounded by vertices of 5, 6, 7, 8, 9. So, in terms of vertex, the positions of outside five vertices keep unchanged after affine transformation.Only the positions of inside four vertices (1, 2, 3, 4) will change.But, in terms of triangles, all the triangles will warp because some vertices change their positions.If certain pixel falls in the k-th triangle, then the affine transformation can be performed based on the pre-defined smile extent.Of course, we know that the affine transformation matrix should be obtained before affine transformation between any two triangles.How to calculate the affine transformation matrix will rely on the given conditions.Then, this matrix can be used to perform affine transform between the points inside the two triangles.

Experimental Results
This section illustrates the experimental results dealt with our expression deformation algorithm.The expression image is preprocessed according to the method in previous section.Steps includes face detection, eyes detection, lips detection, and uniform (550×550).Of course, the preprocessing doesn't change the posture and expression of face image.As shown in Fig. 3, we select six typical faces of minority nationalities to perform smile deformation.The neutral expression faces are arranged in the first row.Based on these neutral faces, smile face is made using PSO and affine transformation.In order to assess the deformation, we compare the smile deformation result with the real smile image.The images processed by our method are arranged in the third row (artificial smile).As a comparison, the images of real smile images are arranged in the second row (real smile).Results show that our method can realize deformation smile basically for different faces.Especially, the first and third smile faces are simulated better.But due to individual and cultural differences, the way and degree of each smile face are different.So, there is a gap between the deformation smile and artificial smile.The gap is that the lips corners rise after the middle lips stretched, and the eyes changed slightly as well.The number of triangles plays a great role in expression deformation.Due to the reason that the lips triangles are only defined as two, the deformation seems a little rough.In order to make more realistic smile expression, the lips triangles should be increased.Of course, at the same time, vertex will also be increased.

Summary
This paper studies a kind of expression deformation problem, mainly focuses on the smile expression.In the preprocessing stage, PSO is utilized to search the optimal position of rectangle containing face and eyes.Simulation results show that the facial features are detected and labeled successfully, which help to divide triangles conveniently.The PSO algorithm can search effectively and reduce computational complexity, therefore reduce the search time.In the deformation stage, the affine transformation is applied to warp related triangles to deform expression.The advantage of our deformation approach is to get a good deform result without complex labeling, since the PSO can search the features automatically.In the future, the approach can be improved in terms of the number of triangles so that it can produce more real smile expressions.So, the proposed deformation approach has an important significance in studying the similarities and differences of multi ethnic expressions.
Fig. 1 Face Features Detection

Fig. 2
Fig. 2 Deformation process of smile expression