Face swapping (explained in 8 steps) – Opencv with Python

In this article I’m going to explain how to do face swapping using Opencv with Python in 8 simple steps.

This is a quick explanation of each step, but I’ve also done for each of them an entire full tutorial where I show how to do the coding.

You can download the full source code at the end of this article.

1) Take two images

The “source image” is the one we take the face from and “destination image” is where we put the face extracted from the source image.

img = cv2.imread("bradley_cooper.jpg")
img_gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

img2 = cv2.imread("jim_carrey.jpg")
img2_gray = cv2.cvtColor(img2, cv2.COLOR_BGR2GRAY)

2) Find landmark points of both images

We use the dlib library to detect the facial landmark points.
In the code below I show how to find the landmark points.

In this specific code I’m showing as example I’m detecting the landmarks of the source image, you need to apply that also to the destination image.

# We load Face detector and Face landmarks predictor
detector = dlib.get_frontal_face_detector()
predictor = dlib.shape_predictor("shape_predictor_68_face_landmarks.dat")

# Face 1
faces = detector(img_gray)
for face in faces:
    landmarks = predictor(img_gray, face)
    landmarks_points = []
    for n in range(0, 68):
        x = landmarks.part(n).x
        y = landmarks.part(n).y
        landmarks_points.append((x, y))

3) Triangulation source image (Delaunay triangulation)

We’re going then to segment the face into triangles. This step is the core of our face swapping, as later we will simply exchange each triangle with the correspondent triangle of the destination image.

Why do we devide the face into triangles?
We can’t just cut out the face from the source image and put it into the destination image as they have different size and perspective.
Also we can’t change it’s size and perspective right away because the face would lose the original proportions.
Instead if we split the face into triangles, we can simply swap each triangle and in this way it will keep the proportions and also it will match the expressions of the new face, like for example if you smile, close eyes or open the mouth.

In this code we’ll see how to do the delaunay triangulation with opencv.

# Delaunay triangulation
rect = cv2.boundingRect(convexhull)
subdiv = cv2.Subdiv2D(rect)
subdiv.insert(landmarks_points)
triangles = subdiv.getTriangleList()
triangles = np.array(triangles, dtype=np.int32)

4) Triangulation destination image

The triangulation of the destination image needs to have the same patterns of the triangulation of the source image.
That means that the connection of the points has to be the same.

So after we do the triangulation of the source image, from that triangulation we take the indexes of the landmark points so that we can replicate the same triangulation on the destination image.

# we get the Landmark points indexes of each triangle
indexes_triangles = []
for t in triangles:
    pt1 = (t[0], t[1])
    pt2 = (t[2], t[3])
    pt3 = (t[4], t[5])

    index_pt1 = np.where((points == pt1).all(axis=1))
    index_pt1 = extract_index_nparray(index_pt1)
    index_pt2 = np.where((points == pt2).all(axis=1))
    index_pt2 = extract_index_nparray(index_pt2)
    index_pt3 = np.where((points == pt3).all(axis=1))
    index_pt3 = extract_index_nparray(index_pt3)

    if index_pt1 is not None and index_pt2 is not None and index_pt3 is not None:
        triangle = [index_pt1, index_pt2, index_pt3]
        indexes_triangles.append(triangle)

Once we have the triangles indexes we loop through them and we triangulate the destination face.

# Triangulation destinazion face
for triangle_index in indexes_triangles:
    # Triangulation of the Second face
    tr1_pt1 = landmarks_points2[triangle_index[0]]
    tr1_pt2 = landmarks_points2[triangle_index[1]]
    tr1_pt3 = landmarks_points2[triangle_index[2]]
    triangle2 = np.array([tr1_pt1, tr1_pt2, tr1_pt3], np.int32)

5) Extract and warp triangles

Once we have the triangulation of both faces we take the triangles of the source face and we extract them.
We also need to take the coordinates of the triangles of the destination face, so that we can warp the triangles of the source face to have same size and perspective of the matchin triangle on the destination face.

The code below shows how to warp the triangles of the source image.

# Warp triangles
points = np.float32(points)
points2 = np.float32(points2)
M = cv2.getAffineTransform(points, points2)
warped_triangle = cv2.warpAffine(cropped_triangle, M, (w, h))
warped_triangle = cv2.bitwise_and(warped_triangle, warped_triangle, mask=cropped_tr2_mask)

6) Link the warped triangles together

Once we have cutted and warped all the triangles we need to link them together.
We simply rebuild the face using the triangulation pattern, with the only difference that this time we put the warped triangle.
At the end of this operation the face is ready to be replaced.

# Reconstructing destination face
img2_new_face = np.zeros((1155, 849, 3), np.uint8)
img2_new_face_rect_area = img2_new_face[y: y + h, x: x + w]
img2_new_face_rect_area_gray = cv2.cvtColor(img2_new_face_rect_area, cv2.COLOR_BGR2GRAY)

# Let's create a mask to remove the lines between the triangles
_, mask_triangles_designed = cv2.threshold(img2_new_face_rect_area_gray, 1, 255, cv2.THRESH_BINARY_INV)
warped_triangle = cv2.bitwise_and(warped_triangle, warped_triangle, mask=mask_triangles_designed)

img2_new_face_rect_area = cv2.add(img2_new_face_rect_area, warped_triangle)
    img2_new_face[y: y + h, x: x + w] = img2_new_face_rect_area

7) Replace the face on the destination image

The face is now ready to be replaced. We cut out the face of the destination image to make space for the new face.

So we take the new face, and the destination image without face and we link them together.

# Face swapped (putting 1st face into 2nd face)
img2_face_mask = np.zeros_like(img2_gray)
img2_head_mask = cv2.fillConvexPoly(img2_face_mask, convexhull2, 255)
img2_face_mask = cv2.bitwise_not(img2_head_mask)

img2_head_noface = cv2.bitwise_and(img2, img2, mask=img2_face_mask)
result = cv2.add(img2_head_noface, img2_new_face)

8) Seamless Cloning

Finally the faces are correctly swapped and it’s time to adjust the colors so that the source image fits the destination image.

On Opencv we have a built in function called “seamlessClone” that does this operation automatically.
We need to take the new face (created on the 6th step), take the original destination image and it’s mask to cut out the face, we need to get the center of the face and we are ready to go.

(x, y, w, h) = cv2.boundingRect(convexhull2)
center_face2 = (int((x + x + w) / 2), int((y + y + h) / 2))

seamlessclone = cv2.seamlessClone(result, img2, img2_head_mask, center_face2, cv2.MIXED_CLONE)