In this new video series “Gaze controlled keyboard” we’re going to create and app to control the keyboard trough our eyes using Python with Opencv, completely from scratch.

The goal of such app is to write without using the hands. Such applications are really important for people affected by quadriplegia who completely lost the control of their limbs.

The architecture of the app

The app will be built in 2 main parts.

  1. Eye detection: detection of the eyes, their movement and most important their blinking.
  2. Virtual keyboard: a keyboard on the screen where we’re going to select the letters by just using our eyes.

In this first tutorial we will fucus only on eye detection. We are going to detect the face and then the exact location of the eyes.

Let’s start importing the libraries Opencv, numpy and also the dlib library that we will use to detect the facial landmarks points. We then create a cap object to load the videoframes form the webcam.

import cv2
import numpy as np
import dlib

cap = cv2.VideoCapture(0)

We then load the face detector and landmarks predictor to detect the landmarks.

detector = dlib.get_frontal_face_detector()
predictor = dlib.shape_predictor("shape_predictor_68_face_landmarks.dat")

We create a function that we will need later on to detect the medium point.
On this function we simply put the coordinates of two points and will return the medium point (the points in the middle between the two points).

And after this we can run the while loop, take the frames from the webcam and detect the face.

def midpoint(p1 ,p2):
    return int((p1.x + p2.x)/2), int((p1.y + p2.y)/2)

while True:
    _, frame = cap.read()
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

    faces = detector(gray)
    for face in faces:
        #x, y = face.left(), face.top()
        #x1, y1 = face.right(), face.bottom()
        #cv2.rectangle(frame, (x, y), (x1, y1), (0, 255, 0), 2)

Eye detection

We are taking the frames in real time from the webcam, we already detected the face and now it’s the time to detect the eyes.

Using the face landmarks detection approach we can find 68 specific landmarks of the face. To each point there is a specific index assigned.
We need two detect separately the two eyes:

  • Left eye points: (36, 37, 38, 39, 40, 41)
  • Right eye points: (42, 43, 44, 45, 46, 47)
        landmarks = predictor(gray, face)
        left_point = (landmarks.part(36).x, landmarks.part(36).y)
        right_point = (landmarks.part(39).x, landmarks.part(39).y)

Once we know the exact position of the eyes we can start working with them.

Let’s create two lines, one crossing the eye horizontally and one crossing the eye vertically. We will need this for the next tutorial when we will learn to detect the blinking of the eyes.

        center_top = midpoint(landmarks.part(37), landmarks.part(38))
        center_bottom = midpoint(landmarks.part(41), landmarks.part(40))

        hor_line = cv2.line(frame, left_point, right_point, (0, 255, 0), 2)
        ver_line = cv2.line(frame, center_top, center_bottom, (0, 255, 0), 2)

And finally we show everything on the screen.

    cv2.imshow("Frame", frame)

    key = cv2.waitKey(1)
    if key == 27:
        break

cap.release()
cv2.destroyAllWindows()