We’re going to learn in this tutorial how to track the movement of the eye using Opencv and Python.

Studying the eye

Before getting into details about image processing, let’s study a bit the eye and let’s think what are the possible solutions to do this.
In the picture below we see an eye. The eye is composed of three main parts:

  • Pupil – the black circle in the middle
  • Iris – the bigger circle that can have different color for different people
  • Sclera – it’s always white
Eye

Let’s now write the code of the first part, where we import the video where the eye is moving. And later on we will think about the solution to track the movement.

We import the libraries Opencv and numpy, we load the video “eye_recording.flv” and then we put it in a loop so tha we can loop through the frames of the video and process image by image.

import cv2
import numpy as np

cap = cv2.VideoCapture("eye_recording.flv")

while True:
    ret, frame = cap.read()
    if ret is False:
        break

Let’s now select an Roi (region of interest). In this way we are restricting the detection only to the pupil, iris and sclera and cutting out all the unnecessary things like eyelashes and the area surrounding the eye.

roi = frame[269: 795, 537: 1416]
rows, cols, _ = roi.shape
gray_roi = cv2.cvtColor(roi, cv2.COLOR_BGR2GRAY)
gray_roi = cv2.GaussianBlur(gray_roi, (7, 7), 0)

Now we can dive deeper into finding the right approach for the detection of the motion.

Let’s take a look at all possible directions (in the picture below) that the eye can have and let’s find the common and uncommon elements between them all.

What can we understand from this image?
Starting from the left we see that the sclera cover the opposite side of where the pupil and iris are pointing. When the eye is looking straight the sclera is well balanced on left and right side.

Detecting the motion

For the detection we could use different approaches, focusing on the sclera, the iris or the pupil.
We’re going for the easiest approach possible, and probably the best solution anyway.

We will simply focus on the pupil. By converting the image into grayscale format we will see that the pupil is always darker then the rest of the eye. No matter where the eye is looking at and no matter what color is the sclera of the person.

So let’s do this. First conversion to grayscale and then we find the threshold to extract only the pupil.

    gray_roi = cv2.cvtColor(roi, cv2.COLOR_BGR2GRAY)
    gray_roi = cv2.GaussianBlur(gray_roi, (7, 7), 0)

    _, threshold = cv2.threshold(gray_roi, 3, 255, cv2.THRESH_BINARY_INV)

From the threshold we find the contours. And we simply remove all the noise selecting the element with the biggest area (which is supposed to be the pupil) and skip al the rest.

   _, contours, _ = cv2.findContours(threshold, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
    contours = sorted(contours, key=lambda x: cv2.contourArea(x), reverse=True)

    for cnt in contours:
        (x, y, w, h) = cv2.boundingRect(cnt)

        #cv2.drawContours(roi, [cnt], -1, (0, 0, 255), 3)
        cv2.rectangle(roi, (x, y), (x + w, y + h), (255, 0, 0), 2)
        cv2.line(roi, (x + int(w/2), 0), (x + int(w/2), rows), (0, 255, 0), 2)
        cv2.line(roi, (0, y + int(h/2)), (cols, y + int(h/2)), (0, 255, 0), 2)
        break

Finally we show everything on the screen.

    cv2.imshow("Threshold", threshold)
    cv2.imshow("gray roi", gray_roi)
    cv2.imshow("Roi", roi)
    key = cv2.waitKey(30)
    if key == 27:
        break

cv2.destroyAllWindows()