Skip to content

Tutorials

Eye motion tracking – Opencv with Python

January 4, 2019 Sergio Canu 3 minute read
Access community, courses and source codes
Logo

AI Vision Academy

Access the code of this tutorial, computer vision courses and an exclusive community on AI Vision Academy

  • Access to over 50+ source codes from Pysource.com/blog
  • Dedicated video courses about computer vision
  • Access to an exclusive community of professionals
  • Real-World AI Projects – Get hands-on experience building practical AI Computer Vision solutions with a structured path.
  • Monthly Coaching Calls – get support and any of your questions answered

Subscribe to our newsletter to learn more

We’re going to learn in this tutorial how to track the movement of the eye using Opencv and Python.

Studying the eye

Before getting into details about image processing, let’s study a bit the eye and let’s think what are the possible solutions to do this.
In the picture below we see an eye. The eye is composed of three main parts:

  • Pupil – the black circle in the middle
  • Iris – the bigger circle that can have different color for different people
  • Sclera – it’s always white
Eye

Let’s now write the code of the first part, where we import the video where the eye is moving. And later on we will think about the solution to track the movement.

We import the libraries Opencv and numpy, we load the video “eye_recording.flv” and then we put it in a loop so tha we can loop through the frames of the video and process image by image.

import cv2
import numpy as np

cap = cv2.VideoCapture("eye_recording.flv")

while True:
    ret, frame = cap.read()
    if ret is False:
        break

Let’s now select an Roi (region of interest). In this way we are restricting the detection only to the pupil, iris and sclera and cutting out all the unnecessary things like eyelashes and the area surrounding the eye.

roi = frame[269: 795, 537: 1416]
rows, cols, _ = roi.shape
gray_roi = cv2.cvtColor(roi, cv2.COLOR_BGR2GRAY)
gray_roi = cv2.GaussianBlur(gray_roi, (7, 7), 0)

Now we can dive deeper into finding the right approach for the detection of the motion.

Let’s take a look at all possible directions (in the picture below) that the eye can have and let’s find the common and uncommon elements between them all.

What can we understand from this image?
Starting from the left we see that the sclera cover the opposite side of where the pupil and iris are pointing. When the eye is looking straight the sclera is well balanced on left and right side.

Detecting the motion

For the detection we could use different approaches, focusing on the sclera, the iris or the pupil.
We’re going for the easiest approach possible, and probably the best solution anyway.

We will simply focus on the pupil. By converting the image into grayscale format we will see that the pupil is always darker then the rest of the eye. No matter where the eye is looking at and no matter what color is the sclera of the person.

So let’s do this. First conversion to grayscale and then we find the threshold to extract only the pupil.

    gray_roi = cv2.cvtColor(roi, cv2.COLOR_BGR2GRAY)
    gray_roi = cv2.GaussianBlur(gray_roi, (7, 7), 0)

    _, threshold = cv2.threshold(gray_roi, 3, 255, cv2.THRESH_BINARY_INV)

From the threshold we find the contours. And we simply remove all the noise selecting the element with the biggest area (which is supposed to be the pupil) and skip al the rest.

   _, contours, _ = cv2.findContours(threshold, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
    contours = sorted(contours, key=lambda x: cv2.contourArea(x), reverse=True)

    for cnt in contours:
        (x, y, w, h) = cv2.boundingRect(cnt)

        #cv2.drawContours(roi, [cnt], -1, (0, 0, 255), 3)
        cv2.rectangle(roi, (x, y), (x + w, y + h), (255, 0, 0), 2)
        cv2.line(roi, (x + int(w/2), 0), (x + int(w/2), rows), (0, 255, 0), 2)
        cv2.line(roi, (0, y + int(h/2)), (cols, y + int(h/2)), (0, 255, 0), 2)
        break

Finally we show everything on the screen.

    cv2.imshow("Threshold", threshold)
    cv2.imshow("gray roi", gray_roi)
    cv2.imshow("Roi", roi)
    key = cv2.waitKey(30)
    if key == 27:
        break

cv2.destroyAllWindows()

33 comments

  1. I am working on a similar project.
    But while running your code i get an error while converting it to gray.
    cv2.error: OpenCV(4.0.0) C:\projects\opencv-python\opencv\modules\imgproc\src\color.cpp:181: error: (-215:Assertion failed) !_src.empty() in function ‘cv::cvtColor’
    what should i do

    1. It looks like you’re trying to convert an image that doesn’t exist. So probably you’re not loading the image correctly.
      I suggest you to write the code line by line and test it, following the video tutorial so that if any error appears it will be easy for you to uderstand where the problem is.

      1. sir, i’m following same code as shown in video but it is showing same problem.

        while True:
        ret, frame = cap.read()

        roi=frame[269:795,537:1416]
        gray_roi = cv2.cvtColor(roi,cv2.COLOR_BGR2GRAY)
        gray_roi = cv2.GaussianBlur(gray_roi,(7,7),0)
        _, threshold = cv2.threshold(gray_roi,5,255,cv2.THRESH_BINARY)
        _, contours = cv2.findContours(threshold,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE)
        # print(contours)
        for cnt in contours:
        cv2.drawContours(roi, [cnt], -1, (0, 0, 255), 1)
        # cnt=contours
        #cv2.drawContours(roi, [cnt], -1, (1, 0, 255), 1)
        ctr = np.array(cnt).reshape((-1,1,2)).astype(np.int32)
        cv2.drawContours(roi, [ctr], -1, 255, -1)
        cv2.imshow(“threshold”,threshold)
        cv2.imshow(“grey roi”,gray_roi)
        cv2.imshow(“roi”,roi)

        error: OpenCV(4.1.1) C:\projects\opencv-python\opencv\modules\imgproc\src\drawing.cpp:2509: error: (-215:Assertion failed) npoints > 0 in function ‘cv::drawContours’

  2. I Have problem with this in code
    _, contours, _ = cv2.findContours(threshold, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
    ValueError: not enough values to unpack (expected 3, got 2)
    why

    1. You can solve the problem in this way:
      _ , cnts = cv2.findContours(mask, cv2.RETR_TREE, cv2.CHAIN_APPROX_NONE)

      This happen because you’re most likely using Opencv 4, while the code I wrote is for Opencv 3.

      1. Traceback (most recent call last):
        File “C:/Python37/project/test3 tracking.py”, line 18, in
        contours = sorted(contours, key=lambda x: cv2.contourArea(x), reverse=True)
        File “C:/Python37/project/test3 tracking.py”, line 18, in
        contours = sorted(contours, key=lambda x: cv2.contourArea(x), reverse=True)
        cv2.error: OpenCV(4.0.0) C:\projects\opencv-python\opencv\modules\imgproc\src\shapedescr.cpp:272: error: (-215:Assertion failed) npoints >= 0 && (depth == CV_32F || depth == CV_32S) in function ‘cv::contourArea’

        even when i’ve tried this solution
        _, contours = cv2.findContours(threshold, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)

      2. hi i got the same problem, when i try to fix in this way
        it is said , cnts = cv2.findContours(mask, cv2.RETR_TREE, cv2.CHAIN_APPROX_NONE)
        NameError: name ‘mask’ is not defined

  3. I Have problem with this in code
    _, threshold, = cv2.threshold(gray_roi, 3, 255, cv2.THRESH_BINARY_INV)
    _ , cnts = cv2.findContours(threshold, cv2.RETR_TREE, cv2.CHAIN_APPROX_NONE)

    Traceback (most recent call last):
    File “C:\Python37\project\test2.py”, line 18, in
    cnts = sorted(cnts, key=lambda x: cv2.contourArea(x), reverse=True)
    TypeError: ‘NoneType’ object is not iterable
    why

    1. this is the whole code
      i am trying to run it on live video and your flv
      i seduced to run it with your flv but the result only one image in three windows
      no video
      would you mind providing me your Facebook or email
      i have further queries on open cv
      import cv2
      import numpy as np

      cap = cv2.VideoCapture(0)

      while True:
      ret, frame = cap.read()
      if ret is False:
      break

      roi = frame[269: 795, 537: 1416]
      rows, cols, _ = roi.shape
      gray_roi = cv2.cvtColor(roi, cv2.COLOR_BGR2GRAY)
      gray_roi = cv2.GaussianBlur(gray_roi, (7, 7), 0)

      _, threshold, = cv2.threshold(gray_roi, 3, 255, cv2.THRESH_BINARY_INV)
      _ , cnts = cv2.findContours(threshold, cv2.RETR_TREE, cv2.CHAIN_APPROX_NONE)
      cnts = sorted(cnts, key=lambda x: cv2.contourArea(x), reverse=True)

      for cnt in contours:
      (x, y, w, h) = cv2.boundingRect(cnt)

      #cv2.drawContours(roi, [cnt], -1, (0, 0, 255), 3)
      cv2.rectangle(roi, (x, y), (x + w, y + h), (255, 0, 0), 2)
      cv2.line(roi, (x + int(w/2), 0), (x + int(w/2), rows), (0, 255, 0), 2)
      cv2.line(roi, (0, y + int(h/2)), (cols, y + int(h/2)), (0, 255, 0), 2)
      break

      cv2.imshow(“Threshold”, threshold)
      cv2.imshow(“gray roi”, gray_roi)
      cv2.imshow(“Roi”, roi)
      key = cv2.waitKey(30)
      if key == 27:
      break

      cv2.destroyAllWindows()

    2. It means that no contours are detected on the image.
      So cnts is equals to None. You’re trying to sort None and that’s why you get the error.
      you can put an if statement to solve the problem.

      if cnts:
      cnts = sorted(cnts, key=lambda x: cv2.contourArea(x), reverse=True)

    1. Hello Sergio,
      thank you for the explanation, it is really good.
      I am also interested to know how to print the eye positioning (right, left, top, or down).
      Thank you in advance for your support
      Toni

  7. I have the same question as mashie, I’m trying to change the video on the webcam, putting
    cap = cv2.VideoCapture (0)
    but it gives me an error:

    Traceback (most recent call last):
       File “emt.py”, line 45, in
         client.send_message (“/ x”, x)
    NameError: name ‘x’ is not defined
    [WARN: 0] terminating async callback

  8. Can you please help me in running this. I’m using OpenCV version 2.4.9.1
    And I’m getting an error as:
    Traceback (most recent call last):
    File “eye_motion_tracking.py”, line 17, in
    _, contours, _ = cv2.findContours(threshold, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
    ValueError: need more than 2 values to unpack

    Thank You

  9. It seems the problem is the way findContours returns data depending on the installed version.
    Try changing from:
    _, contours, _ = cv2.findContours(threshold, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
    To:
    contours, _ = cv2.findContours(threshold, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)

    And it should work.

  10. im trying to do that but there is an errors i dont know what is wrong

    warning: Error opening file (/build/opencv/modules/videoio/src/cap_ffmpeg_impl.hpp:901)
    warning: eye_recording.flv (/build/opencv/modules/videoio/src/cap_ffmpeg_impl.hpp:902)

  13. There are already the coordinates of the pupil in the code:
    (x, y, w, h) = cv2.boundingRect(cnt)
    similarly ,Can you please mention Iris coordinates if any in your code?

  15. thank you very much sir for this video, please can you let a video be captured from a web camera and again calibrate your x and y coordinates. thank you once again

  16. sir i am trying to plot the graph from the input data from the video, But inside of the while loop graph plot for a every single frame and it overlaps again and again.is that have any idea to resolve this…Thank you

Join the discussion