Skip to content

Gaze controlled keyboard · Tutorials

Write using your Eyes – Gaze controlled keyboard with Python and Opencv p.10

February 7, 2019 Sergio Canu 2 minute read
Access community, courses and source codes
Logo

AI Vision Academy

Access the code of this tutorial, computer vision courses and an exclusive community on AI Vision Academy

  • Access to over 50+ source codes from Pysource.com/blog
  • Dedicated video courses about computer vision
  • Access to an exclusive community of professionals
  • Real-World AI Projects – Get hands-on experience building practical AI Computer Vision solutions with a structured path.
  • Monthly Coaching Calls – get support and any of your questions answered

Subscribe to our newsletter to learn more

This is the final tutorial of the project “Gaze controlled keyboard with Python and Opencv”.

The code is approximately 300 lines. I added the missing parts to the code, I improved the graphic of the keyboard and I made a few adjustments. I’m not going to do any coding in this tutorial, instead I will show you the result of the final project and on the video I’ll do a quick review of the entire code.

How does this work?

Once we run the script we see see three different windows containing respectively: the keyboard, a white board and the capture from the webcam.

The keyboard is divided in two parts, and before pressing each letter we need to select the part.
The left side contains the letters: Q, W, E, R, T, A, S, D, F, G, Z, X, C, V.
The right side contains the letters: Y, U, I, O, P, H, J, K, L, V, B, N, M and spacebar.

1) Choose keyboard’s side:

The webcam is controlling the position of your eyes, so if you want to select the left side you need to look on the left for around a second, for the opposite side you need to look on the right.

Keyboard side

2) Press a key

Let’s supposed that we choosed the Right side.
Now we will see the keys contained on the right side.
To press a key we need to wait that the key lights up, and then we blink our eyes. We keep them closed until we hear a sound. That sound means that the key has been pressed.

3) Repeat step 1 and 2 to press a new key.

Each time we press a key, we need to repeat the step 1 and 2.

Each time the key is pressed, it is added to the white board. For example I typed the word “Hello” and below you see the result.

Final considerations:

This project was realised with the only purpose to teach you how to detect the eyes, theyr gaze and the blinking.

A few parts regarding the eye detection have been hardcoded to work specifically with my eyes and with a specific lightening condition and with a certain webcam.
This means that if some of these thress variables is different for you, the code might not work properly or might not work at all.

It’s of course possible to improve it for production environment.
If you’re the owner of a company and you want to develop this project further, feel free to contact me.

34 comments

  2. Hi,

    Before anything tanks.
    I’d like to tell you how much i like your video tutorials.
    I’m from Brazil and you explain your projects very well and, please, don’t change your way to teach.
    Keep beeing the teacher you are.

    1. Hi, I’m slowly fixing all the posts in my blog and putting the code so it will take some times before I get to this post.
      I’m going to send you the code via mail.

      3. Hi,I copied your code.It is working but I have sound error.When I choose left ,coming left sound is non-stopping and right sound also.Please help me…

  7. having issues with the dlib installation …..Using pycharm and also system python
    ERROR: Command “‘C:\Users\12267\PycharmProjects\Tracker\venv\Scripts\python.exe’ -u -c ‘import setuptools, tokenize;__file__='”‘”‘C:\\Users\\12267\\AppData\\Local\\Temp\\pip-install-7ro0gxrn\\dlib\\setup.py'”‘”‘;f=getattr(tokenize, ‘”‘”‘open'”‘”‘, open)(__file__);code=f.read().replace(‘”‘”‘\r\n'”‘”‘, ‘”‘”‘\n'”‘”‘);f.close();exec(compile(code, __file__, ‘”‘”‘exec'”‘”‘))’ install –record ‘C:\Users\12267\AppData\Local\Temp\pip-record-jw8rjwoy\install-record.txt’ –single-version-externally-managed –compile –install-headers ‘C:\Users\12267\PycharmProjects\Tracker\venv\include\site\python3.6\dlib'” failed with error code 1 in C:\Users\12267\AppData\Local\Temp\pip-install-7ro0gxrn\dlib\

  8. Hi Sir,
    When I try this code i encounter the “dlib error” using python version 3.7.4.Can you please help me to solve this problem.And I tried to download the dlib but I don’t know how to solve the error.

  11. Thank you very much. Useful lessons and examples. I’ve checked it on Ubuntu 18.04 with Python 3.7.5/Anaconda 4.6.11. Everything working but not too fast (~2-3 FPS on Pentium G640T 2.40GHz)
    Greetings from POLAND.

    1. They’re not implemented, but it will be possible integrating some python library to interact with the keyboard commands.
      My idea for this project is only a prototype, and of course it can be developed further and improved in many aspects.

  15. I tried a lot of ways but didnt got solution
    detector = dlib.get_frontal_face_detector()
    AttributeError: module ‘dlib’ has no attribute ‘get_frontal_face_detector’

    1. Hi Abdul,
      Yes and it’s simple. Usually when we add a letter we add a specific character to the string text.
      If you add the backspace, you simply can remove the last character from the text when click on backspace.

      After selecting the backspace:
      text = text[:-1]

  17. Sir, Thanks for your wonderful tutorial. It really helped me a lot to work on. But there’s a slight issue, can you help me with it?

    When the blink detection is recognised on a particular letter, it gets paused over there and the letter isn’t getting printed. Any solution for it?

Join the discussion