How work Optical Flow?

Optical flow is the vector who describe the moviment of the object betweent two consecutive frames.

If I want to track an object what can I do? There is an algorithm called Lucas-Kanade, let’s take this toothbrush as an example, I indicate a point that I want to trace in a specific area. The algorithm checks the object in the previous area and if it does not find it it assumes that it has moved and then selects a new point.

For more details on how the function works, I suggest you read the official OpenCv guide: Optical Flow with Lucas-Kanade method

The Lucas-Kanade algorithm needs some conditions to work. For example, the object must move really close, no matter if it is fast because the algorithm analyzes frame by frame.

Optical flow with Lucas-Kanade example code

The first step is to call the usual OpenCV and Numpy in our code, then we have to write the code to retrieve the data frame by frame from the camera. I recommend you also read my article on the subject: Control webcam with servo motor and raspberry pi – Opencv with Python which could help you with your project.

Select the point

The first necessary step is to select a point. To do this you need a mouse callback function

# Mouse function
def select_point(event, x, y, flags, params):
    global point, point_selected, old_points
    if event == cv2.EVENT_LBUTTONDOWN:
        point = (x, y)
        point_selected = True
        old_points = np.array([[x, y]], dtype=np.float32)

cv2.namedWindow("Frame")
cv2.setMouseCallback("Frame", select_point)

point_selected = False
point = ()
old_points = np.array([[]])

We need to initialize the arrays and remember that to be used with OpenCV they must be converted to nparray.

calcOpticalFlowPyrLK() function

Now we can use the calcOpticalFlowPyrLK () tracking function. The first thing to do is to convert the frame to grayscale format.

    _, frame = cap.read()
    gray_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

We therefore consider the parameters to be used in the optical-flow function:

  • old_gray : old frame array
  • gray_frame : current frame
  • old_point : array with old point
  • None : empy param
  • **lk_params : some configuration about the function
# Lucas kanade params
lk_params = dict(winSize = (15, 15),
maxLevel = 4,
criteria = (cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 10, 0.03))

...

        new_points, status, error = cv2.calcOpticalFlowPyrLK(old_gray, gray_frame, old_points, None, **lk_params)
        old_gray = gray_frame.copy()
        old_points = new_points

        x, y = new_points.ravel()

I have tried to make this as simple as possible and surely you can take more points as a reference and everything can be improved.