Kalman filter, predict the trajectory of an Object

In this tutorial, we will see a practical approach on how to use the Kalman filter to track and predict the trajectory of an object. If you need the mathematical explanation I suggest you read the Wikipedia page.

At first, I will show simple examples by drawing dots on the screen and having the trajectory predicted, and then we will see in reality how to predict the trajectory of an orange.

Kalman filter is an algorithm that takes measurements over time and creates a prediction of the next measurements. This is used in many fields such as sensors, GPS, to predict the position in case of signal loss for a few seconds and this is what we will also see in computer vision.

To use Kalman filter these are the requirements

We will see the simplest possible implementation with OpenCV so if you have not yet installed OpenCV I recommend installing opencv-python with pip, then download the necessary files from the download link below.

Follow the instructions carefully and download the material you need to best follow this tutorial.

1. Kalman filter with dots on solid background

In the downloaded files, you will also find the main.py file already complete but in the tutorial I will explain it step by step and I advise you to write it yourself from scratch to better memorize the procedures.

First of all import kalmanfilter.py and the OpenCV library

from kalmanfilter import KalmanFilter
import cv2

Now initialize Kalman filter and we try to insert values to get a prediction

# Kalman Filter
kf = KalmanFilter()

predicted = kf.predict(50,50)

print(predict)

As you can see from the image below, if we print the result, we will get (0,0) because the Kalman filter analysis function needs more values to make a prediction.

More precision with more values

By inserting more position points you will get a more and more precise prediction. Here is an example of code to insert more points that will make our Kalman filter better.

# Kalman Filter
kf = KalmanFilter()

predicted = kf.predict(50,100)
predicted = kf.predict(100,100)
predicted = kf.predict(150,100)
predicted = kf.predict(200,100)

print(predict) # result (238,114)

Simulate the movement of a ball

To verify and put into practice the theory we have seen in the previous paragraph, we simulate the trajectory of a ball that goes from right to left. We can do everything with Opencv by specifying a series of points where the ball will be.

At the same time we pass all points to the function kf.predict(x, y)

from kalmanfilter import KalmanFilter
import cv2

# Kalman Filter
kf = KalmanFilter()

img = cv2.imread("blue_background.webp")

ball_positions = [(50, 100), (100, 100), (150, 100), (200, 100), (250, 100), (300, 100), (350, 100), (400, 100), (450, 100)]

for pt in ball2_positions:
    cv2.circle(img, pt, 15, (0, 20, 220), -1)
    predicted = kf.predict(pt[0], pt[1])

    cv2.circle(img, predicted, 15, (20, 220, 0), 4)

cv2.imshow("Img", img)
cv2.waitKey(0)

By executing the code you can see from the image the solid red balls that go from left to right and the green circles that make the prediction of the next position.

As you can see at the alignment the first green circle is on the point with coordinate (0,0) because it does not have enough values to do the processing. Instead, the last green circle indicates, in an absolutely plausible way, the next possible position of the red ball.

Trying to add more points for prediction with this code

for i in range(10):
    predicted = kf.predict(predicted[0], predicted[1])
    cv2.circle(img, predicted, 15, (20, 220, 0), 4)

    print(predicted)

you can see how the Kalman filter manages to make a correct prediction. In the image below you can see other green circles that assume the position of the red ball

The same procedure is valid if I simulate the launch of a ball whose trajectory draws an arc. Again Kalman filter improves from time to time as new points are awarded. The one in the image below is the result.

2. Use kalman filter to predict the trajectory of real object

In the previous chapter, we used the Kalman filter to predict a simulated red ball, now we will do it with a real object: an orange from a real video. You can already find everything in the orange_prediction.py file but I recommend that you follow the steps carefully.

Detect the object

The first step consists of object detection, in this case of an orange, identified with the color recognition method. We will not dwell on this aspect because it is not the subject of this tutorial and you just need to include the orange_detector.py file to make it work with the video I am using in the example.

We import everything necessary and proceed with obtaining the frames from the video through a loop

import cv2
from orange_detector import OrangeDetector
from kalmanfilter import KalmanFilter

cap = cv2.VideoCapture("orange.mp4")

# Load detector
od = OrangeDetector()

# Load Kalman filter to predict the trajectory
kf = KalmanFilter()

while True:
    ret, frame = cap.read()
    if ret is False:
        break

Now We use the functions for our orange detector and we get the coordinates of 2 points. However, these correspond to the top-left point and the bottom right point of the object. With a small mathematical operation, we obtain the coordinates of the center.

    orange_bbox = od.detect(frame)
    x, y, x2, y2 = orange_bbox
    cx = int((x + x2) / 2)
    cy = int((y + y2) / 2)

Now we have what we need, we just need to add the Kalman filter function to predict the future position of the object.

    predicted = kf.predict(cx, cy)
    #cv2.rectangle(frame, (x, y), (x2, y2), (255, 0, 0), 4)
    cv2.circle(frame, (cx, cy), 20, (0, 0, 255), 4)
    cv2.circle(frame, (predicted[0], predicted[1]), 20, (255, 0, 0), 4)

Showing a small red circle for the current position and a blue circle for prediction through Kalman filter, this is the result extrapolated from a single frame

3. Kalman filter on real scenarios

This algorithm is very important for object tracking. For example, in the case of overlapping objects with a specific ID, it is possible not to confuse them thanks to the prediction of their displacement.

Kalman filter is also used in more complex tracking algorithms such as SORT and Deep SORT and this allows its use in real scenarios. I have covered these topics in my Object Detection (Opencv & Deep Learning) course where there are complete projects with the use of Deep SORT.