Identify and Measure precisely Objects distance | with Deep Learning and Intel Realsense

Identify and Measure precisely Objects distance

We will see how to check the distance of an object from the camera. This is not an operation that can be done with a simple webcam because it does not provide depth information, you need a camera with an infrared sensor or other dedicated technology for distance measurement. There are several options you can choose from.

So in this tutorial, we will see how to best use the Intel realsense D435i. We have already talked about this device in another tutorial – Distance detection with Depth Camera (Intel Realsense d435i) – but in this case, we will see how to associate the technology with object recognition with artificial intelligence. In any case, for all the technical details, I refer to the intel website.

We see only a small diagram indicating the main cameras.

We’ll see:
How to use the intel realsense d435i and how to install
Use object detection to understand where objects are located on the screen.

Install and load intel realsense D435i

First of all open the terminal (no matter what operating system) and run these commands:

pip install opencv-python

pip install pyrealsense2

Of course also make sure you have Opencv and Python installed otherwise you will get errors and you will not be able to proceed with the project.

To speed up the project I have already prepared some files with all the configurations and functions to be used which will then be recalled in the main file. Make sure you download the files.

Load the file

Remember: these are not libraries so you need to put the files you just downloaded in the same folder as the main file. Calling them up is quite simple, just use this code

import cv2
from realsense_camera import *
from mask_rcnn import *

# Load Realsense camera and Mask R-CNN
rs = RealsenseCamera()
mrcnn = MaskRCNN()

Get BGR frame from realsense camera

As we know, videos are a sequence of frames, generally 30 per second, for this reason, the extraction must take place in a While loop.

while True:
	# Get frame in real time from Realsense camera
	ret, bgr_frame, depth_frame = rs.get_frame_stream()

With the OpenCv command cv2.imshow we can see a first result, but we still don’t have the information for the depth

cv2.imshow("Bgr frame", bgr_frame)

This is the result

Get depth data from realsense camera

To accurately measure the distance of every single pixel we need the measurement relative to our frame. This data is contained in the depth_frame array. If in fact we print the result

We see an array in which the same size of the RGB frame is present, but also the precise distance of each single pixel in millimeters.

It will not be needed for our project but we can do a simple check with this code:

point_x, point_y = 250, 100
distance_mm = depth_frame[point_x, point_y]

cv2.circle(bgr_frame, (point_x, point_y), 8, (0,0,255), -1)

Detect object with AI

For object recognition we use an artificial intelligence model Mask R-CNN, we have already talked about it on this site Instance Segmentation MASK R-CNN | with Python and Opencv .

We use the artificial intelligence model to identify objects, their position, define the contours and the center exactly. Let’s put all this in the arrays. All this information will be obtained from the frame (bgr_frame) extracted in our realsense camera.

	# Get object mask
	boxes, classes, contours, centers = mrcnn.detect_objects_mask(bgr_frame)

With all the coordinates we can draw the outlines of the objects

	# Draw object mask
	bgr_frame = mrcnn.draw_object_mask(bgr_frame)

I always remember that it takes a few seconds to run and I highly recommend running everything through a video card. In the image you can see the first result.

Mask R-CNN correctly identified a person and the bottle. Beyond this, he defined all the contours quite correctly. As a first result, it is satisfactory.

The last thing to do is to merge the data to derive the distance to the object with the object detection.

	# Show depth info of the objects
	mrcnn.draw_object_info(bgr_frame, depth_frame)

With this you get the object distance.

In the final example frame you can see how the object is correctly identified and the class name matches.

Objects distance

As you can see, it can simultaneously recognize multiple objects and accurately indicate the distance of each. The measurement is in real time and this is a huge advantage for the application possibilities in the industrial field.