Detect tetris board and Tetrominoes – Python Plays Tetris p. 3

We’re going to see in this tutorial how to detect the tetris board and the tetrominoes from this image below.

How to detect the board?

There are more methods that we can use to detect the board.

• Color detection: the board is the biggest black square on the screen, so we could detect the black color.
• Edge detection: an edge is when there is a sharp change in intensity or color, so all the edges here will be the boundaries of the different elements.
• Template matching: we could cut manually with an image editor the board from the screenshot and use it as a template to be found on the original screenshot.

Probably also other methods are going to work as well, as itself is a quite simple detection.

For this project I will stick with the easier and most reliable one, which is color detection.

First we need to use a color picker tool on Gimp, to know exactly the BGR values of the board, then we use these values and we create a variable with them on line 11.

This is enough then to find the contours of the entire board. As the same color is used also outside the board, we sort all the contours by their area and we pick only the first one, which is the bigges one (so the board).

# Detecting the board
board_color = np.array([35, 35, 36])
board_mask = cv2.inRange(img, board_color, board_color)
_, contours, _ = cv2.findContours(board_mask, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
contours = sorted(contours, key=lambda x: cv2.contourArea(x), reverse=True)

We can then draw the contours sorrounding the board, to make sure that the detection is correct. In the image below we can see that a green line is exactly sorrounding the board, that means that it was correct.

We also create a virtual board where we are going to replicate all the elements detected, board and tetrominoes.

cnt = contours[0]
(board_x, board_y, board_w, board_h) = cv2.boundingRect(cnt)
cv2.drawContours(img, [cnt], -1, (0, 255, 0), 3)
cv2.drawContours(virtual_board, [cnt], -1, (0, 255, 0), 3)

How to detect the tetrominoes?

We have 7 types of tetrominoes and we can call them with a letter based on their shape.

In order in the picture below reading from elft to right we have the: I, J, L, O, Z, T, S tetrominoes.

To detect them we could use similar approaches to the ones we could use for the board detection, with the the major difficulty here that the tetrominoes can move around the board and can rotate.

Considering that each tetrominoe has a different color, it’s an easy choice to go with the color detection even for them.

Let’s first assign the color to each tetrominoe:

# Detecting Tetrominoes
tetrominoes = {"i_polyomino": [116, 98, 0],
               "o_polyomino": [0, 102, 116],
               "t_polyomino": [127, 0, 106],
               "j_polyomino": [127, 67, 0],
               "l_polyomino": [0, 85, 127],
               "s_polyomino": [35, 127, 0],
               "z_polyomino": [0, 0, 116]}

Once we stored the color values for each tetrominoe we can proceed by detecting them.
Simply we can loop trough the colors and if anything is detected for that color, we draw a contour.

for key in tetrominoes:
    bgr_color = tetrominoes[key]
    bgr_color = np.array(bgr_color)
    mask = cv2.inRange(img, bgr_color, bgr_color)
    _, contours, _ = cv2.findContours(mask, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
    for cnt in contours:
        (x, y, w, h) = cv2.boundingRect(cnt)
        cv2.rectangle(virtual_board, (x, y), (x + w, y + h),
                      (0, 0, 255), 2)

To make sure that the detection was done correctly we can compare the screenshot with the virtualboard created.
On the virtual board we drew all the elements detected.

The detection is working fine, if you notice that some part of the S tetrominoe is missing in the picture below, is just because I stopped the loop before showing everything on the screen, but also that one works well.

In the next tutorial we will see how to create the “Intelligence” which plays the game.
Stay tuned.