This weekend I took a bit of time to read up on OpenCV (Open Source Computer Vision Library), I wanted to capture images of Magic the Gathering cards, then identify them using a Python library called ImageHash.
Below is a demonstration of what I was able to accomplish in about 2 days of research and hacking:
I’ll try and break down the steps and image manipulation functions I used to achieve this.
1. Capturing Stream from Webcam
Capturing the video stream, then displaying it on screen, is quite simple. What I did was create an endless loop where I capture a single frame, then display that frame. This loop will complete when the q key is pressed.
import cv2
cap = cv2.VideoCapture(0)
while True:
ret, frame = cap.read()
cv2.imshow('frame', frame)
if cv2.waitKey(1) == ord('q'):
break
cap.release()
cv2.destroyAllWindows()
For demonstration I’m going to show you a snapshot of the stream through this process:
2. Convert Frame to Grayscale
It is common to convert your image to grayscale, this helps us threshold and identify our contours in later steps:
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
3. Convert Frame to Grayscale
A Threshold allows us to convert our images to absolute black, or absolute white. Here, I’m converting any pixel at color 130 or higher to color 255:
_, thresh = cv2.threshold(gray, 130, 255, cv2.THRESH_BINARY)
4. Find all Contours of Threshold Image
Using my threshold frame, we can find all contours:
_, contours, _ = cv2.findContours(thresh,cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
sorted_contours = sorted([ (cv2.contourArea(i), i) for i in contours ], key=lambda a:a[0], reverse=True)
for _, contour in sorted_contours:
cv2.drawContours(frame, [contour], -1, (0, 255, 0), 2)
This ends up looking something like this:
5. Find card contour
Here is where I got lazy and decided to just hack it together.
If you notice in the above image, there is a contour for the entire frame and the next largest contour is our card’s edges. I decided for the time being to hard code our card contour as the 2nd largest contour:
_, card_contour = sorted_contours[1] cv2.drawContours(frame, [card_contour], -1, (0, 255, 0), 2)
6. Drawing Card Corner Points
Next I pass our card_contour into opencv’s minAreaRect function, this gives us 4 points, one for each corner:
rect = cv2.minAreaRect(card_contour)
points = cv2.boxPoints(rect)
points = np.int0(points)
for point in points:
cv2.circle(frame, tuple(point), 10, (0,255,0), -1)
7. Identify Corners and Warp Image
This part was a little challenging, but I found an excellent explanation by Adrian over at pyimagesearch.com.
Basically, he takes advantage of numpy and how opencv provides box points in row / columns.
# create a min area rectangle from our contour
_rect = cv2.minAreaRect(card_contour)
box = cv2.boxPoints(_rect)
box = np.int0(box)
# create empty initialized rectangle
rect = np.zeros((4, 2), dtype = "float32")
# get top left and bottom right points
s = box.sum(axis = 1)
rect[0] = box[np.argmin(s)]
rect[2] = box[np.argmax(s)]
# get top right and bottom left points
diff = np.diff(box, axis = 1)
rect[1] = box[np.argmin(diff)]
rect[3] = box[np.argmax(diff)]
(tl, tr, br, bl) = rect
widthA = np.sqrt(((br[0] - bl[0]) ** 2) + ((br[1] - bl[1]) ** 2))
widthB = np.sqrt(((tr[0] - tl[0]) ** 2) + ((tr[1] - tl[1]) ** 2))
maxWidth = max(int(widthA), int(widthB))
heightA = np.sqrt(((tr[0] - br[0]) ** 2) + ((tr[1] - br[1]) ** 2))
heightB = np.sqrt(((tl[0] - bl[0]) ** 2) + ((tl[1] - bl[1]) ** 2))
maxHeight = max(int(heightA), int(heightB))
dst = np.array([
[0, 0],
[maxWidth - 1, 0],
[maxWidth - 1, maxHeight - 1],
[0, maxHeight - 1]], dtype = "float32")
M = cv2.getPerspectiveTransform(rect, dst)
warped = cv2.warpPerspective(frame, M, (maxWidth, maxHeight))
The warped frame came out looking as follows:
7. Hash Image and Compare
First off, this solution will not scale! Be forewarned.
I grabbed 4 Magic cards from my collection, then grabbed an official image from the Gatherer website and stuck it in a directory.
After that, I hashed my warped image, then iterated over each official image comparing hashes:
from PIL import Image, ImageFilter
from glob import glob
import imagehash
# hash our warped image
hash = imagehash.average_hash(Image.fromarray(warped))
# loop over all official images
for orig in glob('orig/*.jpeg'):
# grayscale, resize, and blur original image
orig_image = Image.open(orig).convert('LA')
orig_image.resize((maxWidth, maxHeight))
orig_image.filter(ImageFilter.BLUR)
# hash original and get hash
orig_hash = imagehash.average_hash(orig_image)
score = hash - orig_hash
print('Comparing image to {}, score {}'.format(
orig, score
))
print('-' * 50)
And that is it, the Goldenglow Moth card resulted in the following print:
Comparing image to orig/doom_blade.jpeg, score 25 Comparing image to orig/forst_breath.jpeg, score 26 Comparing image to orig/goldenglow_moth.jpeg, score 7 Comparing image to orig/kessig_wolf.jpeg, score 13 --------------------------------------------------