This weekend I took a bit of time to read up on OpenCV (Open Source Computer Vision Library), I wanted to capture images of Magic the Gathering cards, then identify them using a Python library called ImageHash.
Below is a demonstration of what I was able to accomplish in about 2 days of research and hacking:
I’ll try and break down the steps and image manipulation functions I used to achieve this.
1. Capturing Stream from Webcam
Capturing the video stream, then displaying it on screen, is quite simple. What I did was create an endless loop where I capture a single frame, then display that frame. This loop will complete when the q key is pressed.
import cv2 cap = cv2.VideoCapture(0) while True: ret, frame = cap.read() cv2.imshow('frame', frame) if cv2.waitKey(1) == ord('q'): break cap.release() cv2.destroyAllWindows()
For demonstration I’m going to show you a snapshot of the stream through this process:
2. Convert Frame to Grayscale
It is common to convert your image to grayscale, this helps us threshold and identify our contours in later steps:
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
3. Convert Frame to Grayscale
A Threshold allows us to convert our images to absolute black, or absolute white. Here, I’m converting any pixel at color 130 or higher to color 255:
_, thresh = cv2.threshold(gray, 130, 255, cv2.THRESH_BINARY)
4. Find all Contours of Threshold Image
Using my threshold frame, we can find all contours:
_, contours, _ = cv2.findContours(thresh,cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE) sorted_contours = sorted([ (cv2.contourArea(i), i) for i in contours ], key=lambda a:a[0], reverse=True) for _, contour in sorted_contours: cv2.drawContours(frame, [contour], -1, (0, 255, 0), 2)
This ends up looking something like this:
5. Find card contour
Here is where I got lazy and decided to just hack it together.
If you notice in the above image, there is a contour for the entire frame and the next largest contour is our card’s edges. I decided for the time being to hard code our card contour as the 2nd largest contour:
_, card_contour = sorted_contours[1] cv2.drawContours(frame, [card_contour], -1, (0, 255, 0), 2)
6. Drawing Card Corner Points
Next I pass our card_contour into opencv’s minAreaRect function, this gives us 4 points, one for each corner:
rect = cv2.minAreaRect(card_contour) points = cv2.boxPoints(rect) points = np.int0(points) for point in points: cv2.circle(frame, tuple(point), 10, (0,255,0), -1)
7. Identify Corners and Warp Image
This part was a little challenging, but I found an excellent explanation by Adrian over at pyimagesearch.com.
Basically, he takes advantage of numpy and how opencv provides box points in row / columns.
# create a min area rectangle from our contour _rect = cv2.minAreaRect(card_contour) box = cv2.boxPoints(_rect) box = np.int0(box) # create empty initialized rectangle rect = np.zeros((4, 2), dtype = "float32") # get top left and bottom right points s = box.sum(axis = 1) rect[0] = box[np.argmin(s)] rect[2] = box[np.argmax(s)] # get top right and bottom left points diff = np.diff(box, axis = 1) rect[1] = box[np.argmin(diff)] rect[3] = box[np.argmax(diff)] (tl, tr, br, bl) = rect widthA = np.sqrt(((br[0] - bl[0]) ** 2) + ((br[1] - bl[1]) ** 2)) widthB = np.sqrt(((tr[0] - tl[0]) ** 2) + ((tr[1] - tl[1]) ** 2)) maxWidth = max(int(widthA), int(widthB)) heightA = np.sqrt(((tr[0] - br[0]) ** 2) + ((tr[1] - br[1]) ** 2)) heightB = np.sqrt(((tl[0] - bl[0]) ** 2) + ((tl[1] - bl[1]) ** 2)) maxHeight = max(int(heightA), int(heightB)) dst = np.array([ [0, 0], [maxWidth - 1, 0], [maxWidth - 1, maxHeight - 1], [0, maxHeight - 1]], dtype = "float32") M = cv2.getPerspectiveTransform(rect, dst) warped = cv2.warpPerspective(frame, M, (maxWidth, maxHeight))
The warped frame came out looking as follows:
7. Hash Image and Compare
First off, this solution will not scale! Be forewarned.
I grabbed 4 Magic cards from my collection, then grabbed an official image from the Gatherer website and stuck it in a directory.
After that, I hashed my warped image, then iterated over each official image comparing hashes:
from PIL import Image, ImageFilter from glob import glob import imagehash # hash our warped image hash = imagehash.average_hash(Image.fromarray(warped)) # loop over all official images for orig in glob('orig/*.jpeg'): # grayscale, resize, and blur original image orig_image = Image.open(orig).convert('LA') orig_image.resize((maxWidth, maxHeight)) orig_image.filter(ImageFilter.BLUR) # hash original and get hash orig_hash = imagehash.average_hash(orig_image) score = hash - orig_hash print('Comparing image to {}, score {}'.format( orig, score )) print('-' * 50)
And that is it, the Goldenglow Moth card resulted in the following print:
Comparing image to orig/doom_blade.jpeg, score 25 Comparing image to orig/forst_breath.jpeg, score 26 Comparing image to orig/goldenglow_moth.jpeg, score 7 Comparing image to orig/kessig_wolf.jpeg, score 13 --------------------------------------------------
since you grayscaled the orignal images, should the warped image be grayscaled too?
Well the images I’m comparing the warped image against are not grey-scaled, they were high resolution downloads from http://gatherer.wizards.com/Pages/Default.aspx, so I don’t think that would be needed.
How are you even getting the snapshots from the video? After step 1, I’m not sure where does gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY) go, I think we missed a step. Thanks for this by the way.
Well in part #1 I’m using cv2’s VideoCapture method in a infinite loop to demonstrate streaming from a webcam, on each iteration I save a single snapshot (or picture) from the webcam stream as a variable named frame:
In the following steps it is assume the loop was exited, and we work directory with the frame (our single image from webcam).
I hope this helped.
Hi, i get the next error ” Expected cv::UMat for argument ‘points’ “. The error is located in this line of the code ” rect = cv2.minAreaRect(card_contour) ” how can i fix this error? thanks for showing us your amazing code!!!