Taking screenshots with OpenCV and Python
Today’s blog post is broken down into two sections.
In the first section, we’ll learn how to install the PyAutoGUI library. This library is responsible for actually capturing our screenshots to disk or directly to memory.
From there we’ll learn how to use PyAutoGUI and OpenCV together to obtain our screenshots.
Installing PyAutoGUI for screenshots
You can find instructions for installing PyAutoGUI in their install documentation; however, as a matter of completeness, I have included the instructions below.
I highly recommend that you install the PyAutoGUI into your Python virtual environment for computer vision (as we have done for all other install tutorials here on PyImageSearch).
Discussing virtual environments in detail is beyond the scope of this blog post; however, I encourage you to set up an environment for computer vision (including OpenCV and other tools) by following the installation instructions for your system available here.
Installing PyAutoGUI for macOS is very straightforward. As stated above, you’ll want to be sure you’re “inside” your virtual environment prior to executing the following pip commands:
Ubuntu or Raspbian
To install PyAutoGUI for Ubuntu (or Raspbian), you’ll need to make use of both Aptitude and pip. Again, before the pip commands, be sure that you’re working on your Python virtual environment:
Screenshots and screen captures with OpenCV and Python
Now that PyAutoGUI is installed, let’s take our first screenshot with OpenCV and Python.
Open up a new file, name it take_screenshot.py , and insert the following code:
On Lines 2-5 we’re importing our required packages, notably pyautogui .
From there we’ll take a screenshot via two different methods.
In the first method, we take the screenshot and store it in memory for immediate use:
# take a screenshot of the screen and store it in memory, then
# convert the PIL/Pillow image to an OpenCV compatible NumPy array
# and finally write the image to disk
Line 10 shows that we’re grabbing a screenshot with pyautogui.screenshot and storing it as image (again, this image is stored in memory it is not saved to disk).
Not so fast!
PyAutoGUI actually stores the image as a PIL/Pillow image, so we need to perform an additional step before the image can be used with OpenCV.
On Line 11 we convert the image to a NumPy array and swap the color channels from RGB ordering (what PIL/Pillow uses) to BGR (what OpenCV expects). That’s all that’s required for making our screenshot image OpenCV-compatible.
From here the sky is the limit with what you can do. You could detect buttons displayed on the screen or even determine the coordinates of where the mouse is on the screen.
We won’t do either of those tasks today. Instead, let’s just write the image to disk with cv2.imwrite to ensure the process worked correctly (Line 12).
The second method (where we write the screenshot to disk) is even easier:
# this time take a screenshot directly to disk
As shown, this one-liner writes the image straight to disk. Enough said.
We could stop there, but for a sanity check, let’s make sure that OpenCV can also open + display the screenshot:
# we can then load our screenshot from disk in OpenCV format
Here, we read the image from disk. Then we resize and display it on the screen until a key is pressed.
As you can tell, PyAutoGui is dead simple thanks to the hard work of Al Sweigart.
Let’s see if it worked.
To test this script, open up a terminal and execute the following command:
And here’s our desktop screenshot shown within our desktop…proving that the screenshot was taken and displayed:
Notice how in the terminal the Python script is running (implying that the screenshot is currently being taken).
After the script exits, I have two new files in my working directory: in_memory_to_disk.png and straight_to_disk.png .
Let’s list contents of the directory:
drwxr-xr-x@5adrian staff 160Jan0110:04.
drwxr-xr-x@8adrian staff 256Jan0120:38..
-rw-r--r--@1adrian staff 4348537Jan0109:59in_memory_to_disk.png
-rw-r--r--@1adrian staff 5248098Jan0109:59straight_to_disk.png
-rw-r--r--@1adrian staff 703Jan0109:59take_screenshot.py
As you can see, I’ve got my take_screenshot.py script and both screenshot PNG images
Now that we have our screenshot in OpenCV format, we can apply any “standard” computer vision or image processing operation that we wish, including edge detection, template matching, keypoint matching, object detection, etc.
In a future blog post, I’ll be demonstrating how to detect elements on a screen followed by controlling the entire GUI from the PyAutoGUI library based on what our computer vision algorithms detect.
Stay tuned for this post in early 2018!
In today’s blog post we learned how to take screenshots using OpenCV, Python, and the PyAutoGUI library.
Using PyAutoGUI we can easily capture screenshots directly to disk or to memory, which we can then convert to OpenCV/NumPy format.
Screenshots are an important first step when creating computer vision software that can automatically control GUI operations on the screen, including automatically moving the mouse, clicking the mouse, and registering keyboard events.
In future blog posts, we’ll learn how we can automatically control our entire computer via computer vision and the PyAutoGUI.
To be notified when future blog posts are published here on PyImageSearch, just enter your email address in the form below!