Playing Minesweeper with Computer Vision
A bot that plays Minesweeper without touching the game's code or memory, with an overlay to explain the bot's thinking process.
Overview
The Steam game Clean Minesweeper ('Minesweeper') doesn't have any APIs or save files, so the only way to get
information is to use computer vision. The goal was to complete every one of the game's achievements without
having to actually play the game.
A video demonstrating the bot's thinking is located at the bottom of the page.
Reading the Board
When the bot starts, it finds which window contains the game, resizes it, and places it
in the top left of the screen. It takes a screenshot (using mss) and then
OpenCV's template matching is used to find the pixel coordinates for every tile on the grid.
The matches are compared against known board sizes (9x9, 16x16, and 30x16) to ensure that the correct amount
of tiles have been located.
From then on, every turn starts with a new screenshot being taken, and each tile is classified by testing it
against templates for every possible state: unrevealed (base state), blank (confirmed safe), flagged (mine),
or numbered from one to eight. Whichever template scores highest wins, however if the tile doesn't score
above a confidence threshold, it is marked as unknown so the solver knows to treat it with care. A region of
the screen is read with pytesseract to track remaining mine count, which is used in probability
calculations.
Finding Mines: Deterministically
For each numbered tile on the board, the bot builds a constraint: the amount of mines that could exist in
neighbouring cells. Cells that share constraints are grouped into clusters so the solver can look at
independent regions instead of looking at the entire grid.
The solver has two rules: if the constraint's count is zero, then every neighbouring cell must be safe, and,
if the constraint's count equals the number of cells left that are unrevealed then every one is a mine. When
one constraint is a subset of another constraint, then a smaller, tighter constraint is formed. This can
often reveal moves that have been missed previously.
Finding Mines: Exhaustively
The two rules are quick to run, but can't see every deduction. Classic patterns (such as a 1-2-1 pattern)
are logically easy to solve but impossible to solve using the two rules. When the fast solver finishes a
cluster but hasn't resolved every cell in it, the bot falls back to the exhaustive solver: an in depth look
at the cluster.
This solver enumerates every possible mine/safe cell combination of the cluster's cells that doesn't break a
a constraint. If a cell is a mine in every arrangement, it is definitely a mine. Likewise, if it is safe in
every arrangement, then it is definitely safe. Because the exhaustive search is only run on clusters that
the fast solver couldn't finish, the slow search is kept to a minimum.
Finding Mines: Probabilistically
An ideal game of Minesweeper would be able to be solved deterministically, where the safe tiles' and mine's
locations are known with absolute certainty. However, when no cell can be chosen with 100% certainty, then
the bot falls back to picking the cell with the smallest probability of being a mine.
The bot calculates probabilities by generating, for each cluster, every possible mine/safe cell layout that
satisfies its constraints, then weighs each layout by calculating how many ways the remaining mines (read by
OCR) can be arranged over the unrevealed tiles that no number touches. Layouts that leave more room for
remaining mines are more likely to be correct, so cells get assigned probabilities of being a mine.
At every turn, the bot maximises certainty. A
Any probability for a cell that is calculated as 0% (definitely
a mine) or 100% (definitely safe) is clicked first. If nothing is certain, then the bot makes the least
risky guess.
Playing the Game
Every tile is mapped to a screen pixel. PyAutoGUI clicks the screen: a left click reveals
cells, and a right click flags cells containing a mine. On a new board there is nothing to reason about, so
the bot starts by clicking in the middle of the grid.
After each move, OpenCV's template matching is used to detect if the game has been won or not.
The game has an emoji-style graphic at the top of the board which reflects the game state: a happy face
shows a win, a sad face shows a loss, and a neutral face shows a game still in progress.
Watching the Bot Think
A bot that only clicks is hard to trust and even harder to debug, so I have created a transparent overlay
that sits on top of the game's window, built with tkinter using the Win32 API via
pywin32.
The overlay shows text that describes what the bot is doing at that moment: taking a screenshot, reading the
board, running the fast solver, the exhaustive solver, calculating probabilities, or clicking a tile. Every
cell that the solver reasoned about during that turn is labelled with the percentage chance of it being
safe, coloured from red (certainly a mine) to green (certainly safe).
Demo
The video below shows a full run with the overlay switched on.