This is a Python tool that watches a region of the screen and detects objects in it in real time, built on OpenCV. I built it to automate the minigames in a game: the tool reads the screen, finds what matters and responds, so the minigame plays itself.
the detection itself is color and contour tracking: isolate the color you care about, find the contours in the result, then filter them by shape and size. That last step matters more than it sounds. Color alone matches plenty of things that aren't the target, and the shape and size filters are what cut the false positives down to something usable.
on top of that sits a pattern-recognition module that tracks sequences of detections and responds to them, with timing precision finer than a single captured frame. Reacting to a sequence is a different problem from reacting to a single detection, and that fits minigames, which tend to hinge on short timed sequences.
the boring but important piece is the region selector. You pick the capture region once, and it's saved to a config file that gets loaded on the next run, so nobody redoes setup every time. Capturing just a region instead of the whole screen also keeps each frame cheap to process.