1. Averaging: the naive fill
Take the colours around the hole and paint it with their average. It is fast and it always looks wrong, because real images are not flat. Any line or texture that enters the box stops at its edge.
2. Diffusion: let the edges bleed in
Repeatedly replace each hidden pixel with the average of its four neighbours, and the border colours seep inward until everything settles. Mathematically this solves the Laplace equation inside the hole, and it gives the smoothest possible fill. Bertalmío and colleagues’ 2000 SIGGRAPH paper Image Inpainting refined the idea to push edges along their direction. Diffusion is ideal for skies, skin and gradients, and useless for texture: bricks, grass and fabric turn to mush. The video tool’s “fill from edges” is a one-step cousin of this, the same weighted blend FFmpeg’s delogo filter uses.
3. Patch copying: borrow from the photo itself
Instead of mixing colours, copy whole small patches from elsewhere in the image that match what surrounds the hole. Criminisi, Pérez and Toyama (2004) filled holes patch by patch, and Wexler, Shechtman and Irani (2007) refined all patches together. Barnes et al.’s PatchMatch (2009) made the search fast enough for interactive use, and it powers Photoshop’s Content-Aware Fill. CleanMarks’ on-device eraser is a multi-scale PatchMatch: it first fills a tiny version of your photo, then refines at each larger size, so large structures line up before fine texture is added.
Patch copying keeps the photo’s real grain and colour, and it needs no server. Its limit: it can only reuse what exists. If a watermark crosses the only eye in a portrait, no patch elsewhere is a matching eye.
4. Generative AI inpainting
Diffusion image models (the same family as text-to-image generators) are trained on millions of photos and learn what usually belongs in a scene. Given your photo and a mask, a model such as FLUX.1 Fill redraws only the masked area, conditioned on everything around it. It can continue a shoulder, a railing or a letterform that exists nowhere else in the image. The trade-offs: it runs on a GPU server, it costs money per image, and it invents content, so it is a plausible guess and never a recovery.
That is why CleanMarks gives you both: patch copying on your device for texture and backgrounds, and AI fill only for the strokes that need it. Only a crop around your mark is sent, and the result is pasted back strictly inside your brush strokes.
Why video is different
A video is 24–60 photos per second. Generative inpainting on each frame is slow and expensive, and frames filled independently flicker, because each guess is slightly different. Professional tools track the mark and borrow the hidden pixels from neighbouring frames where the camera moved. In a browser, a fast deterministic fill (blend from edges, blur or pixelate) is the practical choice. See it at work in the video watermark remover, which includes its own pixel-level demo.
What no remover can do
- Recover hidden content. A face under an emoji, or text under a black bar, is gone from the file. AI can only fabricate a replacement.
- Remove a watermark cleanly when it covers everything. Tiled marks leave no clean reference. That’s why they protect best.
- Make removal legal. The technology is neutral, and the copyright in someone else’s photo is not.
If you like learning this way, ahaboo has narrated interactive explainers on how everyday things work, from why the seasons happen to how compound interest grows.
Questions
What is inpainting?
Inpainting is filling in a missing or unwanted region of an image so the result looks natural. Watermark removers, object erasers and photo restoration tools all use it.
Can a watermark remover recover the original pixels?
No. The pixels under an opaque mark are gone. A remover produces a plausible replacement, which is why detailed areas can look slightly different from the original.
Is AI inpainting better than Content-Aware Fill?
For structured content (faces, hands, straight edges, text-like detail) generative AI is usually better. For repeating texture, patch-based fill is fast, exact to the photo’s own grain, and needs no server.