How AI Background Removal Works (And Why It Matters for Your Twin)
A behind-the-scenes look at the on-device matting model that powers Hey Twin's pixel-perfect cutouts — and how it differs from typical background removers.
Priya Natarajan
ML Engineer
“Background removal” sounds like a solved problem — plenty of apps do it. But the difference between a usable cutout and a convincing one comes down to a step most tools skip: matting.
Segmentation vs. matting
Most background removers use segmentation — drawing a hard line around a subject, pixel by pixel, and either keeping or discarding each one. It’s fast, but it struggles with anything that isn’t a solid edge: hair strands, fuzzy sweaters, glasses frames, motion blur. The result is often a subject with a faint white or dark halo where the old background almost-but-not-quite got removed.
Hey Twin instead uses alpha matting, which estimates a partial transparency value for edge pixels instead of a binary in-or-out decision. A strand of hair might be rendered at 40% opacity rather than fully kept or fully cut — which is much closer to how that edge actually looks against a new background.
Why this needs to run on-device
Cloud-based matting models tend to be larger and slower, with a network round-trip on top. We optimized our model specifically to run on Apple’s Neural Engine, which let us hit two goals at once:
- Speed: most cutouts complete in under two seconds, with no upload wait.
- Privacy: your source photo never has to leave your iPhone to get a high-quality cutout.
The trade-off is real — an on-device model has to be smaller than a massive cloud model — so we spent the bulk of our development time on training data quality rather than raw model size: thousands of hand-annotated edge cases covering hair, semi-transparent fabric, and reflective surfaces like glasses.
What this means for your Twin
In practice, it means the Twins you create hold up under real scrutiny — zoom in on a hairline in AR and it won’t fall apart into a jagged edge. It’s also why we recommend well-lit, in-focus source photos: the model has more genuine edge information to work with, which shows up directly in cutout quality.
If a cutout still isn’t quite right, the manual edge refinement tool (tap any Twin > Edit Edges) lets you nudge the matting boundary by hand for tricky spots — useful for busy backgrounds or unusual hairstyles.
Curious about the AR side of the equation? Read how we match lighting and scale once your Twin is ready to place.