Researchers at Adobe have collaborated with the Beckman Institute for Advanced Science and Technology to develop a new system, based on deep convolutional neural networks, which can extract foreground content from its background intelligently and accurately – and with no need for the blue/green-screen techniques which have dominated cinema for nearly a century.
The paper Deep Image Matting outlines the process of evaluating the object which needs to be ‘clipped’ out of its background, which involved the generation of a novel dataset containing 49300 training images intended to accustom algorithms with the challenges of distinguishing backgrounds and eliminating them.
Traditional methods of extracting actors or elements from backgrounds, so that they can be inserted into other footage, have always centred around recording the elements (actors, miniatures, etc.) to be extracted in front of a flat field of colour, and relying on photochemical or (later) digital procedures to remove the background.
In earlier times, film production workflows generally used blue as a key colour to remove, though Walt Disney studios (which famously took on visual effects work for Alfred Hitchcock’s chiller The Birds) used a sodium-based process which keyed on yellow – however, its greater accuracy was offset by the complexity and weight of the equipment required, and the sodium process never gained widespread industry popularity.
In the last 15-20 years, green has been adopted as a drop-out colour, since it was proved to be present in less foreground material than blue (for the filming of Superman in the late 1970s, it proved necessary to shoot title actor Christopher Reeve in a costume which was nearer violet than the traditional blue of the man of steel, and to tweak the costume’s colour chemically later, so that Reeve did not completely disappear when extracted from a blue background).
Adobe has been at the forefront of this field for at least 27 years. Acclaimed visual effects producer John Knoll, then working as an employee at Lucasfilm’s Industrial Light and Magic (ILM) effects facility, worked with his brother Thomas in the late 1980s to develop the early versions of Photoshop, which pioneered the digital alpha matte and ultimately joined CGI to transform the visual effects landscape. Later these principles were incorporated into the video-effects suite After Effects, and an industry-wide raft of programs which put the Chroma Key compromises of the 1970s to shame.
But the prospect of casual background removal via the use of neural networks seems likely to be a game-changer not only for the VFX industry but also for the much more potentially lucrative VR/AR sphere – which has some issues to face in this particular regard. Naturally latency will be a critical issue in any AI-driven approach to foreground extraction.