One of the highlights of the Adobe MAX show is their Sneaks segment, where they show off cute demos from their research labs. Some, but not all of these end up in future products. They can be related to any product category, from audio editing to photo touch up to video. This one was hosted by Kumail Nanjiani, a comedian and actor on HBO’s Silicon Valley, and Paul Trani of Adobe.
For a complete list, check out their blog post, but I’ll highlight some that interest me.
First, Project Cloak is like content-aware fill, but for video. Using a command-line tool, because research projects are research projects, the algorithm follows a mask as it moves over time. It then performs content-aware fill on this mask, and it does so in a way that tries to look natural from frame-to-frame. About two-thirds of the way through the clip, you see a backpack strap be removed, even as lighting conditions change throughout the environment. They then remove the whole people from the scene because why not.
Second, Project Sidewinder allows 3D VR video to be viewed from slightly different angles. If you play PC games, then you will probably notice the effect devolve like parallax occlusion mapping does, but that’s not its goal. The human brain tracks the tiny movements of the head and uses it to build a 3D environment in our mind. Since these movements are tiny, it should stay within the believable tolerance and enhance the immersion… from a stationary, pre-recorded camera.
Third, Project Scene Stitch. This one looks very practical, and it seems like something Adobe will want to roll out. Content-aware fill is an algorithm that looks at your selection, tries to find other places on the image that is compatible, and pastes from there into the fill. Project Scene Stitch is similar, except that it crawls through Adobe Stock (using their Adobe Sensei deep-learning algorithm) to find parts of other images that are compatible. They joke about how it breaks down in certain situations, but I could definitely see this being a part of their Adobe Stock push going forward.