Videos come in different sizes, resolutions and aspect ratios; the problem is, so do screens. Try to watch a video shot in a boxy 4:3 width to heighth ratio on your 16:9 HDTV, and things are going to look a little stretched out. Alternatively, you can have your 4:3 image appear in the middle of your 16:9 screen with black margins on the side or enlarge the image to fill the screen, which will necessarily cut off some content.
Resizing video has become more annoying as more people watch on mobile screens of varying sizes, but apparently Google Research has been working on a fix. Today Google Research announced an algorithm for resizing videos that will preserve the width to heighth ratio for “salient” images while squishing and stretching the “non-salient” parts of the shot.
Six frames demonstrating Google’s retargeting algorithm. Original frame is shown on the left, the resized result on the right.
“Think of salient content as actors, faces, or structured objects, where the viewer anticipates specific, important details to perceive it as being correct and unaltered. We cannot change this content beyond uniform scaling without it being noticeable. On the other hand, non-salient content, such as sky, water or a blurry out-of-focus background can be squished or stretched without changing the overall appearance or the viewer noticing a dramatic change,” Google researchers said.
The technique was developed with researchers from Georgia Tech. It’s called “discontinuous seam carving” because it divides the salient and non-salient regions of the image with irregular chains of pixels, or “seams.” The algorithm preserves the aspect ratio for important parts of the image in each frame and minimizes distortion to the non-important parts, so distortion in the resized version is not as obvious to the viewer.
The algorithm processes the video sequentially, frame-by-frame, so it could theoretically be used for streaming video more faithfully to the variously-shaped screens on PCs, mobile devices and the upcoming Google TV box.
The new method is novel and appears to solve some of the problems with resizing video, but it’s got a long way to go before Google integrates it with YouTube. The researchers noted that the algorithm does not always recognized the correct parts of the frame as salient, so they built an interface that allows the user to correct the saliency by highlighting parts of the frame – but it’s hard to picture how this could be incorporated without disrupting the video-watching experience.
Also the algorithm doesn’t always work for fast action sequences or artsier scenes that have too much or too little salient content. “In these cases, just like other approaches, our video retargeting might produce unsatisfactory results,” the researchers said.
Read about the algorithm in more detail in this paper.
