Home Google Research Carves Space-Time to Optimize Video for Any Size Screen

Google Research Carves Space-Time to Optimize Video for Any Size Screen

Videos come in different sizes, resolutions and aspect ratios; the problem is, so do screens. Try to watch a video shot in a boxy 4:3 width to heighth ratio on your 16:9 HDTV, and things are going to look a little stretched out. Alternatively, you can have your 4:3 image appear in the middle of your 16:9 screen with black margins on the side or enlarge the image to fill the screen, which will necessarily cut off some content.

Resizing video has become more annoying as more people watch on mobile screens of varying sizes, but apparently Google Research has been working on a fix. Today Google Research announced an algorithm for resizing videos that will preserve the width to heighth ratio for “salient” images while squishing and stretching the “non-salient” parts of the shot.

Six frames demonstrating Google’s retargeting algorithm. Original frame is shown on the left, the resized result on the right.

“Think of salient content as actors, faces, or structured objects, where the viewer anticipates specific, important details to perceive it as being correct and unaltered. We cannot change this content beyond uniform scaling without it being noticeable. On the other hand, non-salient content, such as sky, water or a blurry out-of-focus background can be squished or stretched without changing the overall appearance or the viewer noticing a dramatic change,” Google researchers said.

The technique was developed with researchers from Georgia Tech. It’s called “discontinuous seam carving” because it divides the salient and non-salient regions of the image with irregular chains of pixels, or “seams.” The algorithm preserves the aspect ratio for important parts of the image in each frame and minimizes distortion to the non-important parts, so distortion in the resized version is not as obvious to the viewer.

The algorithm processes the video sequentially, frame-by-frame, so it could theoretically be used for streaming video more faithfully to the variously-shaped screens on PCs, mobile devices and the upcoming Google TV box.

The new method is novel and appears to solve some of the problems with resizing video, but it’s got a long way to go before Google integrates it with YouTube. The researchers noted that the algorithm does not always recognized the correct parts of the frame as salient, so they built an interface that allows the user to correct the saliency by highlighting parts of the frame – but it’s hard to picture how this could be incorporated without disrupting the video-watching experience.

Also the algorithm doesn’t always work for fast action sequences or artsier scenes that have too much or too little salient content. “In these cases, just like other approaches, our video retargeting might produce unsatisfactory results,” the researchers said.

Read about the algorithm in more detail in this paper.

About ReadWrite’s Editorial Process

The ReadWrite Editorial policy involves closely monitoring the tech industry for major developments, new product launches, AI breakthroughs, video game releases and other newsworthy events. Editors assign relevant stories to staff writers or freelance contributors with expertise in each particular topic area. Before publication, articles go through a rigorous round of editing for accuracy, clarity, and to ensure adherence to ReadWrite's style guidelines.

Get the biggest tech headlines of the day delivered to your inbox

    By signing up, you agree to our Terms and Privacy Policy. Unsubscribe anytime.

    Tech News

    Explore the latest in tech with our Tech News. We cut through the noise for concise, relevant updates, keeping you informed about the rapidly evolving tech landscape with curated content that separates signal from noise.

    In-Depth Tech Stories

    Explore tech impact in In-Depth Stories. Narrative data journalism offers comprehensive analyses, revealing stories behind data. Understand industry trends for a deeper perspective on tech's intricate relationships with society.

    Expert Reviews

    Empower decisions with Expert Reviews, merging industry expertise and insightful analysis. Delve into tech intricacies, get the best deals, and stay ahead with our trustworthy guide to navigating the ever-changing tech market.