A new method for frame generation is on the horizon as Intel introduces ExtraSS, its new XeSS framework solution which was shown at SIGGRAPH 23 in Sydney, Australia. The new framework uses frame extrapolation, instead of frame interpolation, which both NVIDIA DLSS 3 and AMD FSR 3.0 utilize in their frame generation approach. While XeSS has proven itself to be a worthy competitor on the super sampling front only time will tell if this new framework can perform equally as well. Extrapolation of data has a downside in that it can introduce additional artifacts to the image but Intel’s approach does have the advantage of reduced latency in comparison to using interpolation, something that is increasingly becoming a greater issue with the added image processing of frame generation technologies.
ExtraSS: A Framework for Joint Spatial Super Sampling and Frame Extrapolation (via Intel):
“We introduce ExtraSS, a novel framework that combines spatial super sampling and frame extrapolation to enhance real-time rendering performance. By integrating these techniques, our approach achieves a balance between performance and quality, generating temporally stable and high-quality, high-resolution results. Leveraging lightweight modules on warping and the ExtraSSNet for refinement, we exploit spatial-temporal information, improve rendering sharpness, handle moving shadings accurately, and generate
temporally stable results. Computational costs are significantly reduced compared to traditional rendering methods, enabling higher frame rates and alias-free high resolution results. Evaluation using Unreal Engine demonstrates the benefits of our framework over conventional individual spatial or temporal super sampling methods, delivering improved rendering speed and visual quality. With its ability to generate temporally stable high-quality results, our framework creates new possibilities for real-time rendering applications, advancing the boundaries of performance and photo-realistic rendering in various domains.”
Interpolation vs. Extrapolation
“Interpolation vs. Extrapolation. Frame interpolation and extrapolation are two key methods of Temporal Super Sampling. Usually frame interpolation generates better results but also brings latency when generating the frames. Note that there are some existing methods such as NVIDIA Reflex [NVIDIA 2020] decreasing the latency by using a better scheduler for the inputs, but they cannot avoid the latency introduced from the frame interpolation and is orthogonal to the interpolation and extrapolation methods. The interpolation methods still have larger latency even with those techniques. Frame extrapolation has less latency but has difficulty handling the disoccluded areas because of lacking information from the input frames. Our method proposes a new warping method with a lightweight flow model to extrapolate frames with better qualities to the previous frame generation methods and less latency comparing to interpolation based methods.”
Even as Intel introduces ExtraSS it acknowledges there are still challenges in making it work. From missing temporal information in offscreen and disoccluded areas to temporal stability, there are still multiple challenges that can introduce artifacts. Intel is using joint temporal and spatial supersampling combined with AI neural networking to combat these issues.