NVIDIA QUEEN: an AI model that allows you to change the angle of viewing video in real-time at 350 fps
NVIDIA, in collaboration with the University of Maryland, has introduced a new AI model called QUANTIZED Efficient ENcoding ( QUUE ) that allows for fast and efficient reconstruction of dynamic 3D scenes, even in real time. This technology can be used in free-view video streaming.
Here's What We Know
The model's special feature is its ability to efficiently compress data while maintaining high image quality. This allows it to be used in areas where speed and accuracy are important: sports, medicine, industry, and media. QUEEN uses the 3D Gaussian Splatting method to create streaming videos with a free viewpoint. It is capable of training in less than 5 seconds and rendering scenes at approximately 350 frames per second, reducing the model size to 0.7 MB per frame. This is achieved by efficiently encoding residual attributes between consecutive frames and using a quantum sparse structure to compress the data. This is how it looks like in reality:
Video: NVIDIA Queen demonstration, source: NVIDIA
Our QUEEN method is capable of capturing dynamic scenes with high visual quality and reducing the model size to just 0.7MB per frame, training in less than 5 seconds and rendering at ~350 FPS.
The QUEEN model will be presented at the NeurIPS 2024 conference, which begins on 10 December in Vancouver, Canada. The code for the model is expected to be published with open access on the project page. This development is one of more than 50 NVIDIA works that will be presented at NeurIPS 2024, demonstrating the company's achievements in the field of artificial intelligence with potential applications in simulation, robotics, and healthcare.
Video: NVIDIA Queen demonstration, source: NVIDIA
Why It Matters
The Queen model will help you create very cool 3D images and videos in real time that can be seen from different angles. Imagine watching your favourite team's match and being able to choose which angle to watch a goal from! Or, during a lecture, instead of the usual illustrations, a real 3D animation appears on the board to explain how something works. This will make learning and entertainment even more interesting.
