At this year's WWDC, Apple unveiled an exciting new feature for its Apple Vision Pro VR/AR headsets, enabling users to craft personalized environments using their panoramic photos. This enhancement, set to launch this fall, promises to elevate the immersive experience of spatial photos.
The Vision Pro's environmental capabilities are one of its most captivating aspects. Users can seamlessly switch between immersive 3D backdrops, akin to adjusting a dimmer switch, allowing them to blend the real world with virtual elements according to their preferences. Additionally, users can engage with others through their unique Vision avatars in shared virtual spaces, enhancing social interactions within these user-generated environments.
However, these environments extend beyond mere photographic backdrops. They are comprehensive 3D scenes that incorporate sound, creating a fully immersive experience. While Apple has not disclosed extensive details, the transformation of panoramic images into these environments is likely linked to improvements in their spatial photography technology. Currently, the Vision Pro allows for the creation of 3D images, though these are limited in depth and can only be viewed from the angle they were captured. The gap between these basic 3D photos and fully immersive environments--which provide a 360-degree view of the surroundings--remains significant. Even panoramic photos captured on iOS devices often leave gaps that need to be filled, particularly in the sky and ground areas.
Spatial Reframing: A Leap Towards Realistic 3D
Apple's forthcoming "spatial reframing" technology hints at how the company intends to generate more immersive 3D experiences from flat images. During the presentation, Apple highlighted how this feature could enhance snapshots, while also suggesting that generative AI might be employed to fill in the missing elements of photos. This innovative approach could pave the way for richer 3D images and comprehensive 360-degree environments.
As someone who has enjoyed the instant 3D photography capabilities of the Vision Pro, I am eager to explore the possibilities of Gaussian Splats--scans that allow for true 3D experiences of familiar places and people. However, I have reservations about the role of AI in enhancing these memories. While the technology is undoubtedly impressive, the authenticity of Gaussian Splats lies in their faithful recreation of captured scenes. The idea of AI altering personal memories feels unsettling, yet I remain enthusiastic to experience this new feature firsthand.
