Apple traditionally holds its WWDC (Worldwide Developers Conference) every June to announce and explain the year’s OS updates. This year was no exception. While it wasn’t flashy as there were no new hardware products, in a way, it might have been the most important event in the last decade.
The reason is that Apple Intelligence, its AI function, has been relaunched. This AI will become the foundation for all functions in Apple products going forward.
And this is also true for “visionOS,” the operating system for Apple Vision Pro.
Furthermore, some of the features incorporated into other devices are originally influenced by visionOS.
So, how will it actually change? Let’s explain based on on-site coverage and the performance of the beta OS on actual devices.
visionOS 27 Beta 1
Please note that the screenshots shown in this article are from a development version and may differ from the final released version. Installing a beta version is not recommended unless you are using it for development purposes.
■Enhanced AI, recognizing what the user is looking at.
First, I’d like to talk about AI.
In 2024, Apple announced its AI feature, “Intelligence Apple,” and planned to incorporate it into its products, including iPhones. However, the plan did not go as expected. The AI model lacked sufficient capabilities and was unable to achieve the intended functionality.
As a result, Apple revised its AI development plan and, with the cooperation of Google, decided to “distill Gemini’s frontier model to create a smaller-scale, proprietary AI” and “use servers jointly developed by Google and NVIDIA.” This is the “third-generation Intelligence Apple” that was announced this time.
I won’t go into the details of the features as it’s not the main point, but I feel that Apple has successfully achieved the features it aimed for in 2024, and that it has become a pretty good AI with fast operating speeds.
VisionOS also features third-generation Apple Intelligence and its AI assistant, “Siri AI.” However, it differs considerably from the versions available on other devices.
Firstly, the visuals are quite elaborate.
The following images were captured using visionOS 27 Beta 1. The round sphere in the center is Siri AI. Speaking to it will initiate the process.
A round sphere is Siri AI
The sphere is positioned in space as a self-illuminating object, and when it aligns with the actual landscape, the light falls onto the real scene.
When superimposed on a real-world scene using a video see-through, the effect of the light source is superimposed on the real-world scene.
“The process progresses when you talk to the sphere” is exactly what it says. While it naturally recognizes speech, when it’s not recognizing speech, simply looking at the sphere switches it into speech recognition mode. In other words, it’s a UI that leverages eye-tracking.
From Apple’s demo video. Staring at the sphere in the center triggers voice recognition.
Apple Intelligence has a feature called “Visual Intelligence” that recognizes what is depicted in an image. This, too, is actually linked to eye-tracking recognition.
In short, everything that is in the line of sight of what is displayed inside Vision Pro is recognized along with the command it receives.
During the WWDC demo, he was looking at a website selling backpacks and asking questions like, “Can all my belongings fit in this?”
From Apple’s demo: Recognizing content while browsing the web and using it to interact with AI.
In addition to this, it can also recognize what is in its line of sight and respond to commands, even when capturing the external world with a camera (i.e., the scenery in front of it).
As a result, it becomes possible to achieve collaboration between the external world and AI without having to prepare a special recognition application.
■Custom “environments” can also be set up.
Another easily noticeable change is the strengthening of the “environment.”
An environment is a landscape used in Vision Pro as a substitute for real-world scenery via video see-through. Up until now, quite elaborate environments have been added, ranging from deserts to the moon and Jupiter. This time, one more will be added, but that’s not all.
The newly added “Thórsmörk”
You can take your own panoramic photos, turn them into 3D models, and place them as “environments.” The images below show a Parisian landscape I photographed, and a panoramic view of Apple headquarters taken from the rooftop of the Visitor Center next to Apple’s headquarters, which I then turned into an “environment.”
Set a panoramic view taken in Paris as the “environment”.
Similarly, I’ve set the view from the Visitor Center next to Apple headquarters as my “Environment.”
VisionOS has a function that converts 2D photos into 3D (spatial photos). Strictly speaking, this is a different function from the “spatial photo” function included in the “Photos” app on iOS and other platforms. By applying this function, panoramic photos are first transformed into three-dimensional spatial photos, and then these are placed, so that when you move your body, the scenery appears to be in a somewhat 3D space.
Apple’s official “Environment” feature covers the entire 360 degrees, but using your own photos basically covers 180 degrees. The official feature includes ambient sounds, but naturally, there are no sounds when using your own photos.
■Numerous UI improvements
There’s also a basic UI overhaul.
First, the Control Center display has changed. Basically, you can think of it as a “reconfiguration.”
Control The Center. The layout has changed quite a bit.
Next, there’s the performance improvement. While visionOS itself isn’t a particularly slow OS, the overall performance has become much smoother. This is noticeable even in the beta version.
In terms of performance, the biggest improvement is that Wi-Fi connection speeds are up to three times faster. Vision Pro, in particular, had the impression of being slower to recognize Wi-Fi compared to other Apple products, but this has been dramatically improved.
The notification system has also changed significantly. While the notifications themselves remain the same, eye-tracking recognition has been introduced, allowing you to open notifications simply by looking at them. You can then open them and reply. Since you no longer need to move your hands, this is incredibly convenient.
Furthermore, the Safari web browser now allows users to display up to five websites side-by-side simultaneously. When displayed this way, the entire page is curved. Even when displaying a single website, if the size exceeds a certain limit, it automatically curves. The curvature cannot be changed.
Safari now allows multiple screens to be displayed side-by-side in a curved view.
Similarly, the TV app also apparently supports curved display, but this is only applied when using multi-angle display for sports and other content.
The fact that screenshots and screen recordings now support “high quality” is also a big plus.
Screenshots and other images now support “high-resolution recording”.
Previously, Vision Pro alone applied Foveated Rendering, resulting in reduced resolution recordings outside the center. To avoid this, it was necessary to install a development environment on a Mac, connect to Vision Pro, and capture from within the development environment.
However, visionOS 27 includes a “high-quality” mode that allows you to capture the screen at full resolution without connecting to a Mac. Videos are limited to a maximum of 3 minutes, but this should not be a practical problem. All the images and videos of visionOS 27 Beta 1 used in this article were also recorded in high-quality mode.
Screenshot video taken in high-quality mode. Unlike before, it no longer blurs outside the center. However, the maximum length is 3 minutes.
■ Improvements for B2B customers are also noticeable.
This update also includes practical features for B2B use.
The Foveated Streaming Framework has been updated, improving streaming quality from PCs. This update is particularly intended for integration with NVIDIA solutions used in areas such as physical AI, robotics, factory simulations, and automotive design. In fact, there is a high demand for Vision Pro in these areas, making this a practical update.
The “Spatial Preview” feature now allows you to pull content from compatible apps on your Mac into Vision Pro and experience it in 3D. This feature would be particularly useful for situations where you need to collaboratively check objects during a meeting.
Furthermore, information for developing “spatial accessories” intended for use with specific applications has also been released. This is more suited to developing specialized devices required for business purposes rather than creating peripherals for operating Vision Pro.
Reality Composer Pro, used for developing spatial applications and content, has also been updated to Reality Composer Pro 3, with significantly improved functionality.
It has become a standalone app and now incorporates game engine-like elements. It has shifted from being primarily a tool for creating 3D assets to a tool for creating simple spatial applications. “Reality Composer Pro 3” can be downloaded from the following link.
https://developer.apple.com/jp/reality-composer-pro/
■Fragments of “spatial computing technology” visible in other operating systems
Looking at visionOS 27, there’s absolutely no indication that Apple is stopping its investment in visionOS. Apple representatives all seemed to agree that “spatial computing is in its early stages.” Of course, that might be the rule they have to follow, but I got the impression that they genuinely believe that.
Features like eye-tracking, AI, and spatial photography will likely become part of Apple’s overall value proposition.
iOS 27 introduced a feature called “Reframe,” which allows users to change the angle of a photograph. This feature converts a photo into a 3D spatial image, then changes the angle, and uses AI to generate a “photo from a different angle.” This is also made possible by 2D-to-3D conversion technology and the development of technology for visionOS.
An example of processing using iOS 27’s “Reframe” feature. It generates a photo with a changed field of view from the original image.
Apple Maps’ 3D display, “Flyover,” now uses Gaussian splatter for major cities. This has resulted in increased realism, which is due to improvements in how Gaussian splatter is handled, including in visionOS.
Maps of Tokyo are also rendered in detail using Gaussian splatting for a 3D display.
It’s worth paying attention to how Apple is utilizing spatial-related technologies across the entire company.