The release of the additional technical specifications of the Apple Vision Pro in January 2024 has given birth to a novel buzzword. It is called photon-to-photon latency. Apple has claimed that the dedicated Apple R1 coprocessor used in its mixed reality and spatial computing headset features a 12-millisecond photon-to-photon latency. This has been received with awe. Several enthusiasts have commended Apple while other observers were either skeptical or unmoved. What exactly is photon-to-photon latency? Is it a mere marketing buzzword or a legitimate technical term? How is it relevant to XR devices such as the Apple Vision Pro?
Explaining Photon-to-Photon Latency
Definition and Relevance in Computer-Altered Reality or XR Devices
This term is not a mere buzzword. It is a specifically a metric. Apple was also not the first to use photon-to-photon latency within the domain of computer-altered reality or XR technologies such as VR and MR headsets and AR features of smartphones. A paper from researchers at Microsoft and the University College London used the term in relation to the measurement of the latency of digital pass-through XR systems such as a VR headset or an AR camera.
Latency represents the time it takes for something to happen before it is initiated. Nevertheless, in computer terminologies, it represents the time it takes for data to travel between the point of origin and to point of destination. A pass-through or passthrough describes the technology used to capture real-world images using cameras and project them on the screens of XR devices like virtual reality headsets in real time. It is a defining feature of XR systems.
Hence, based on the aforementioned, the quality or latency of a digital pass-through is described in numerical terms or values and is called photon-to-photon latency. Furthermore, to understand better what photon-to-photon latency is, it can be described as the time it takes for the device to transmit captured images from the cameras and display it on the screen in real-time. This is still an oversimplification since advanced devices have higher-quality digital pass-through.
It is important to note that the pass-through technologies used in XR devices such as the Apple Vision Pro do not just capture light using their built-in camera sensors and project them on the screens as images or videos. Their hardware needs to calculate how to superimpose VR or AR elements or mix both VR and AR elements together with real-world images in real-time while taking into consideration lighting and the position or orientation of the user.
Difference With Motion-to-Photon Latency and Screen Refresh Rate
Some commenters have confused photon-to-photon latency with motion-to-photon latency while others have opined that the former is a mere marketing buzzword used to describe screen refresh rate. These confusions are understandable to a certain extent. The technologies involved in the implementation of mixed reality and spatial computing are still new. Some of these technologies also overlap with existing technologies used in existing consumer electronics.
It is still important to underscore the difference between photon-to-photon latency and motion-to-photon latency and screen refresh rate. Hence, by definition, motion-to-photon latency is the time it takes for user or device movements to take into account the corresponding changes in the display output. Most XR devices are motion-dependent. The refresh rate of display, on the other hand, is the number of times per second a display updates the image on the screen.
Take note that latencies are measured in milliseconds while refresh rate is measured in hertz. All three are important factors that define the quality or set of features of advanced XR devices such as the Apple Vision Pro. Nevertheless, to reiterate the definition of photon-to-photon latency as mentioned above, remember that it describes the time it takes for the camera of an XR device to capture images from the real world and project them on the display screen in real time.
It is also important to note of the fact that XR devices do more than just capturing images and displaying them on a display output. These devices alter the real-world environment through the superimposition or integration of digital objects. A specific mixed reality headset also combines VR and AR objects together with the real-world environment. This entails a lot of computations of data inputs from the camera sensors and even artificial intelligence algorithms.
FURTHER READINGS AND REFERENCES
- Choi, S.-W., Lee, S., Seo, M.-W., and Kang, S.-J. 2018. “Time Sequential Motion-to-Photon Latency Measurement System for Virtual Reality Head-Mounted Displays.” Electronics. 7(9): 171. DOI: 3390/electronics7090171
- Gruen, R., Ofek, E., Steed, A., Gal, R., Sinclair, M., and Gonzalez-Franco, M. 2020. “Measuring System Visual Latency Through Cognitive Latency on Video See-Through AR Devices.” 2020 IEEE Conference on Virtual Reality and 3D User Interfaces. 791-799. DOI: 1109/VR46266.2020.00103
- Warburton, M., Mon-Williams, M., Mushtaq, F., and Morehead, J. R. 2022. “Measuring Motion-to-Photon Latency for Sensorimotor Experiments with Virtual Reality Systems. Behavior Research Methods. 55(7): 3658-3678. DOI: 3758/s13428-022-01983-5
- Ubik, S. and Pospisilik, J. 2021. “Video Camera Latency Analysis and Measurement.” IEEE Transactions on Circuits and Systems for Video Technology. 31(1): DOI: 1109/tcsvt.2020.2978057
