The power of AR/VR   

The power of AR/VR   

Being a consultant within TNO for about two and a half years, Tom de Koninck has a vast telecommunications background and an impressive curriculum. He is currently involved in several XR projects, besides being a spokesperson for his XR team. He explains the different applications for Augmented and Virtual Reality, gives some insights on the future of entertainment and social interactions, and he explains what this somewhat new buzzword, XR, stands for.   

TNO has been working heavily with AR and VR for a couple of years now. According to Tom de Koninck, TNO also has a team entirely focused on 5G technology research and development. Furthermore, as Tom highlights, the company holds their own 5G network and 5G license, “not in order to host customers, but to be able to do end-to-end proof of concepts, tests, and experiments, so that we indeed know how it works.” He clarifies that 5G, as we know it today, has not reached its full potential yet, being a technology still under development on what concerns, for example, its standardization. A lot of advancement in this area is predicted over the upcoming years.  

AR or VR? We are talking XR  

According to Tom, the preference for employing Augmented Reality or Virtual Reality lies in how much the real world matters for the experience. When the reality is not relevant for the interaction, and only the virtual world is to be explored, then Virtual Reality is preferred. “In our virtual conversation, my real world is not important. I don’t need anything around me to be able to speak to you. So I could do this in VR. We could even have the experience of feeling that we are both in the same (virtual) environment, the same room. Therefore, for an interaction like this, virtual reality would be on point,” he describes.  

On the other hand, consider a remote maintenance use case: at a given production plant, an employee is investigating a defective machine. However, he is not an expert on that particular problem, and the person who is happens to be in a remote location. In this case, as Tom explains, the real world is essential, because there is a real machine, and its image must be captured. “Then, Augmented Reality is much more important because it enables you to have the necessary grasp on the real world in such a situation,” he says.   

Nevertheless, Tom emphasizes that the distinction tends to be unimportant at this point in technology. As such, TNO opts for the term Extended Reality, or XR, a relatively new buzzword in the field, compared to AR and VR. As he details: “The technology we are developing goes more and more towards hybrid.” That is what XR stands for: a term that encompasses all the Realities – Augmented, Virtual, and Mixed – under the same “umbrella.” We can imagine a scenario where VR and AR are combined, having one user completely immersed in VR, and another person in AR. Tom exemplifies going back to the use case of remote maintenance: “The remote person could be in VR, having the full view of the production plant in his VR headset; while the maintenance engineer onsite would just have AR glasses with a body camera capturing the world. Of course, he sees the real world, but he needs to have some indications given to him by his colleague remotely. For instance, ‘turn that knob, go there,’ and so on.” For him, this kind of interaction will be dependable in the coming years, when the exchange of all those forms will be something ordinary. “These are not siloed technologies anymore,” Tom states.  

XR in different segments  

One of TNO’s projects in the entertainment sector is called VR Together. The aim of this social VR project is that multiple people are transported into a virtual environment and can enjoy scripted content about a murder case. Tom explains that users “watch some scripted content, almost like a small movie, but they can also see and talk to each other.” He says that in this type of environment, it is crucial that participants are seen accurately, as they really are, for the purpose of the experience.    

Another example is a project created for a Dutch healthcare provider concerning the wellbeing of the elderly population. Due to the Covid-19 pandemic, many of these people have been isolated, either in their own houses or in care homes, without the possibility of receiving any visits from their loved ones. There are, of course, alternatives, like FaceTime, Teams, Zoom, and so many other platforms for social video calls and remote interaction. However, TNO went further and developed an application that mimics the visit. As Tom explains, “with AR, we can make it as if the person visiting that elderly family member was really in the room with them. What we’re using today is still a statically-placed tablet. In the future, that can also be achieved with AR glasses. That is clearly a social activity for bringing people together again.”

As an institute based on research, creating proof of concepts and demonstrators, TNO builds the technology that other parties will turn into products and commercialize. “We never manufacture a product. Between the technology and a product, there are many operational elements to be built around it, and this is not our focus. We always work with external partners to develop the technology towards a real product that can be used by the public,” Tom points out. Part of the projects mentioned above, for example, are indeed already in development by other companies and TNO’s clients. 

Tom de Koninck

Much more than faster 4G  

With the popularization of 5G, the first and obvious idea that comes to mind is that there will be more bandwidth available. However, the differentiation between downstream and upstream bandwidth must be made here. According to Tom, 4G already offers a considerable downstream bandwidth: “most of the time, you don’t have issues watching a high-quality video on your 4G mobile phone,” he says. With 5G, on the other hand, you can have even more downstream bandwidth, yes, but there is also the more robust upstream bandwidth needed for AR and VR. “We’re talking about social VR where you, as a person, have to be transported, immersed in a virtual reality. Then, the upstream bandwidth is more important,” he highlights.   

Currently, mobile phones, like the iPhone 12, have a LiDAR, a depth camera that can capture the user as a 3D image. That image is much more complex than just a photo or a video, though. Tom says that much more upstream bandwidth is needed to upload that image. Furthermore, and especially in the case of gaming, for example, latency is vital. “When the players are immersed in the virtual environment, as they move their hands, the response needs to happen immediately. 5G is offering that improved latency, too,” he exemplifies.  

In the case of Business Services, Tom mentions the importance of prioritization and the possibilities it opens when it really matters: “When you need to provide a service that works nonstop, then you can prioritize your application on the network, which is also relevant.” For him, it is clear that 5G can offer more than just added downstream bandwidth: “otherwise it wouldn’t be needed,” he says, and adds: “It would be too little just to make sure that 4G has yet another upgrade and nothing else.”  

XR and 5G: a formidable alliance  

Tom perceives a good match between XR and 5G because of how these networks are built or can be built, moving computing and storage operations to the edge and bringing many capabilities to the next level. As he emphasizes, 5G’s unique features, such as its lower latency and higher bandwidth, are needed to achieve widespread XR services. Moreover, there is an intention to provide for consumers low-cost devices that, nevertheless, hand over high-quality images and videos. 

Tom describes the three main pillars for defining and managing fixed core networks as media with virtual reality, 5G, and cloudification software-defined networking. For him, “bringing together research on XR, 5G itself, and virtualization of the network produces a perfect blend.” All of these technologies are at our disposal, he says, so the future of networking and applications built around 5G are very beneficial for the media industry but also remarkably favorable for IoT for business services. 

To learn more on how TNO is using 5G to reshape the XR landscape, check their Smart Society pages on their website.