Simulated weapons training for new army recruits, virtual tours of planned community developments, and training the next generation of bio-hazard diagnosticians and controllers – these are just a few of the ways governments are harnessing the massive potential of immersive technologies.
Governments across Australia are turning their spotlights on to extended reality (XR) technologies, such as virtual reality (VR), augmented reality (AR), and mixed reality (MR).
The foray appears timely. Meta (formerly Facebook), in September, announced a $67 million investment in its ‘Metaverse’, a virtual space for diverse creation and collaboration. It promptly upped its spend to $13.3 billion in metaverse technology a month later.
Meta owns Oculus, a popular VR headset brand, available in consumer and enterprise models. Facebook and Instagram will be linked to the new Oculus enterprise headset, Quest for Business. Apple is expected to release its own AR, VR and mixed reality headsets next year.
The pros and cons
A meta-analysis of peer-reviewed research on VR training and education vouches for its potential as a tool for safe training. It found that, on average, using virtual technologies was “more effective than traditional training in developing technical, practical, and socio-emotional skills”.
So, what exactly is VR? According to Wired.com, VR uses computer-aided stimuli to create the illusion of being somewhere else, in short an immerse experience. Donning a visual and audio headset allows wearers to simulate the real world without being exposed to dangerous situations – think firefighting, chemical handling, or hostage crises. Staff can train solo or in multiplayer mode. VR can use technology to stimulate the senses and be kinaesthetic – that is, good for tactile learners.
AR can be accessed via an electronic device or AR glasses. Users harness the technology to enhance, interact with and view a real-world environment. It is child-friendly and is predicted to be as ubiquitous as the smartphone. But detractors fear health and safety, privacy, anonymity, user manipulation and digital ownership implications. They also fret that the costs and challenges of adopting the technology could deepen the digital divide. Mixed reality, meanwhile, is simply a blend of VR and AR.
Futurist Matt Coleman, who organises Magnify World summits that showcase emerging innovation across the globe, says immersive technology has gained prominence across training sectors.
“Whether it’s a Telstra technician, banking staff, the military, or tourism workers, these people can be easily trained with VR or AR headsets at home. That’s particularly useful if Covid-19 rules mean you can’t train in the same facility.”
Coleman says the US Army is a major investor in augmented reality. Earlier this year, it signed a 10-year, US$21.9 billion deal with Microsoft for HoloLens headsets. The technology is customised with thermal sensing and night-vision for training and battlefield use.
Use case: Australia’s defence force
A spokesperson for the Australian Department of Defence says governments choosing not to adopt immersive tech now would be akin to “choosing not to adopt email in the 1990s”.
“Using VR almost guarantees a smile from the user, whereas demonstrations are usually dry and intellectual”.
The Australian Army uses immersive technology widely for highly realistic training, including AR-enabled night vision, but it has yet to fully adopt it for operations. The Defence Innovation Hub is working with Deakin University on a $5.1 million project to build a VR/AR trainer to help navy personnel train for maritime firefighting.
The spokesperson says the Army has used the technology to “enhance situational awareness, lethality and command and control”. It is also considering mixed reality for personnel undertaking technical maintenance tasks, medical care and navigation, or coordinating military assets, including robots and autonomous systems.
However, the spokesperson hastened to add, “it can’t be assumed that simply giving an operator a set of AR or VR goggles will lead to operational improvement”.
“A poorly designed interface can hinder operations or even lead to safety concerns.
“Care must be taken to develop applications that take maximum advantage of the technology and not have it get in the way”.
As technology companies improve the software and hardware within these systems, the Army can then “focus on defence applications and let the commercial world do the heavy lifting”.
Use case: A virtual municipality
Moreland City Council in Melbourne, a pioneer in public sector XR, launched Virtual Moreland in June 2019.
The multi-award-winning approach uses three-dimensional mapping and modelling, VR, AR and 3D Geographical Information System (GIS), including digital twins, to help inform planning and design decisions and boost community engagement in the process.
The $837,000 joint project saw the council partner with the University of Melbourne, technical design consultancy Lost in The Machine, immersive technology company KASA, as well as mapping and GIS experts AAM.
Chris Staring, with the unique title of Virtual Moreland Officer (part of the Moreland City Council in Melbourne), says VR works better than other media or technology to convey an approximation of a proposed development, especially its scale.
Most users notice additional details while immersed in VR designs – details that can be easily missed on 3D renders and 2D plans alone, he says.
“Previously, the community would usually be presented with still 3D renders or artist impressions and a set of traditional – sometimes difficult-to-understand – 2D plans and left to themselves to interpret how the space looks and feels,” Staring says.
“VR is able to remove that subjective barrier by putting the user inside the design at 1:1 scale, presenting the design to all users the same way; there are far fewer assumptions.
“It’s an improvement in understanding of the design and gives better feedback overall.”
Staring says the council has experimented with community members designing their own spaces inside VR, using pre-defined 3D models of outdoor furniture for parks.
During the Covid-19 lockdowns, the technology also enabled council officers and their stakeholders to inspect sites when regulations prevented physical visits.
Staring manages Virtual Moreland 3D’s digital twin and other 3D datasets to ensure it is accurate and well maintained. He says that while the council cannot mandate that external property developers offer 3D models of their plans, Moreland volunteers its 3D data to assist them with their preparing their applications and developing those models.
A bonus is that the 3D data allows the council to identify developers’ inaccurate representations of developments, Staring says. That is where those developers have used their own data and surveys, which can sometimes have inaccuracies.
“There are quite often mistakes in the surrounding 3D building context and vegetation, including buildings which are sometimes smaller or larger than reality, have inaccurate roof forms, or positioned incorrectly.”
Those inaccuracies make the development appear larger, smaller, more obscured or less so from certain viewpoints than it would be in reality. VR can do a far better job of exposing these errors.
As well, VR helps the general public get a clearer sense of the development, he says. (This link offers more details on the outcomes, benefits and lessons learnt since the project went live).
Other possibilities include the use of AR to create a fourth dimension (that is, ‘temporality’), which could be used to do things such as immortalise historic buildings and places of significance that have been lost or demolished. The technology could be accessed via QR codes along heritage trails, for example.
On the theme of experience, the State Library Victoria is also set to harness immersive tech to build sustainable visitor experiences, such as a literary escape room, AR games, and multi-sensory dining events.
Director Experience State Library Victoria Harlinah Teoh says 14 successful external applicants each received $20,000 to develop the concepts. They will pitch their work by mid-December, with the library set to trial some of them early next year.
Use case: animal disease investigations
Veterinary officer within the Federal Government’s Department of Agriculture, Water and the Environment, Corrie Croton, went to Nepal in 2019 to do real-time training in diagnosing foot-and-mouth disease (FMD); however, Covid-19 put a spanner in the works.
His director suggested training could happen virtually, so Croton took up the opportunity to explore how the Department could supplement their training with VR, one of his hobbies.
Croton has managed a pilot program to investigate VR training for on-farm FMD diagnosis. (The agriculture department’s Biosecurity Operations Division is also working on separate projects in immersive tech). It is a vital endeavour to protect Australia’s agricultural sector: a wide-scale outbreak of FMD, a highly contagious viral infection of cloven-hoofed animals, could cost Australia $51.8 billion in lost revenue over a decade, a government report has estimated.
For the VR pilot, the Department contracted Adelaide VR games studio Novus Res to create five farm scenarios, including a cow examination and taking laboratory samples. Thirty departmental staff took part. Croton says their feedback was positive, with more ‘hands-on’ virtual experiences being demanded from staff.
“We felt it was a step forward using the [Oculus Quest] headset, because you can be completely untethered from a PC. It has full tracking, so you can move your hands in the environment.
“All the stakeholders are very keen, and we have a [departmental] commitment this financial year to do the next phase of this project.”
The technology’s tracking, frame and refresh rates have improved since the pilot program. It has become cheaper and allows multi-play, which could see training occur in a ‘metaverse’ environment.
Croton says the long-term goal was for VR training to become part of a veterinarians’ (and other animal industry workers’) educational experience, to improve trainee’s skills in examining animals for signs of exotic diseases.
The pitfalls of immersive tech
Digital reality technology has its detractors, and its problems. It is not ideal for people with epilepsy, those prone to motion sickness or vertigo and some people might be uncomfortable navigating the world through a headset. Fatigue, head and neck pain have been reported with prolonged use.
Sanitising shared headsets, particularly with the ongoing risk of coronavirus, can also prove resource-heavy. Deloitte has also raised the issue of costs, which can be prohibitive. Indeed, headsets represent just one-sixth of the technology’s cost to enterprises, with software development, services and network integration also needed on top.
Dutch IT researchers Dhoya Snijders and Rinie van Est warn that immersive technology should be regarded as “surveillance” technology – one that constantly monitors users and their living environment.
They say the technology needs to be fed rich data about users, including biometric data, raising “questions about privacy, anonymity and digital ownership”.
The researchers also note the technology can be used to manipulate users and warn that conflicts can morph from the virtual world to the real world, as seen with AR app Pokémon GO. Issues include people being transfixed with the game as they walked, as well as riding their bikes or driving their cars while in-game, resulting in injuries. In 2019, the developer settled a class action about players trespassing on private properties, resulting in the removal of ‘Pokéstops’, essentially checkpoints, near single-family homes and pledge not to add new ones.
Their suggested ‘rules’ for governments and others include guaranteeing VR/AR users’ and non-users’ privacy, creating legal definitions for digital property, protecting users’ mental and physical health and against being manipulated.
Another consideration is the digital divide. Not all constituents are online, have smartphones, or the skills or cognitive capacity to navigate a virtual world. Internet download speeds are patchy, particularly in remote areas.
There might be resistance or slow uptake from government staff and, chances are that qualified, immersive technology trainers are in short supply, unless the ‘Metaverse’ can remedy that in the medium term.
Immersive tech is definitely a ‘watch this space’ territory for governments.
