The Future of Training: VR and Conversational AI
How Virtual Reality and AI-Powered Conversations are Shaping Tomorrow's Workforce Training
We’ve probably all imagined the future of training technology: doctors performing dangerous surgeries before ever scrubbing up, technicians repairing complex machinery in potentially dangerous locations from the safety of an office, business or political leader engaging in high-stress negotiations ahead of time, all as virtual simulations.
For years I've seen demos of things like this, using VR, or augmented reality (AR) technology, in the form of overlaid schematics, or assistive maps, provided via smart glasses or goggles.
It’s been a wonderful dream, but never practical.
Until now. Increasingly, with next-gen AI training technologies, I see this very form of assistive technology becoming reality. I believe we can now look forward to a time when AI-driven virtual training will be both widespread and affordable, used for everything from onboarding off-site colleagues to cybersecurity training to putting employees directly into specific workplace scenarios.
GenAI has come about on chips made for the 3D environments of video games, and the technology has been developed, augmented, and continues to be trained using them. As long ago as 2013, video games like Super Mario Brothers were being used as a benchmark for testing AIs.
In the near future, we will see this come full circle, in the form of hands-on learning with VR and AI that put people into such make-believe realities, but for the purpose of pragmatic, real-world training.
AI for Training
Already we’ve seen great strides with AI in training. With the emergence of copilots, real-time support is already available for workers in a variety of fields. From drafting initial passes to completing code snippets to reviewing and providing suggestions, AI copilots in training are aiming to improve both efficiency and consistency.
In software development, it’s been a mixed bag, admittedly, with Github publishing a survey that shows some 97% of developers are already using AI coding assistants, but also showing with an increase in bugs (Uplevel estimates 41% more on average) that accompany this faster pace. This kind of assistive work is less about training and more about completion, and while new solutions continue to be developed rapidly, they have some ways to go.
In customer service we see faster gains in training itself, where there is more repetition. In addition to being able to place and field calls itself, AI systems are also monitoring live calls with human agents, analyzing and providing real-time feedback based on message, consistency, factuality, and pacing. With the capacity to be seemingly everywhere at once, these AI trainers give callers personalized training on the fly at a level that was not previously possible.
Digital twins—or the virtual replica of a real-world thing augmented by a flow of updated data—already allow workers to practice in virtual spaces prior to ever setting foot in the real place. This not only protects workers from harm, but also prolongs the life of equipment and improves the end product, ensuring safer, more efficient environments as new employees are onboarded.
Digital twins allow cybersecurity teams to visualize scenarios and reactions, setting up virtual catastrophes to practice implementing response plans at a level never before possible. Taking Chaos Engineering to the next level, these digital twins enable playing out worst-case scenarios in the safety of the virtual.
In medicine, digital twins of organs and patients are already being used to enable students to practice in a scenario much as the one we opened with. Diagnosis accuracies are being raised with increased specificity—using virtual versions made with the actual data.
And real-time translation, one of the early implementations of AI technologies alongside improved compression, help collaboration as well as training, with teams gaining familiarity even as they are not hindered by barrier—able to understand each other in real time.
We know AI can thrive at personalization, and this is already being put in practice on educational platforms, where plans adjust to a given learner’s progress, already boosting retention rates.
VR and AR: Immersive Training
Ever since the concept was first created decades ago, there have been attempts to harness VR for employee training.
And with heavy construction equipment and in dangerous scenarios, such as for firefighters and NASA astronauts, VR has successfully been implemented, with varying rates of success.
AR overlays, when coupled with digital twins, for example, can bring digital insights into the real world, providing immediate assistance, though widespread, practical development of these has lagged far behind tantalizing prototypes.
Bosch is one example of a company that’s made strides here, implementing tiny lasers on smart glasses to paint data on the back of the eye (as demonstrated at CES in 2020), even before the breakthrough in mainstream GenAI.
And John Hopkins neurosurgeons first performed AR surgery in living patients around the same time: using a headset with a clear display functioning like a GPS system. This followed on years of using AR for training medical students.
Bringing it Together
Combining these two developments promises to make truly immersive training with AI and VR together. Factoring in cloud technology as the third component can make the prohibitively expensive practical, delivering it from high-resource data centers more easily around the world to wherever it is needed.
With AI algorithms analyzing a trainee’s performance, virtual reality in education becomes far more pragmatic, reducing the need to manually creative extensive environments and scenarios in a one-size-fits-all fashion that’s both more expensive and less effective. If an employee struggles to get up to speed (or inversely thrives), the system can modify the scenario and avoid wasting precious time, only giving more practice scenarios, at each difficulty, as required, and even fabricating new ones to keep the learning fresh.
Feedback is critical to effective learning, and AI-powered training solutions are already capable of this. Coupled with audio/visual capacity to place a learner into a scenario and react accordingly, training becomes far more immersive.
As technology continues to advance in arenas like haptics, VR experiences will also provide sensory feedback in additional ways, to further enhance realism and improve learning and retention.
Conversational AI is a game-changer for the use of AI in training applications across the board. Allowing open-ended dialogue, it converts language to and from the necessary structures, flowing far more naturally as a human being converses.
Companies like MindBank are transforming the creation of virtual avatars of people—video/audio likenesses which can present from given scripts or play out in set scenarios—into full-fledged digital twins of people by adding in actual knowledge. From communications to speeches, to publications, it’s easy to imagine a scenario where real-time audio and video capture continues to feed the model, enabling ever more realistic outcomes from a digital twin.
How does this impact training? Imagine having a digital twin of the foremost expert in any field able to interact directly, personally, one-on-one with any pupil in real-time. Extended from here, this use of AI in corporate training could allow virtual mentorships from anywhere, at any time, in the same fashion.
The Future of Workforce Development
Increasingly, we see companies around the world recognizing the value of skills-based hiring. With some 80% of current engineering skills likely to be outdated in just a few years (many skills now having an expected shelf-life of just 2.5 years), work in the future must also include learning, on a weekly if not daily basis.
Harnessing conversational AI and VR and AR solutions in training, workers could begin from their first days working with virtual experts, experience hands-on situations, and learn by doing, right out of the gate.
Upskilling and reskilling can be integrated on a daily basis, before or after breaks, for example, with programs that track and customize to user progress, and adjust to teach what they most need to know, based on workplace performance and the goals of the organization itself.
Conclusion
I’ve certainly speculated here to some degree, but far more fascinating are the things we’re already seeing done: AI systems that call, discuss, schedule, observe, and model, giving direct, real-time feedback to those doing all manner of jobs today, from customer service to open-heart surgery.
VR and AR technologies, like the Metaverse, were embroiled in hype so far have offered far more promise than actual delivery. But with AI’s ability to crunch large amounts of data fast, to handle conversion and craft conversational responses of its own, to devise real-world scenarios that adjust to user actions on the fly, we will see these things finally converge to come to fruition in the training arena.
By experimenting with AI-driven training and automation solutions now, companies can implement the technology in a safe, sandboxed fashion as it continues to grow out, improving the skills and experience of their workers at the same time.