A number of years ago my wife and I had a chance to try out a friends augmented reality, virtual reality (AR/VR) system. Deborah proved to be graceful in the artificial world(1). On the other hand, I had a dramatic fall when “running” down a virtual mountain(2). In hindsight this is an example of a problem arising from open loop system.
Closing the virtual loop
Feedback is not enough, the feedback needs to be synced to the environment. This means that models of the person and the physical world are accurate to a degree that it fools our highly tuned sensor. As a starting point, \games have given us accurate physics models of the real world (3), however they fall short as they do not close the loop on the person in question.
A bespoke AR/VR suite
How do we seriously Taylor(4) the AR/VR to the individual? This is where an adaptive Deep Learning system can come into play. Giving the persons “overshoot” as inputs the system can learn to provide the correct feedback for any situation, “increasing” the force needed to lift an object, or by making the ground come up at a proper rate.
Avoiding “God Mode”
Video games often have a concept of “God Mode”. In this mode the player has unlimited powers, can’t be hurt, can run 1,000 km/hour. This is why an observer is needed for the Deep Learning system, to prevent feedbacks going in the wrong direction. Here is where traditional “bounded” values can be observed for any and all objects in the virtual world; e.g. the “force” to lift a 1kg object will always fall between X and Y, with the final value tuned for each user.
Learn the guitar!
As a child learning to play guitar my instructor tapped her finger on my shoulder to help me learn the rhythm of the song, saying the name of the note to play as it came down the staff. As I got better she tapped less and said less. I can now imagine a AR/VR/DL/CL(5) system that
- Watches my eyes see the notes
- Gives light pressure to the fingers to guide towards the string
- Learns when to step back…
Helping me relearn the guitar (or any other physical skill) much more quickly…
Footnotes
- As she is in the real world.
- The world was real enough that I “knew” I needed to jump and that, since I was going down hill I would have 1/10 of a second more before I landed.
- Accuracy going up if the model is of something being shot or blown up; the “seeing water flowing past you as you tube down a river” simulations are a bit further behind.
- You can curve fit the first parts (e.g. a Taylor series) but the fine tuning requires…
- Going for the longest set of abbreviations I could, Augmented Reality, Virtual Realty, Deep Learning Closed Loop system.