Microsoft’s latest video for the HoloLens AR experience (embedded below) is a blast, and if I go on to write with any critical bias (yes, I am going to), I should state right away that I believe Augmented Reality in general, and HoloLens in particular, will eventually change society in a profound way that will leave VR as an inconvenient and niche curiosity. In my opinion MS is developing our future with this technology, and I await the next crazy, sci-fi-packed video with a fanboy’s anticipation.
However, there’s a ‘World’s Fair’-style optimism to HoloLens’ current fervour that needs some serious counterweighting. The latest video, which departs from HoloLens’ only-faintly-sober business rationale of recent outings, is a much more anticipative look at what AR will one day be capable of than has been offered before – a fanciful trailer for a world your kids are more likely to inhabit than you are.
In the video a young interior designer/architect is tasked with creating a corporate space in a relatively uninspiring barn, and uses non-specific (but highly interactive) AR architectural tools to experiment with ideas; her frustration eventually causes her to turn to friends (one of whom is seated at a desk enjoying a VR experience), all of whom ultimately assemble in a mad virtual space that’s horribly reminiscent of Microsoft Bob, in order to find elements that might cause a moment of inspiration.
The video, intended as pure inspiration for the technology, is tremendous fun, even featuring a Star Trek transporter noise as one of the team joins our heroine in her plight – ‘Cute!’. The mini-adventure is also facilitated by a floating, Wall-E- style future iteration of (presumably) Cortana, who is able to zip into and out of diverse VR and AR platforms to interact with the characters in the form of a cute floating head.
Eyes front for the HoloLens
The AR on display in Windows Holographic: Enabling a World of Mixed Reality has a literally wider view than previous HoloLens promo videos, nearly all of which feature people interacting with discrete objects of limited size…one at a time. Paul Thurrott identified (along with other writers) the core issue with HoloLens’ field-of-vision a year ago, and it’s one that seems to be exacerbated as the product evolves towards general consumer release – the AR experience is ‘letterboxed’, with the portable technology struggling to focus its untethered processing power onto a central rectangle in the user’s view – like a minuscule QuickTime embed from a 1997 web page:
‘The field of vision is far too small. That is, as you look forward out through the HoloLens headset, you can of course see peripherally, but the area in which you can see holograms is a small rectangle in the middle of your vision. It’s like looking through a small portal, or a submarine periscope.’
I can well understand fellow fanboy Adam Dachis defending this limited ‘area-of-interest’ last week, and I agree with him that we have to indulge the postage-stamp AR experience if we want to work towards the enveloping vistas of the latest HoloLens video fantasy. But even as a fan, Dachis posts illustrations which make clear the current gap between the experience depicted in the new video and what select journalists have been able to see in using the technology so far:
Portable, powerful, long-lived – choose one.
I further agree that compressing a Windows 10 install that’s crunching some extraordinary graphical problems into a headset that doesn’t break the user’s neck, require a VR-style base-processing station or have a battery life of ninety seconds is going to lead to trade-offs, and that further development will ultimately improve the balance of those trade-offs in the consumer’s favour.
But these compromises are likely to stay critical for a very long time, since some of them concern far more general problems in computing technology, such as heat from GPU activity – a greater hazard in a head-wearable even than in a smartphone. Stepping up graphical processing capability and FOV will add weight and/or heat under present limitations of passive heat-sink approaches.
Even solving those issues – currently a fairly science-fictional proposition in itself – leaves HoloLens with the problem of power consumption as the GPU cycles higher to achieve greater FOV, resolution, frame-rate and involvement, burning through local power storage in a device that Microsoft seems Quixotically determined will be independent of remote processing or wired connections.
The miracle of HoloLens auto-mapping
Some of the most ‘aspirational’ moments in the new video are quite subtle, such as when our heroine enters the empty architectural space and the HoloLens-to-be maps it instantly, so that her subsequent projections are bounded properly by the space:
Well, Microsoft has been accused of cheating more than once in live HoloLens demos, and here it glosses over a huge problem in portable AR – mapping the boundaries – by choosing a practically rectangular space with no occlusions to ‘auto-map’. We see the HoloLens (presumably) using image recognition technology and AI to identify six planes (4x walls/ceiling/floor) and assign base nodes to represent them, a feat that’s realistically achievable with AI and IR (though perhaps not in two seconds!).
Pretty lucky that there are no occluded aspects to this brick box; no major items of furniture, deep buttresses, angles, obstructions or concealed geometry of any kind, because mapping that kind of architecture would currently be an afternoon’s work with specialised equipment – and image recognition would be of limited use. Get it wrong, and your HoloLens phantoms would fly straight through objects like a classic videogame clipping glitch.
To be fair, even in its more fanciful video outings, MS does not pretend that you will be able to walk randomly into complex, unmapped spaces, map them automatically and have your holograms respect those boundaries. Most HoloLens videos deal with specific, relatively small objects under central-focus scrutiny in controlled environments. However, the setup is not likely to ever be quite as easy as the new video suggests.
Hands up! Err, no, down. Hands down.
At one point in the video Microsoft seems to be having a bit of fun with Oculus and other VR-based competitors, showing a geeky guy tethered and blind at a desk, enjoying a full-VR game – goofing off and escaping the world while his team-mate is out there trying to do some real work with the HoloLens and Augmented Reality.
It’s not a mean moment, but it does unintentionally reveal one of the fundamental limitations of AR – and it’s one that is perhaps further away from resolution than any of the others that HoloLens developers are facing – occlusion.
In our geek friend’s POV we see his virtual hand interacting with the entirely virtual world:
Fine – the hand is 3D geometry, it can do what it likes in the VR sphere, presumably synced to the user via Kinect-style real-time mapping and motion-capture.
But later, for the first time in a HoloLens demo, Microsoft shows something which the HoloLens not only cannot currently do, but seems unlikely ever to be capable of without major advances in across-the-board technology: it shows our heroine’s real hand matted out over a hologram:
It’s not much of a matte – just the tip of an index finger, but it’s pure science-fiction right now: the ability to recognise and clip out unpredictable real world items in real time, without chroma or keying elements available. Thus the video shows us the deep level of interaction that VR is capable of, and the ‘hands off’ approach that AR has to live with.
In all its live demos and videos, MS has until now been very careful not to depict clipping in this way, and previous videos have shown non-direct, gesture-based controls where the user’s hand is either near (but not on top of) the virtual object, or else melds subtly into it, revealing the object’s non-substantive state.
With a properly mapped AR environment, clipping can be calculated for static objects, such as chairs and tables (although you’d better not jog them or otherwise move them, once mapped), but the kind of cut-out made possible in movies via green-screen would require extraordinary processing power – and not the kind that’s likely to become lightweight and portable in the near future.
As for user interaction with clipping, even if one were to require that AR users wear distinctive coloured arm-apparel, thus reversing the typical green-screen proposition by dropping the foreground object (the arm) out of the background object (the hologram), there’s still no way to drop player #2 out of a hologram, should he or she walk directly in front of it.
In terms of remapping real-world environments, where fantastical or historical elements might be overlaid on top of genuine locales, such as cities, the proposition of clipping out scores of lamp-posts and other obstructions is more than daunting.
The future of AR and the HoloLens
But these are the Icarus days of AR, the long years where we resign ourselves to not seeing the best of a new technology either within our lifetime or within our youth. And despite these, and many other problems, I retain that Augmented Reality is set to be the most transformative technology in decades.
Progress for the HoloLens will depend on how stubborn Microsoft and other AR developers are determined to be with the variables: frame-rate vs. resolution, FOV vs. battery life, portability vs. depth, spontaneity vs. accuracy. I’m certainly pleased that the problems are being addressed – though not always with maximum candour.