Apple is building a transformative platform for augmented reality

Apple has shared some details about the accessibility features it’s working on, dropping some pretty big clues about how it sees daily augmented reality. Will we see more of this at WWDC 2022, and how will it be applied?

Make life accessible, make data real

Two of the upcoming accessibility enhancements seem to suggest Apple’s approach: door detection and live captions. Here’s what they do:

  • Door detection: Using the iPhone camera, it will detect a door, direct a user to that door, let them know if the door is open or closed, tell them how to open the door, and they can understand and read things like door numbers.
  • Live captions: Your Apple device will listen to any audio content and give you a real-time transcription of that conversation.

Both are amazing features, but when you consider them a bit, they become quite amazing. I see it this way: once an Apple device can create a real-time transcription of what it hears, why wouldn’t it be able to translate that transcription into different languages?

What could this mean

We know Apple has the technology to do this – we use it every time we translate a web page. This process is lightning fast, so why not just extend this translation to the transcription provided by your Apple device?

It could also work two ways, with your device speaking the language you can’t, allowing you to join complex conversations in multiple languages.

Door detection uses technologies that Apple has been exploring for some time. You can easily use them yourself – open Photos and search for “Lamp Post” images and you can explore any photos you have that include a lamp post.

Now, I don’t know about you, but if your device can recognize things in photos, it should be able to recognize them elsewhere using the same machine vision intelligence.

Vision + Intelligence + Context = ?

This means that just as a blind or partially sighted person might hope to use door detection to find and open a door, it’s reasonable to assume that they will be able to use similar technology to recognize anything AI on Apple devices has. a name for:

“Hey Siri, where are the oranges in the vegetable store?”

“They are three paces to your right, in the second box from the front. They cost $1.

Door detection tells us this will happen because the technology already exists to enable it. It just needs to be built.

So, what is revolutionary in all this? This means that Apple has already assembled a host of building blocks that enable its technologies to recognize and interact with the world around us. Once technology understands this world, it can help guide our interactions, augmenting our decisions with information we can use.

A blind or partially sighted person about to buy a $1 orange might be told that the same fruit is available for half that price down the street. Or a field service engineer may discover that his device has already opened the troubleshooting manual for the hardware he is looking at.

[Also read: Apple calls out Meta for hypocrisy]

What we have here are two technologies, ostensibly designed for accessibility, which also give enterprise devices an interactive understanding around sight and sound. This understanding allows the device to provide the user with useful contextual information regarding what is seen and what is heard.

This can be in response to direct questions or, reflecting the work Apple has done with Siri Suggestions, driven by on-device knowledge of the kind of help you usually ask for.

The rise of human experience has begun

You don’t have to be an enterprise pro to recognize that this opens up a range of opportunities for powerful tools and services for mainstream users, as well as extremely powerful enterprise applications around the world. intelligence of artificial vision and industry 5.0 in many sectors.

One of the great things about these apps is that because they’re based on accessibility technologies, they also allow those who may not yet be represented as fairly as they should be in certain areas to participate. more actively.

This is what I call augmented reality. And that’s what I think we’re going to learn a lot more about at WWDC 2022.

No wonder Apple has started leaking information about introducing these technologies to business leaders and the design challenges of developing the most logical vehicle for such technology, Apple Glass.

Bit by bit, the building blocks of this multi-year effort are falling into place more quickly now. I can already hear the critics getting ready to be wrong again.

Please follow me on Twitteror join me in AppleHolic’s bar & grill and Apple Discussions groups on MeWe.

Copyright © 2022 IDG Communications, Inc.

Leave a Comment