Seeing AI, my pick for the best iOS app of 2017

A long time ago, and it almost seems like it was in a galaxy far, far away because times were so different, I encountered my first Kurzweil Reading Machine.

It was about the size of a washing machine, lived in its own dedicated room at the school for the blind with a special electrostatic mat, and it sounded like a jet engine when it booted up.

As a child, I remember having a giggle at one of the demonstration documents that came with this wonderous monstrosity, which said, “I can read X rated books, but I don’t get excited about them”.

That same document, highlighting the capabilities and limitations of this device costing tens of thousands of New Zealand dollars at the time, warned that reading handwritten material was beyond the machine’s capabilities.

Over the years, the reality of Moore’s Law, which Ray Kurzweil has spoken so entertainingly about in numerous presentations, has meant that the accuracy, speed and capabilities of that initial huge device have both improved and shrunk, to the point that something far more powerful is now on my iPhone.

And yesterday, another barrier came down. As someone who has been using assistive technology for around 40 years, it was a special, and deeply moving moment to download an app that, at long last, could recognise handwriting and speak it back to me as a totally blind person.

Not only has this technology experienced a massive power increase and a remarkable size decrease, the price has gone down to 0.

The release of version 2.0 of Microsoft’s Seeing AI app, in only the first year of its availability is a triumph for the company, and particularly for those involve in its creation.

On Mosen Consulting’s podcast, The Blind Side, I interviewed Seeing AI’s creator, Saqib Shaikh.

Saqib is one of us. He’s a blind guy who has grown up using assistive technology and has seen it evolve. First-hand, he experiences the challenges we face in a world full of visual information, be it printed text, handwritten text, colour, lighting or photos. As one of their employees, Microsoft gave him the space and the resources to dream of innovative solutions to complex problems, to consider practical applications of emerging technologies that will meet the challenges we encounter.

There are a couple of take-aways from this. First, if you’re a young person thinking about career options, it can be rewarding to shape the future in some way. Whether it be through coding, product management, or some other field in this industry, if you can take a step back and think to yourself, “I was able to make a difference”, that’s a special and worthwhile thing. Saqib is just one of several blind people doing just that, and I salute them all.

Second, remarkable things occur when a company offers some space for talented coders to brainstorm and dream. Well done to Microsoft for providing that space.

The feature set of the Seeing AI app is impressive. But what also makes this app so good is that its user interface is elegant, optimised to suit its end users.

As a trainer, I’ve seen many totally blind people struggling with any app that uses the camera. Perhaps part of this is psychological. The idea that blind people can make practical use of something as visual as a camera doesn’t compute for some. Seeing AI’s clear documentation and narrated video tutorials give the user confidence. Without being patronising, they clearly explain the relationship between the size of objects and the distance a camera needs to be from an object.

The “getting warmer” method employed to detect barcodes means that many who gave up on other barcode-detection solutions in iOS are up and running. The picker item to navigate between channels is simplicity itself.

In the episode of The Blind Side Podcast in which I interviewed Saqib, I also demonstrated the feature set of the Seeing AI app as it was then. But I’ve product managed, written about, and used enough assistive technology to know that what really counts is whether people find a place for a piece of assistive technology when all the hubbub has died down. For me, there is no question, Seeing AI very quickly became one of the most important apps on my iPhone. It has pride of place on the first page of apps, where I keep the apps I use daily.

Here are just a few of the ways I have used the app.

The short text feature, which is the default channel when one launches the app, has made life so much simpler. When the printed mail arrives, what could be simpler than hovering the camera above the envelope to know whether the mail is something that needs to be open, or junk mail that can be sent straight to the big round file.

Often, opening the mail and hovering the camera above the printed page is sufficient to glean the information necessary. But if not, the Document channel gives me voice guidance to ensure I have the entire document in the camera’s view, then the app takes a picture. To add icing on the cake, Seeing AI then attempts to intelligently format the resulting text, meaning that key parts of the page are often navigable by heading.

Short text is useful for much more than words on a page. I use it to read information on a range of displays. A couple of weeks ago, the Sony Bravier smart television I demonstrated on The Blind Side Podcast back in April required a software update. Like many such devices, when it’s being updated, a blind person receives no accessible indication that the update is even taking place, let alone how far through the update the device is. Having waited a while and grown concerned about the time the update was taking, I got the bright idea of getting Seeing AI on the job. The Short Text channel read the screen clearly, letting me know that the update was still proceeding. It even left me in no doubt when the TV was rebooting.

When shopping arrives, we regularly use a combination of the short text channel and the barcode channel to identify boxes and cans that might be ambiguous. Thanks, Seeing AI, for ensuring we don’t open a can of dog food, thinking it’s the tinned peaches.

Seeing AI has changed social media for me, with its ability to describe photographs, and read text contained in them. I follow many political journalists around the world. When they’re in a hurry, they might take a screen shot of some text a press secretary has sent them, or of a piece of paper that has been handed to them. Seeing AI brings all that to life for me as a totally blind person. It’s also empowering not to feel excluded when someone sends a tweet which contains a photo, with no text giving me any indication as to what the photo is about. Yes, it’s good practice to caption images and we must continue to encourage that, but the world’s not perfect, so solutions like Seeing AI are helpful.

Now, the app has a handy-dandy light detector, very useful for a totally blind guy with no light perception to be sure the kids haven’t left any lights on. It’s also useful to check lighting conditions before taking a photo.

The currency identifier will be helpful when I can use it here in New Zealand. Presently, NZ currency isn’t supported, so hopefully Microsoft will follow through on its promise to add other currencies soon.

And there is a colour detector, something I’m still experimenting with at this early stage to assess its accuracy.

This is a veritable Swiss army knife for overcoming the information barriers blindness can pose.

What can I say except Microsoft has hit it out of the park with this intuitive, capable, game-changing, totally free app.

With Apple now offering hardware at a range of price points, and the technologies powering Seeing AI improving constantly, I believe this app will be remembered as an app that redefined access to information. Well done to all involved.

Being able to feel so uplifted and positive about an app is a terrific way to conclude the blog posts for this year.

I can’t wait to see what 2018 brings in technology.

Categories