Random thoughts on Android Jellybean and Google Nexus 7 Accessibility

If you follow developments in mobile technology, you may already know that a new version of the Android operating system called “JellyBean” has been launched last month along with a new 7 inch tablet device called the Google Nexus 7. It was also announced that JellyBean came with a number of accessibility improvements. To know more about these improvements, either watch this hour long or the announcement on the Eyes-Free Android blog.

I needed to test the new accessibility improvements in JellyBean and so I ordered a Google Nexus7. I was happy to note that the help documentation for Google Nexus 7 has a section dedicated for accessibility features. Anyone considering a purchase of an Android JellyBean device should read through this information.

In spite of reading through this section and doing a bit of research in advance, I ran into a couple of problems while setting up the new Google Nexus 7. I thought I will document them here with possible workarounds that I found.

Turning the device on for the first time

If you are blind and do not have sighted assistance, don’t be in a hurry to unbox your Nexus 7 and turn it on. It would help to prepare a bit.

Firstly, you will need to have a pair of headphones ready. You don’t have to use them right from the start: Nexus 7 does have an external speaker but it would be very hard to complete the setup without them as you will see shortly. Secondly, it would help to have a flat surface like a desk. There is a gesture to turn on Accessibility mode which fires up Talkback with Explore by Touch mode active. This gesture is rather unreliable and seems to work best when the device is placed flat on a desk.

If you are holding Nexus 7 in the portrait mode, the power button and the volume rocker should be along the right edge of the device towards the top. The 3.5mm headphone socket is along the bottom edge towards the right. Don’t forget to connect the device to a power source before you start. The mini USB port is along the bottom edge towards the middle. It seems to ship with an almost empty battery.

When you turn on the device by pressing the power button, you will be placed on a welcome screen. You can turn on the Accessibility mode right here. You need to touch the screen with two fingers placed slightly apart and hold for 4seconds. When you place 2 fingers, in the first second or so, the device should recognise the gesture and will ask you to continue holding for a few more seconds to turn on Accessibility Mode. Don’t remove your fingers from the screen and no matter what you do, don’t let a third finger touch the screen in these few seconds. I did this exact thing by mistake and the device said “cancelling accessibility mode” and wouldn’t talk to me at all for the next 3 hours no matter what I tried.

It appears as if the gesture works better when you use both hands with one finger touching towards the top half of the screen while the other touches towards the bottom half. If for some unfortunate reason, you end up in a situation where the device says “cancelling accessibility mode”, you can try the same two finger gesture again. Even though the device may not speak, you may hear a click sound after 4 seconds or so. At this point the Accessibility mode may come up on its own. If not, try drawing a rectangle on the screen in the clockwise direction. This gesture is supposed to enable Talkback and explore by touch although it didn’t work in my case. As a last resort, you may try the two finger gesture again a few times. You might just be lucky!

Of course, if you have the chance, you can also setup the device with sighted help and then turn on Talkback from Apps > Settings > Accessibility > Talkback. A number of users on Twitter and on the Eyes-free mailing list seem to be having issues with the gesture to turn on Accessibility mode. I tried the two finger gesture and the rectangle gesture countless number of times and spent close to three hours on it before it somehow worked. I hope that this issue will be addressed at some point in the near future. Turning on Accessibility mode with hardware keys might be a better way for most users. A large majority of Android devices will have at least a power button and volume control keys.

Assuming that you got the two finger gesture to work and you have the Accessibility mode turned on, you will be placed in the explore by touch tutorial. It only has three steps but I found it to be very useful. It covers the important gestures that you will need to learn to use Talkback with explore by touch mode enabled. Once the tutorial is finished, you will return to the welcome screen where you need to pick a language and activate the “Start” button to start setting up the device.

When you get to the screen to enter the password for the Wi-Fi network, look for the checkbox which says “show password” or “display password” and check it. If you don’t do this, all characters on the virtual keyboard are read as “dot” by Talkback. Unfortunately, when entering your Google account details, the “show password” option isn’t available (for obvious reasons). This is where the headphones come in handy. In order to enter the password with the virtual keyboard, you need to connect the headphones. Once headphones are connected, the characters on the virtual keyboard are announced correctly. Otherwise, you will just hear “dot” no matter which character you touch.

This is a great idea for security reasons but I don’t think it is smart to enable this by default. What if I don’t have a pair of headphones handy when I am setting up the device? This feature can be turned off by selecting “speak passwords” checkbox in apps > settings > Accessibility but it isn’t possible to get to settings from the initial setup screen. I just don’t think this is a smart usability decision.

Accessibility mode gesture isn’t available on all devices

It is important to remember that the 2 finger tap and hold gesture to turn on Accessibility mode may not be available on most devices. Both the current Google devices, Nexus 7 tablet and Galaxy Nexus smartphone support this. on other Android devices from manufacturers like HTC, Samsung, Motorola and so on, the components required for accessibility support will have to be either downloaded from the Google Play app store or enabled in Apps > Settings > Accessibility. In addition, these manufacturers may also customise the user interface of Android with skins. These skins may or may not work with Talkback and other accessibility features. For now, it is best to stick to Google devices which run stock Android (with no customisation) if you need to use a screen reader. There are ways to install stock Android builds on non-Google devices but this is not easily accomplished.

New gestures

Explore by touch mode was introduced in Ice cream Sandwich (Android 4.0) and this is improved in JellyBean. In addition to touch exploration where you would move your finger along the screen for Talkback to announce the items you have touched, you can also use gestures to navigate through these items. For example, swipe right and swipe left moves focus to the next and previous item on the screen respectively. Double tap anywhere on the screen to activate the last focussed item.

While you can get to any item on the screen by touching it if you remember the exact position, a combination of touch exploration and swipe gestures is the most efficient way to navigate a touch screen in my opinion. Introduction of swipe gestures is an excellent addition to Android Accessibility.

A list of all Accessibility gestures in Android 4.1 (JellyBean) are available under the accessibility section in Nexus 7 help documentation. By default, the left and right swipe gestures move focus by item. But you can change this to move by character, word or paragraph. Swipe down with one finger and in the same gesture swipe up. Talkback will say “read by character”. You can now use the left and/or right swipe gestures to read the current item character by character. To return to the default navigation mode (moving by item), keep using the swipe down and up gesture until Talkback says “default reading level”. Swipe up and down toggles backwards among the available navigation modes.

There are a couple of very important gestures missing and I haven’t found an alternative way to perform these actions:

· A gesture to pause or stop speech. This doesn’t seem to be possible on the Nexus 7 tablet but Marco Zehe has noticed that tapping his Galaxy Nexus on the top edge usually silences speech. I suspect this is because of the proximity sensor on the phone. But really, a screen reader should have a pause speech command/gesture.

· A gesture to read from the current item to the end. I thought this is a basic requirement for a screen reader. But there doesn’t seem to be a way to do this. For example, if I am reading a long email thread in the Gmail app, I have to read the entire body of the message at one go or read by character/word/paragraph. Further, reading anything remotely lengthy in a web browser is extremely painful without this command.

A word about navigating lists

It is best to illustrate my point with an example. When you open the Apps > Settings screen, Talkback will inform you that you are on a list showing 18 items. There are actually 24 items in the Settings screen on my Nexus 7 but only 18 items are visible on the screen. In order to get to the other 6 items, you will have to manually scroll with scrolling gestures. If you do not do this and continue navigating with left/right swipe gestures, you will only keep moving through these 18 items in the list and other items on the screen. In order to scroll down to the other 6 items, place two fingers on the screen and move up. You can also swipe right and swipe left in the same gesture to go to the next page in a list.

I wanted to point this out because this behaviour is contrary to how most other user interfaces work especially for screen reader users. When navigation commands are used, list controls tend to automatically scroll to display hidden items.

Talkback behaviour is particularly confusing because when you move focus to a list control, it only informs the number of items that are being displayed but fails to mention how many items there are in the list. In the Settings example for instance, instead of just saying “list showing 18 items”, it would be useful for Talkback to say “list showing 18 of 24 items”. Without this information, users will always have to attempt manually scrolling the list just to make sure they are not missing any items.

Virtual Keyboard

Using the default Android keyboard in JellyBean is straightforward. When you encounter an input field, just double tap on the input field to bring up the keyboard. Move your finger along the keys and when you have found the character you would like to insert, just lift your finger. Talkback will read out the inserted character at a lower pitch. On the Nexus 7, I am noticing that I have to be precise with my finger placement on the characters for them to be inserted. If I don’t lift my finger in the exact centre of the character, it doesn’t insert it. to insert numbers, switch to the “Symbols” keyboard by tapping the “symbols” toggle on the bottom left of the keyboard

Do remember that you will need headphones to enter passwords if you haven’t turned on “speak passwords” in Apps > Settings > Accessibility.

Entering a Pin on the lock screen

If the Nexus 7 needs a pin to unlock, the way to enter it is slightly different to the usual way you enter text. On the lock screen, there is an input field without a label and immediately to the right is a button without a label as well. This unlabelled button needs to be activated by double tapping before entering the pin. To insert digits of the pin, just tap the digits on the telephone keypad style virtual keypad. Don’t double tap them. Just tap the digits and lift the finger. Even though there is no feedback, the digits are entered. When you are done entering all digits of the pin, tap the enter key to the right of the digit 0 on the virtual keypad. On the other hand, if the device is locked with a password, you would double tap the unlabelled input field which brings up a standard virtual keyboard.

Voice recognition

Voice recognition is one reason why I keep trying to switch to Android. It does an excellent job recognising my accent. It is much more accurate than Siri on iPhone. Voice Recognition is available throughout Android. Basically, you are able to speak into almost all input fields in Android.

Before you start using voice recognition though, make sure you connect a pair of headphones. Trying to use voice recognition when Talkback is running and speaking out of the built-in speaker is a recipe for disaster!

When you are composing a text message or an email, look for the “voice input” button on the keyboard. This is between the “Symbols” toggle and the spacebar on the bottom left corner of the device. Tap this and lift your finger and you will hear an earcon notifying you that the device is ready for you to speak. Just speak normally and when you are done, wait for a few seconds. The device will play another earcon to indicate that it has paused recognition. You will also notice that the keyboard now only has one item which says “tap to speak”. Double tap this button to start recognition again. In order to get the normal keyboard back, you need to activate the Back button on the bottom left of the device and then double tap the input field again. This process is a bit clumsy but I haven’t found an alternative.

When you are speaking, Talkback will start reading out words as they are recognised by the device and entered in the input field. This can be a bit distracting. And by now you can guess why I suggest using headphones. When the device is in voice input mode, Talkback will read the text inserted by the voice recognition feature. However, the voice recognition features starts to recognise Talkback’s speech output and inserts the same words again in the input field. This results in an infinite loop with Talkback reading out the same words and the voice recognition feature inserting the same words over and over again. This is rather funny to watch but makes for a horrid user experience.

In spite of this, I still say that Android’s voice recognition feature is fantastic. I can probably put up with having to use headphones for the recognition accuracy it gives me especially with my Indian accent.

Further, you can also use Google Voice Search for some simple requests such as asking for local weather information and so on. If Google Voice Search finds a simple answer, it will speak out the answer. If it can’t find an answer to your question, it just displays Google search results. Google Voice Search is located on the home screen’s dock. Surprisingly, Talkback doesn’t interfere with voice recognition when using Voice Search. So this can be used even without headphones.

Google Chrome

Talkback now works with Google Chrome which is the default browser on Android. It does recognise various HTML elements such as lists, headings, landmarks and so on but there is no way to navigate/jump among these. The available reading levels are still character, word and paragraph.

Further, the explore by touch feature doesn’t seem to work reliably on a web page. Although navigating by left and swipe gestures works fine, exploring a web page by moving a finger along the screen doesn’t seem to reliably work on all web pages.

There is an option to “enhance web accessibility” in Apps > Settings > Accessibility which needs to be turned on for Google to insert certain scripts which are supposed to improve accessibility on web pages. If you are having problems on the web, it might be worthwhile to check if this option is enabled.

Other nice features

App developers should always test their apps with Talkback. JellyBean makes this a little easier. There is an option to display speech output under Apps > Settings > Accessibility > Talkback > Settings > Developer Settings. Enable this option and there will be a bubble on the screen which displays what Talkback is speaking.

I was also happy to notice that the default email app on Android and the Gmail app have been made accessible. They now work with Talkback for most part except for minor annoyances. The read/unread status of an email message isn’t spoken by Talkback. Plus the body of an email is displayed in web view which means explore by touch doesn’t work correctly. One has to use the left/swipe right gestures to navigate and read the message. But I can read email messages from my work account as well as Gmail. There is also K9 email which is supposed to be an accessible email client but I haven’t used it in a while.

I was trying to find an accessible Twitter client for Android. UberSocial seems to be the answer but the app displays timelines in web view which makes it a bit difficult to use.

I am sure I missed a few great features but as the title of the article indicates, these are random thoughts and I am not trying to do a full review. Further, there are other accessibility features that I haven’t tried out. There is a large text feature in Accessibility options. I am not sure how this is supposed to work. And there are a number of third-party accessibility services which I may test when I have the time.

If you have any tips or tricks or recommendations on apps, please get in touch on Twitter @kirankaja12.

About these ads

Tags: , , , ,

6 Responses to “Random thoughts on Android Jellybean and Google Nexus 7 Accessibility”

  1. Quentin Says:

    Hi Kiran,

    For Twitter, I found Plume, Seesmic, Tweetcaster and Ubersocial all seem to work with Talkback. The official Twitter app doesn’t at this stage. On a similar topic the official facebook app does read, and I could get around it, but many things are simply announced as ‘link’. Seesmic and Ubersocial both let you access both Twitter and Facebook.

    I’m still looking for a solution I really like myself, though ideally for me, I’d like one which can do large print (Tweetcaster is the best so far – it works only on Twitter but goes up to equivalent of 18 point when viewed on my Galaxy Nexus, though it looses a couple of things like the time stamp in doing that).

    Regards

    Quentin.

    • kirankaja12 Says:

      Hi Quentin, Thanks for that list of Twitter apps. I will try those. If you have JellyBean on your Nexus, (not sure which Nexus it is), I am curious to know if you have tried the large text option. I am wondering if this is a system-wide large text option or if it is limited to Android menus and such. I haven’t been able to get this tested by a partially sighted user as yet.

      • Quentin Says:

        Hi Kiran,

        I do have Jellybean on my Nexus – the ‘Large print’ option was introduced in ICS, and I must admit I’ve been quite disappointed by it to date – It’s more or less system wide (although individual apps can set their own font), however the difference is pretty minimal – On my old Samsung Galaxy S with a 4.0″ screen, the text in say the play store is about 8pt, and on my Nexus with a 4.6″ screen, the same text is about 10pt (compared visually to a test sheet of paper next to the phone). In actual fact on Android 2 (on my Galaxy S) I used an app called “SMS Text Enlarger” which was designed (as the name implies) to enlarge text messages, but also works in a number of other places such as the play store, and using that, the text on the Galaxy S goes up to about 13pt. I did exchange emails with one of the Google devs who said they have made big improvements with Talkback for blind users and they are now going to put some resources into large print accessibility, so I’ll be interested to see what is in the next version of Android when it comes out.

        Regards

        Quentin.

  2. Quentin Says:

    Actually if you’re curious, my current setup on the Nexus, is using a custom launcher called “Lightning launcher” which allows you to pinch zoom in on the desktop (like you can in the browser). and that at least gives me a bigger home screen, and I have tried using a number of widgets for things which works well though invariably you do have to open whatever the app is to do every task (eg, there are a few twitter apps with widgets, but none which really let you do everything you can in the apps themselves).

    Big Launcher is a great home screen option too – it also works well with Talkback) – the whole screen is taken up by 8 icons and contacts and text messages etc can be displayed in fonts up to about 28pt. It’s only a couple of dollars and is one I have a few clients using.

    Regards

    Quentin.

  3. Juan B Says:

    It was a fantastic post for just being random thoguhts. it hink those posts are more nutritive than a review. For my part i use a sony xperia mini pro with cyanogenmod installed (it is acutally not that hard, the only troublesome part is flashing it which requires sighted assistance) . I have never been able to try voice recognition, I am not sure if it works in spanish (I am from Colombia by the way). As far as twitter goes, I used to have the offical app until thigns started to break. Version 3.1 worked fine, tweets were read outloud when you had opened them using the selector key, but other than that from 3.6 on things really started to get ugly. For browsers I only have found firefox to be the greatest, they have even got their own way of navigation. With the keyboard it makes things a lot easier because you can jump by headings, lists, tables ETC. SOmething happened after version 18 though, which breaks things down for us keyboard users, but since firefox is quite an active browser I think htat will be fixed in a matter of months. THere is a version of facebook messenger that i have kept since it is the last known accessible version. I have the official facebook app, but only to get notifications as things happen. But maybe I would try switching away from the official app sinec it drains the battery and is a bit slow just so I can get notifications in real time, because the web versionwith firefox is just great.

  4. Switching to Android full-time – an experiment | Marco’s accessibility blog Says:

    […] support, or lack thereof, I would like to point you to Kiran Kaja|s excellent Nexus 7 reviews part 1 and part 2. Here, I would like to add a few impressions of apps I use […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Follow

Get every new post delivered to your Inbox.

%d bloggers like this: