Android's screen reader can now answer questions about images

Today is Global Accessibility Awareness Day (GAAD), and, arsenic successful years past, galore tech companies are marking the juncture with the announcement of caller assistive features for their ecosystems. Apple got things rolling connected Tuesday, and present Google is joining successful connected the parade. To start, the institution has made TalkBack, Android's built-in surface reader, much useful. With the assistance of 1 of Google's Gemini models, TalkBack tin present reply questions astir images displayed connected your phone, adjacent they don't person immoderate alt substance describing them.

"That means the adjacent clip a person texts you a photograph of their caller guitar, you tin get a statement and inquire follow-up questions astir the marque and color, oregon adjacent what other is successful the image," explains Google. The information Gemini tin spot and recognize the representation is acknowledgment to the multi-modal capabilities Google built into the model. Additionally, the Q&A functionality works crossed the full screen. So, for example, accidental you're doing immoderate online shopping, you tin archetypal inquire your telephone to picture the colour of the portion of covering you're funny successful and past inquire if it's connected sale.

Separately, Google is rolling retired a caller mentation of its Expressive Captions. First announced astatine the extremity of past year, the diagnostic generates subtitles that effort to seizure the emotion of what’s being said. For instance, if you're video chatting with immoderate friends and 1 of them groans aft you marque a lame joke, your telephone volition not lone subtitle what they said but it volition besides see "[groaning]" successful the transcription. With the caller mentation of Expressive Captions, the resulting subtitles volition bespeak erstwhile idiosyncratic drags retired the dependable of their words. That means the adjacent clip you're watching a unrecorded shot lucifer and the announcer yells "goallllllll," their excitement volition beryllium decently transcribed. Plus, determination volition beryllium much labels present for sounds similar erstwhile idiosyncratic is clearing their throat.

The caller mentation of Expressive Captions is rolling retired to English-speaking users successful the US, UK, Canada and Australia moving Android 15 and supra connected their phones.