Google shows off AI tool for reading handwritten text by rewriting it digitally

Google Research is showing off a new way to use AI to read handwriting that might radically change how machines convert what you put on paper into digital letters. The InkSight system transforms photos of handwritten words into digital text by leveraging AI without the need for any devices as intermediaries.

The idea is to replace the sometimes fallible optical character recognition (OCR) with AI that can emulate how humans actually learn to read, specifically by rewriting existing text to learn what whole words look like and mean. Doing so required the researchers to tutor the AI in both recognizing and mimicking handwriting by humans.

“Digital note-taking is gaining popularity, offering a durable, editable, and easily indexable way of storing notes in the vectorized form, known as digital ink. However, a substantial gap remains between this way of note-taking and traditional pen-and-paper note-taking, a practice still favored by a vast majority,” the researchers explain in their paper. “Our approach combines reading and writing priors, allowing training a model in the absence of large amounts of paired samples, which are difficult to obtain. To our knowledge, this is the first work that effectively derenders handwritten text in arbitrary photos with diverse visual characteristics and backgrounds.”

InkSight is more than just an alternative technique. It makes for more accurate results in circumstances that aren’t ideal. For instance, if the photo is taken in dim light, has partially obscured text, or is on a confusing background when examined with OCR. The researchers found that humans could read 87% of the InkSight-made tracings of text. Two-thirds were good enough that people couldn’t tell them from actual handwriting; you can see below how it looks when InkSight works.

Penned by AI

If you like writing things by hand, InkSight has some potential benefits. Imagine writing by hand in a paper notebook, then showing the notes to your camera to instantly make them searchable and organize them in context with previous notes on physical pages. If you’re like me and have particularly messy handwriting, InkSight could help turn your chicken scratch into typewritten text that is still accurate to what you scribble.

On a bigger scale, this could be a crucial tool for deciphering and converting handwritten text from across the centuries into digital form. Even when the text is in a language without much of a digital presence, InkSight could help preserve handwriting to help build up training sources for those languages.

Google isn’t the only place where AI tools to decipher handwriting are underway. For example, Amazon’s new Kindle Scribe upgrades the e-reader’s ability to transform handwritten notes into legible text. There’s also Goodnotes, a digital notetaking app that can read handwriting, and recently debuted handwriting editing tools using its Goodnotes Smart Ink technology to turn handwriting into typed text. The added tools let you edit handwritten notes as if they were typed, including aligning notes, copying and pasting, and reflowing the text to make it more logical.

You might also like…