Google's Handwriting Recognition

"a silent warning"

This post is not directly about the provenance of medieval manuscripts, but it concerns something that I am sure will increasingly affect provenance research (not to mention research into most other subjects).

The next several paragraphs may seem irrelevant, but bear with me.


Even those of us who avoid using proprietary software [1] and those of us careful about what we share with the other big corporations (Amazon, Facebook, et al.), often make a compromise for Google. Most of us use Google for searching the web, despite the fact that this allows them to collect lots of personal data about us. Similarly, I am willing to trade some of my privacy to Google in exchange for a product as useful as Gmail: Gmail's spam filter is so good that I get fewer than one spam email per month, and about the same frequency of (usually mass-mailed) legitimate emails wrongly being identified as spam.

With a Gmail account comes 15Gb of free cloud storage in Google Drive [Wikipedia], which can be used to store emails, images, documents, music, etc. (just like a hard drive, hence the name). I pay for lots of extra storage space, in addition to the free 15Gb, and use it to synch and backup almost all my files. Apart from the obvious peace of mind of having a complete backup of my data in the cloud (so all is not lost even if my home burns down, or a burglar steals my computer containing my hard drives), this also means that I have access to all my files, even if I am away from my desktop and only carrying my phone.

One advantage of storing documents in Google Drive (although it is also potentially a privacy disadvantage, of course) is that Google searches and indexes the contents of my files for me. This means, for example, that if I store a PDF of a journal article in Google Drive, and later cannot remember the author or title, I can do a free-text search, and find it by its contents.

A few years ago I needed to find a document I had written, but couldn't remember where I had filed it, so I did a search of my Google Drive using a keyword search, and was surprised to see that one of the results was an image file. Puzzled, I opened the image and was shocked to see that my search term appeared in the text of a street sign in the background of the image. Since then, the same thing has occurred often: I search for a word or phrase in my Google Drive, and Google finds the word(s) appearing somewhere in my images.

I now do not remember what search term I used a few years ago, but for the purposes of this blog post, I tried searching my Google Drive for the word "warning". Sure enough, this is what the results look like:

[click to enlarge images]

The image in the lower left corner of the above screenshot includes a painted sign (which I photographed in the beer-garden of a pub I visit this summer):

If you think Google was "clever" to find that prominent sign in one of my images, look at these other images that the same search also retrieved. In this first image, the word "warning" appears in the text on the noticeboard at the bottom: 

In this example (taken while on a guided tour of graffiti and street art), the word "warning" is hand-painted on the wall at the left:

In this next image, the word "warning" does not appear at all, but Google has found an excellent match for the concept: the image includes a tsunami warning-sign on a beach near Los Angeles:

Certainly on Android phones, and I assume also on iPhones, you can search your images for concepts such as "birthday party" or "puppy" or "autumn", and the algorithm will do a pretty good job of finding images that match each concept.

But the reason I am writing this blogpost now, is that I have recently realised that Google's search function is now immeasurably more powerful, and seems to have been "learning" to read scruffy handwriting.

Another result of my Google Drive search for the word "warning", is this image, showing a page of handwritten notes concerning a medieval manuscript:

At first sight I could not see why Google had returned this as a search hit, and then I read the text carefully. On the first line of the last paragraph, the word "warning" does indeed appear, in the phrase "a silent warning":

"A tick at the end of the lines is a silent warning ..."

Lest you think this was an accident of some sort, I searched my Google Drive for the word "distinguished", and among the hits was another page from the same notebook:

Here's a detail:

"Sarum Use is distinguished by ..."

The handwriting recognition is not perfect, of course. Another image retrieved by my search for "warning" is this one, in which the text is written in Latin; I do not know which part of this image was "read" by Google as the word "warning":

Even though this is a false match, this proves that Google's handwriting recognition software is attempting to "read" the script of medieval manuscripts.

We all know that OCR (Optical Character Recognition [Wikipedia]) of printed works has been getting better and better for the past 20 years, but it seems that Google's image- and text-recognition algorithms are a long way ahead of where I previously believed them to be. How long can it be until Google is able to read "easy" medieval handwriting, like rustic capitals and caroline minuscule, and eventually late medieval gothic textualis and irregular cursives?






[1] I only use Linux (Ubuntu [Wikipedia]), for example, and have not bought any Microsoft/Apple software for more than 10 years. (It was the purchase of a laptop with Microsoft Vista pre-installed that pushed me over the edge and made me realise that there had to be better alternatives, and it turned out that most of the best alternatives are free, Open Source, and immune from Windows viruses).