How AI can help map sign languages

Like spoken languages, sign languages evolve organically and do not always have the same origin. This produces different ways of communication and annotation. This is the subject of Manolis Fragkiadakis’s Ph.D. thesis.

“Contrary to what many people assume, there is in fact no universal sign language,” says Fragkiadakis. “Sign language can be even more fragmented than spoken language: there can be multiple languages in the same country. The fragmentation makes it difficult to codify sign language or write about it academically. We want to change that.”

“We have therefore created a tool for dictionaries for sign languages,” Fragkiadakis explains. “For this, we use machine learning, a form of AI. By analyzing the position and movements of the joints in the signing hands, the system has to identify which word from which language is being signed.”

This proved to be not at all straightforward. “The technological developments of recent years are not yet compatible with sign language, so it is still slow,” Fragkiadakis explains. “The large language models we currently have are only trained on one sign language and, in most of the video material used for this, you see the signer standing right in front of the camera, so the signs are easily seen.

“In my research, we trained the AI on both multiple sign languages, and multiple speakers. For this, the camera was positioned between two speakers, so the AI had to be able to extract the same information from a different position.”

For now, this technique has mainly resulted in more information being known about different sign languages, but for the future Fragkiadakis also foresees practical applications within the sign language community. “In the future, AI could assist in recognizing both the commonalities and distinctions among various sign languages, aiding communication between users.”

In addition, Fragkiadakis says it is useful that his research has helped better identify the differences between sign languages. For example, how does culture play a role? “I am Greek, so I use a lot of space to sign,” says Fragkiadakis, “this can change the meaning of what I want to say. Larger gestures can signal, for example, that someone is ‘shouting,’ but they don’t have to be.”

Ultimately, this knowledge should improve the accessibility of sign language. “The hope is that with this research we will be able to make sign language more accessible to everyone, from academics to everyday speakers. There is still too little knowledge about the characteristics of different sign languages, and we want to change that.”