Maria Urbańska, this year's graduate of 1st cycle studies in Technical Computer Science, decided to develop a mobile application that will make the lives of deaf people a little easier.

In the era of online translators, it is now possible to translate most texts from various languages. Of course, the quality of translation offered by applications on your phone or computer may vary, but that's another matter. When we find ourselves in a foreign country and we don't know the language, we can communicate using an application and thus ask for directions or buy food. People who use sign language do not have this option yet. Unless there is someone in an office, clinic or school who "signs", a d/Deaf* person has trouble communicating even on basic topics. Maria Urbańska, a graduate from engineering studies at our Faculty, and more specifically in the field of Technical Computer Science, decided to remedy such situations. She is working on a solution that will help deaf people: “The idea is to develop an application that will be able to interpret some part of Polish Sign Language (PSL) in real time using the phone's camera – she explains.

How is this supposed to work? A deaf person speaks, i.e. shows a message in sign language, the camera on the phone sees it, and the application on the phone interprets the image into a text or voice message.

– At this stage it is about basic communication. The application is intended to help in everyday situations, e.g. in a store, at the post office, at the doctor's, and eventually also at the office. At this point, there is a very big problem when it comes to what difficulties Deaf people have to face on a daily basis – says Maria Urbańska. – There is a beautiful documentary „Głusza" by Anna Goc about the situation of d/Deaf people in Poland. It is shown very clearly how, for example, children of deaf people who can hear have to take an unimaginably huge burden and be interpreters in every situation, even in such that children should not be allowed to participate in. This is because there are no sign language interpreters.

A friend suggests a topic

How did it start? The newly graduated engineer has always been interested in sign language. However, because she has extensive interests and combines her studies with work, she did not have enough time to learn to sign. She made several attempts, but there was always something more urgent on her to-do list. This was the case until she had to choose the topic for her engineering thesis.

– I had no idea what I would like to do as an engineer and then a friend suggested this topic to me. She said – If you're interested in sign language, why don't you do something like this? – says Maria Urbańska. For the purposes of her engineering diploma, she developed the beginning of an application and made the first attempts at interpreting sign language signs into text. We managed to come up with a solution that is not yet perfect, but there is already a neural network that recognizes 38 signs of Polish Sign Language.

Mentoring program, fellow developer and a start-up

In order for the application to become a widely available tool, Maria Urbańska decided to establish a start-up that would develop, then promote and deliver the application. She doesn't work alone. She got help from her classmate, Dima (Dmitry) Goltstein, who is already a very specialized developer and who uses the Flutter framework for developing cross-platform applications, mainly mobile ones. He will be responsible for writing the mobile application.

– When the application is launched largely depends on how quickly we find people who would like to sign for us, because the neural network learns from data. I signed for my engineering job, but I am not fluent in sign language – says Maria Urbańska. – Obviously, if we want to come up with a commercial solution, it must be based on data collected from people who can actually sign.

Our student (she is going to start her second cycle studies in the summer semester) also has support from the „Perspectives" Shesnnovation Academy mentoring program, which helps girls launch start-ups. A mentor from this program provides support in developing the actual solution, i.e. the application, as well as in opening and running a start-up.

Maria Urbańska has ambitious plans. The program ends in June, and perhaps a prototype of the application will be ready by then.

– At the moment, the most important thing is to train the neural network that will be able to interpret sign language. The actual number and form of the signs is an issue yet unknown to me – admits the engineer. She doesn't intend to come up with it herself, she wants to ask those most interested. – We will consult with various groups of d/Deaf people. We plan to analyse about 300-400 signs, which will be enough to ensure basic communication..

The application will of course not solve all problems, but it may be a good start for further tools and for changing awareness. In Poland, many people believe that there is one sign language, but there are several. None of them has yet had an interpreting application. There are only applications that allow you to contact translators, i.e. natural persons.

Signing avatar

The tool will be expanded over time. In the future, Maria Urbańska and her colleague also plan to develop communication in the other direction. This is a much more difficult task, but very necessary.

– d/Deaf people often do not know Polish, so the problem is not only to intepret what such people sign into Polish, but also the other way around – from Polish to sign language – says the author of the application. – While doing research, we came across another application that someone had started to create, but the idea was abandoned. The assumption was that there would be an avatar that would sign the given text. You can definitely create something like this. The question is how difficult it is and what complications we will encounter.

However, before work on the avatar begins, the network needs to be trained.” The neural network, developed for the purpose of the engineering thesis, recognizes 38 signs of Polish Sign Language.

– The neural network learns from data – says Maria Urbańska. – This is where the financial aspect comes in. The application will, of course, be free, but we will try to promote it in such a way that we can still earn some money from it. Therefore, it would be unfair if the people who co-create it did not receive appropriate payment. We are looking for sponsors to finance the work. It would be fantastic if we managed to finish the first version of the application by July this year.

* The capital letter of the word „Deaf" indicates a community perceived as a linguistic minority. However, the word „deaf" written in lower case means a deaf person. The notation d/Deaf combines both of these meanings.