This work was developed as a MSc Thesis dissertation focusing on exploring and listing challenges throughout a translation pipeline from European Portuguese, in text, to Portuguese Sign Language using a 3D avatar.
The pipeline encompasses a Natural Language Processing and Translation module, at the concept level, and then a module to procedurally synthesise an animation of the resulting utterance in order for it to be played by the 3D avatar.
As a result, there is:
This work was but the first step towards the bigger goal of having a final product in the hands of those who need it. For those who want to further develop this project, here are some examples:
Besides isolated words, there is the need to convert all sentences, which is a difficult problem for languages with such a big grammatical gap. To further advance the project in this level, there is the imperative need of more linguistic studies of the LGP by linguists and of particular phenomena such as SLs being parallel or how to handle entities. This work would involve the mapping of Portuguese grammatical constructs (interrogative, negative, verbs (transitive, intransitive...) and complements..) to the corresponding constructs in LGP. This single area can span multiple works, depending on the extent and depth of the focus on the grammatical mapping and the animation synthesis to support it.
Picking up a particularity of SLs, an interesting are of study are gestural names and how to store and reference entities in the grammatical space. Such a work would involve also a study of personal pronouns and other referencing pronouns.
As an animation/graphics only work, I suggest studying gestures according to their types (iconic, referencing, classifiers...) and other typical modes, such as fingerspelling to improve the animation synthesis according to each specific case.
With very interesting applications, a community based dictionary needs to consider distribution, quality standards and the definition of an human-usable high-level description standard for gestures that maps well to all existing gestures and to the needs of the animation synthesis.
The deployment and use cases still need to be better studied. This involves devising an architecture over the network so that the prototype can ultimately be seen on a web browser or mobile device and the data used in the most efficient way possible. The project needs several data (language models, (community/personal?) dictionaries, avatar, animation). It is an interesting problem of how best to store and use this data, considering real-life use cases (offline usage, low data traffic (do not transfer videos, transfer animation data and run real-time), real-time performance considerations, personal or community avatars and dictionaries that can be edited).
This can be a big topic, considering a believability layer for the avatar (blinking, breathing, emotions), that can be further integrated in the project by using sentiment analysis, and making the prototype respond nicely to failure cases (eg. the system can not translate words).