Exploring Challenges in Avatar-based Translation from European Portuguese to Portuguese Sign Language

This work was developed as a MSc Thesis dissertation focusing on exploring and listing challenges throughout a translation pipeline from European Portuguese, in text, to Portuguese Sign Language using a 3D avatar.

The pipeline encompasses a Natural Language Processing and Translation module, at the concept level, and then a module to procedurally synthesise an animation of the resulting utterance in order for it to be played by the 3D avatar.

As a result, there is:

  • an open prototype developed as proof of concept for the architecture of the system
  • the dissertation document with:
    • background overview of Portuguese Sign Language
    • review of existing systems to annotate Sign Languages
    • related work
    • architecture overview and pipeline implementation details
    • list of identified open problems
    • evaluation results and conclusions
  • the extended abstract

Future Work

This work was but the first step towards the bigger goal of having a final product in the hands of those who need it. For those who want to further develop this project, here are some examples:

  • Structural transfer between the Portuguese and Sign Language Portuguese

    Besides isolated words, there is the need to convert all sentences, which is a difficult problem for languages with such a big grammatical gap.
    To further advance the project in this level, there is the imperative need of more linguistic studies of the LGP by linguists and of particular phenomena such as SLs being parallel or how to handle entities.
    This work would involve the mapping of Portuguese grammatical constructs (interrogative, negative, verbs (transitive, intransitive...) and complements..) to the corresponding constructs in LGP.
    This single area can span multiple works, depending on the extent and depth of the focus on the grammatical mapping and the animation synthesis to support it.

  • Gestural names and entities

    Picking up a particularity of SLs, an interesting are of study are gestural names and how to store and reference entities in the grammatical space. Such a work would involve also a study of personal pronouns and other referencing pronouns.

  • Improved animation for gesture types

    As an animation/graphics only work, I suggest studying gestures according to their types (iconic, referencing, classifiers...) and other typical modes, such as fingerspelling to improve the animation synthesis according to each specific case.

  • Building of a community based dictionary

    With very interesting applications, a community based dictionary needs to consider distribution, quality standards and the definition of an human-usable high-level description standard for gestures that maps well to all existing gestures and to the needs of the animation synthesis.

  • Animation playback and usability

    The deployment and use cases still need to be better studied. This involves devising an architecture over the network so that the prototype can ultimately be seen on a web browser or mobile device and the data used in the most efficient way possible. The project needs several data (language models, (community/personal?) dictionaries, avatar, animation). It is an interesting problem of how best to store and use this data, considering real-life use cases (offline usage, low data traffic (do not transfer videos, transfer animation data and run real-time), real-time performance considerations, personal or community avatars and dictionaries that can be edited).

  • Secondary animation

    This can be a big topic, considering a believability layer for the avatar (blinking, breathing, emotions), that can be further integrated in the project by using sentiment analysis, and making the prototype respond nicely to failure cases (eg. the system can not translate words).