Icon Language

Developing icon-based LMs (2018)

Icon language

Icon language is a language employed by speech-language clinicians as a conversational means with people who may have experienced Traumatic Brain injuries, where they struggle to interpret meaning just by the shape of a word. Icons then, are offering a more intuitive solution to communicate with their environment. Icons are also used with the younger population as well, for instance, kids with cerebral palsy may use icons as they may experience difficulty reading.

The back story

As part of working on the brain-computer speller I mentioned in previous projects, we wanted to expand the language symbol sets incorporated into our system. For that we were able to collaborate with a commercial partner on the SymbolStix icon set. The goal, as expected, was to train a language model for immediate integration, but the problem turned out to be that we had no patterns of the icon language. And therefore that became our project, which is how to create a language for a patternless symbol set.

The figure describes an illustration of an icon language model

You can read more on how we created an icon language model in (Dudy & Bedrick, 2018) but you can also watch a 10 minutes talk I gave on it at NW-NLP conference (back in 2018) as well. The repo is below.



  1. block2.png
    Compositional language modeling for icon-based augmentative and alternative communication
    Shiran Dudy, and Steven Bedrick
    In Proceedings of the conference. Association for Computational Linguistics. Meeting, 2018