Building PictoMaker: an AI-powered predictive pictogram keyboard

Communication is a fundamental human right, yet not everyone can rely on spoken or written language to express themselves.

Augmentative and Alternative Communication (AAC) systems provide tools—such as pictogram boards and applications—that enable people with motor disabilities, autism, or speech impairments to communicate effectively.

However, many existing AAC systems are largely static: users must manually navigate categories or search for pictograms, which can slow down interaction and increase cognitive load.

With PictoMaker, my goal was to go one step further: to build a predictive pictogram keyboard capable of anticipating the next word and automatically suggesting the corresponding pictograms, making communication faster and more fluid.

#Challenge 1: Predicting the next word

The first challenge closely resembles traditional text prediction, but with an important constraint: the model must perform well on short, simple sentences and across highly diverse contexts, as is common in AAC usage.

To explore this, I experimented with Transformer-based models, evaluating two main approaches:

• Encoder models such as BERT, which capture bidirectional context within a sentence
• Decoder models such as GPT-2, which are better suited for predictive text generation

The objective was that, given a partial input like:

“I want to go to the …”

the model could suggest likely continuations such as “bathroom”, “park”, or “school”, depending on context.

📓 You can explore the notebook where this part is implemented in this Kaggle notebook.

#Challenge 2: Mapping words to pictograms

Once a word is predicted, the next step is to display the correct pictogram.

Here, a linguistic challenge emerges: pictogram repositories are typically indexed by lemmas (canonical word forms).
For example, “running”, “runs”, and “ran” should all map to the pictogram for “run”.

To address this, I used lemmatization with spaCy, which maps inflected word forms to their lemma or canonical form. That lemma is then used to look up the corresponding pictogram.

📓 The notebook covering this process is available in this Kaggle notebook.

#Putting it all together: the prototype

By combining both components, PictoMaker:

1. Predicts the next word the user is likely to want to express
2. Lemmatizes and translates that word into its corresponding pictogram
3. Displays visual suggestions, enabling faster selection and smoother communication

You can explore the application prototype here.

#Limitations and next steps

At this stage, PictoMaker is an experimental prototype, designed to demonstrate the potential of combining AI with AAC systems. Several important challenges remain:

• The language model is trained on general-purpose Spanish, which means it tends to generate grammatically rich and complex structures that are not always aligned with AAC language patterns. AAC users often rely on more direct, action- or object-focused constructions (e.g., “I eat apple” instead of “I want to eat an apple”), making domain-specific adaptation essential.
• The system is not yet optimized for real-world deployment, and currently serves as a proof of concept.

My intention is to continue developing PictoMaker in collaboration with professionals in speech therapy, special education, and accessibility, ensuring that both the models and the interface align with real user needs.

#Final reflection

AAC is an area where artificial intelligence can have a profound social impact. Beyond technical accuracy, the ultimate goal is to empower communication for those who need it most.

With PictoMaker, I aim to show how AI can act as a bridge between natural language and visual communication, making interaction faster, more accessible, and more human.

If you’d like to dive deeper into the implementation, I invite you to explore the linked notebooks above or watch the full talk from PyConES 2025 in Seville.