Words and worlds: Shaping inclusive AI for every language

In a rapidly evolving global landscape, the role of Artificial Intelligence (AI) is expanding across diverse linguistic and cultural contexts. The latest Large Language Models (LLMs) can give passable answers to questions on everything from nuclear physics to Stoic philosophy, but mainly in English. This is because LLMs are primarily trained on data scraped from the internet, where English predominates.

The Conundrum of Data: Structured vs. Unstructured

AI thrives on data, which comes in two main flavors: structured and unstructured. Structured data, neatly organized and often numerical, is easily ingested by traditional, narrow AI systems designed for specific tasks- imagine a chess-playing AI that knows every possible move. However, life is seldom so neatly arranged, which is where generative AI comes into play. Generative AI excels in handling the messy, unstructured data of everyday life—text, images, and sounds—much like navigating a bustling marketplace full of diverse voices and activities.

At the recent World Bank Global Digital Summit, experts emphasized the potential of generative AI to mimic human-like behavior, underscoring the crucial need for technologies, particularly in developing regions. However, the development of these technologies is often uneven and imbalanced: it typically favors high-income economies and remains concentrated within a few large technology corporations.

Despite the linguistic diversity in Africa, where over 2000 languages are spoken, popular virtual assistants like Siri, Alexa, and Google Assistant are yet to support any native African language.

Moreover, AI model development in languages with non-alphabetic scripts, such as Arabic or Devanagari, which have more complicated structures, is also more expensive.

Voices from the Ground: Emerging Market Innovations

In emerging economies, non-text data, such as voice, becomes a pivotal element. Initiatives such as Mozilla's Common Voice invite global participation to contribute voice data in various languages, helping to build voice recognition systems that reflect the world’s true linguistic diversity. This open-source initiative democratizes AI development, allowing for broader participation and fostering AI that understands accents and dialects from Jakarta to Johannesburg.

Another illustration is Bharat GPT,  a powerful example of the nuanced application of AI in multilingual contexts like India. Supporting over 14 Indian languages, this LLM taps into the country's rich linguistic tapestry, providing access across video, voice, and text mediums. It has successfully engaged users nationwide by catering to multiple local languages, spurring broader AI adoption in diverse settings from urban to rural areas. Despite its successes, the quality of language generation and the inclusion of more dialects remain challenging. Nevertheless, Bharat GPT represents a significant stride toward creating AI that resonates with local cultural and linguistic nuances.

With this in mind, we have three key takeaways:

1. Collecting Diverse Datasets: Collecting datasets that embrace the richness of language—including dialects, idioms, and cultural nuances—is essential for ensuring equitable access and meaningful adoption. This diversity in data helps AI serve not just as a technological tool but as a bridge across cultural divides. Local ecosystems, including community-driven projects and regional tech hubs, play a crucial role in driving this progress.
2. Bridging the Digital Divide: To ensure AI's effectiveness, citizens should be able to access tools in multiple ways. This ensures that everyone, regardless of their level of digital access or literacy, can benefit. This strategy broadens access and fosters deeper, more meaningful engagement with technology across different segments of society.
3. Prioritizing Local Needs and Contexts: While global solutions provide valuable frameworks, AI applications must fit specific community needs and contexts to ensure their effectiveness and sustainability. Training AI systems in local languages, incorporating region-specific knowledge into AI models, and adapting interfaces to local customs are crucial steps. By prioritizing these local dimensions, AI becomes a more powerful tool for empowerment and development.

Looking to the future

As AI technologies evolve, their potential to adjust and shape cultural aspects of human interaction grows. The challenge, however, is ensuring these technologies are developed inclusively, respecting the linguistic and cultural diversity of users worldwide. This requires a concerted effort to meaningfully collect and leverage data, embracing the richness of languages, dialects, and cultural nuances. At the World Bank, we are working with governments to support the entire AI value chain, from the infrastructure to the data and the protocols used to govern them.

The digital divide is more than a technological gap - it's a linguistic and cultural chasm. As we stand on the brink of a generative AI revolution, we must ask: Are we creating technologies that understand us all, or are we coding a new tower of Babel? The future of AI should not be about who it excludes but who it embraces, using the diversity of our languages and the inclusivity of our technologies as the foundation for a truly global AI landscape.