On Tuesday, the Indian AI lab Sarvam introduced a groundbreaking series of large language models, aiming to capture a portion of the market currently dominated by pricier systems from major U.S. and Chinese competitors.
This announcement took place during the India AI Impact Summit in New Delhi, coinciding with the government's initiative to lessen dependency on foreign AI platforms and adapt models for local languages and contexts.
The new offerings consist of models with 30 billion and 105 billion parameters, alongside a text-to-speech model, a speech-to-text model, and a document-processing vision model. This represents a significant leap from Sarvam's previous 2-billion-parameter model launched in October 2024.
According to Sarvam, the latest models utilize a mixture-of-experts architecture, activating only a subset of their parameters at any given time, which effectively lowers computational expenses. The 30B model features a 32,000-token context window tailored for real-time conversations, while the larger model accommodates a 128,000-token window for intricate, multi-step reasoning tasks.
These models were developed from the ground up, not merely fine-tuned from existing open-source frameworks. The 30B model was pre-trained on approximately 16 trillion tokens of text, while the 105B model was trained on trillions of tokens across various Indian languages.
Designed for real-time applications, these models will support voice assistants and chat systems in local languages, enhancing user interaction.
Utilizing resources from the IndiaAI Mission, the models benefited from infrastructure support provided by data center operator Yotta, with technical assistance from Nvidia.
Sarvam's executives highlighted a strategic approach to scaling their models, emphasizing the importance of focusing on real-world applications rather than merely increasing size. Co-founder Pratyush Kumar stated, "We want to be mindful in how we do the scaling. We don't want to do the scaling mindlessly. We want to understand the tasks which really matter at scale and go and build for them."
Looking ahead, Sarvam intends to open-source the 30B and 105B models, although it remains unclear if the training data or complete training code will be publicly available.
The company also has plans to develop specialized AI systems, including coding-centric models and enterprise tools under a product named Sarvam for Work, as well as a conversational AI agent platform called Samvaad.
Founded in 2023, Sarvam has successfully raised over $50 million in funding, with notable investors including Lightspeed Venture Partners, Khosla Ventures, and Peak XV Partners (formerly Sequoia Capital India).