Building foundational artificial intelligence architectures from scratch requires reimagining every layer of the compute stack.
With Falin-300M, Cortiqa set out to demonstrate a sovereign proof-of-concept: that efficient, high-performance language models can be conceived, structured, and trained entirely natively in India.
Why Small Language Models (SLMs)?
While frontier models exceed hundreds of billions of parameters, real-world deployment on edge devices, local servers, and browser extensions demands compact footprints with near-zero latency.
A 297-Million parameter model can be deployed on standard consumer GPUs, mobile CPUs, and embedded hardware without demanding massive cloud infrastructure or high thermal budgets.
Key Architectural Innovations
Falin-300M is built upon Cortiqa's proprietary Menothus architecture:
- Extreme Grouped Query Attention (GQA): Utilizing a 16:2 head ratio (8:1), saving 75% of KV-cache VRAM during inference.
- Parallel Attention + SwiGLU FFN: Executing attention mechanisms concurrently with feed-forward computations to maximize GPU throughput.
- Hybrid Sliding Window Attention (SWA): Balancing local context (512 tokens) with dense global causal attention every 4th layer.
- Native FlashAttention / SDPA: Clean PyTorch scaled dot-product attention without fragile third-party compilation layers.
Open Research Preview
Falin-300M is available on Hugging Face for academic evaluation, non-commercial research, and domain fine-tuning.