For nearly a decade, the architecture of large language models (LLMs) has stayed mostly the same. It remains anchored by the residual connection paradigm introduced in 2015. While the industry has used brute-force compute to scale these models to trillions of parameters, researchers at DeepSeek-AI have taken a different path. Led by founder Liang Wenfeng, the team introduced two foundational innovations called mHC (Manifold-Constrained Hyper-Connections) and Engram. These tools challenge long-standing standards. They aim to achieve “aggressive parameter expansion” through architectural efficiency instead of raw power.
mHC: Restoring Stability to the “Information Highway”
Modern AI models process data through a residual stream. This is often viewed as a highway that carries information through hundreds of neural network layers. Traditional designs enforce a rigid 1:1 ratio between input and new computation. This prevents the model from routing information dynamically. Earlier attempts like ByteDance’s Hyper-Connections (HC) tried to expand this “highway” into multiple parallel lanes for better processing, but they were notoriously unstable. Without constraints, signals were amplified by up to 3,000 times in deep networks. This led to catastrophic training failures known as exploding gradients.
DeepSeek’s mHC solves this instability by applying mathematical “guardrails” through Manifold Constraints. The system projects learnable matrices onto the Birkhoff polytope. This is a manifold of double stochastic matrices where rows and columns always sum to one. It ensures that signals are never amplified beyond a manageable magnitude. In empirical tests, mHC dropped the maximum signal gain magnitude from 3,000 down to approximately 1.6. This represents a three-order-of-magnitude improvement in stability.
This innovation allows researchers to scale model capacity significantly without making the training process fragile. Despite its mathematical complexity, mHC is highly efficient. Through kernel fusion and infrastructure-aware optimizations, it adds only a marginal 6.7% overhead to training time. This effectively enables models to handle pathways up to 8x broader than standard configurations while maintaining performance on complex reasoning tasks.
Engram: The Rise of Conditional Memory
While mHC optimizes the “wiring” of an AI, Engram completely changes its “memory.” Current Transformers lack a native primitive for knowledge lookup. This forces them to “inefficiently simulate retrieval through computation.” When a model needs to recall a static fact, such as the details of “Alexander the Great,” it must use its early attention and feed-forward layers to reconstruct that knowledge. This wastes valuable computational depth that could be used for complex reasoning.
Engram introduces a conditional memory module that modernizes classic N-gram embeddings for O(1) constant-time lookup. By delegating local, stereotyped patterns to this lookup table, Engram relieves the model’s “backbone” from static reconstruction. Mechanistic analysis reveals that this effectively “deepens” the network. It makes shallow layers functionally equivalent to much deeper layers in traditional models.
One of the most significant breakthroughs of Engram is its discovery of a U-shaped scaling law for sparsity allocation. Researchers found that the optimal balance for a fixed parameter budget is to allocate approximately 20% to 25% of the sparse parameters to Engram memory. The rest remains in Mixture-of-Experts (MoE) computation. This hybrid approach consistently outperforms pure MoE baselines.
What This Means for the Future of AI Intelligence
The integration of mHC and Engram represents a shift toward “intelligence density.” Rather than simply throwing more compute at a model, these innovations allow models to get “smarter” with the same number of parameters by making every parameter work more efficiently.
The application of these technologies has shown dramatic results across various benchmarks. While Engram-powered models show expected gains in knowledge-intensive tasks like MMLU (+3.4), they demonstrate even larger improvements in general reasoning (+5.0 on BBH) and mathematics (+2.4 on MATH). Furthermore, by freeing up attention capacity, Engram substantially boosts long-context retrieval. It increased accuracy on “Needle In A Haystack” benchmarks from 84.2% to 97.0%.
For end-users, this means future AI models could possess significantly better reasoning and recall capabilities while remaining fast. Additionally, because Engram lookups are deterministic and rely on the input token sequence rather than dynamic hidden states, they enable “infrastructure-aware” efficiency. Models can asynchronously prefetch embeddings from cheap host RAM (DRAM) into the GPU. This effectively bypasses the expensive GPU memory wall. Eventually, this could allow massive models of potentially 400B-500B parameters to run on consumer hardware by offloading memory requirements to system RAM or even NVMe storage.
The State of Open Source AI and the Global Race
DeepSeek’s release of mHC and Engram signals a fascinating split in the AI landscape. Many Western commercial labs are focusing on AI agents and increasing compute spend. This is exemplified by billion-dollar training runs. Meanwhile, DeepSeek and other labs like Moonshot (Kimi) are digging into macro-architecture and optimization.
This strategy is partly a response to geopolitical constraints. Amid U.S. chip export restrictions that limit access to high-end hardware like Nvidia H100s, Chinese labs are prioritizing algorithmic moats over hardware scale. By achieving parity with models like OpenAI’s o1 using only a fraction of the compute, DeepSeek is proving that architectural ingenuity can bypass hardware bottlenecks.
Crucially, DeepSeek has continued its pattern of open-sourcing these findings for the “advancement of humanity.” This open-weight and open-research policy has fueled massive adoption, with millions of downloads on platforms like Hugging Face. Community members on platforms like Reddit have noted that this spirit of “relentless self-doubt and fundamental reinvention” is exactly how AI will evolve. It creates the potential for “democratizing” frontier-level performance for non-hyperscale players. As 2026 unfolds, these innovations position open-source AI not just as a follower of proprietary models, but as a technical pace-setter in sustainable and efficient scaling.
Disclaimer:
All views expressed are my own and are provided solely for informational and educational purposes. This is not investment, legal, tax, or accounting advice, nor a recommendation to buy or sell any security. While I aim for accuracy, I cannot guarantee completeness or timeliness of information. The strategies and securities discussed may not suit every investor; past performance does not predict future results, and all investments carry risk, including loss of principal.
I may hold, or have held, positions in any mentioned securities. Opinions herein are subject to change without notice. This material reflects my personal views and does not represent those of any employer or affiliated organization. Please conduct your own research and consult a licensed professional before making any investment decisions.
Excellent analysis! Could you elaborate on how Birkhoff polytope ensures stabilty?