AirCoding/AirPlan/docs/architecture/branch-deepcode-cli/deepseek-research-survey.md

DeepSeek Research Survey

Date: 2026-05-28 Status: Reference survey for DeepCode CLI branch

Source: GitHub deepseek-ai organization API + README fetches, arXiv references, training data. Note: Some 2025-2026 details may need further verification against latest arXiv publications.

1. DeepSeek Company Overview

DeepSeek is a Chinese AI research company backed by High-Flyer quantitative fund. They have been remarkably prolific, publishing 15+ significant open-source papers and model releases across MoE architecture, reasoning, code, math, vision, multimodal, and infrastructure — all with open-source weights and MIT/Apache 2.0 licenses.

Key architectural innovations:

• Multi-head Latent Attention (MLA): KV cache compression via low-rank latent representation, ~93% cache reduction
• DeepSeekMoE: Fine-grained expert segmentation + shared experts + topology-aware routing
• GRPO (Group Relative Policy Optimization): RL without separate critic model
• DeepSeek Sparse Attention (DSA): Fine-grained sparse attention for long-context efficiency
• Engram: Conditional memory via N-gram embeddings as a new sparsity axis complementary to MoE

2. Complete Model/Repository Inventory

2.1 Foundation Language Models

Repository	Description	Stars	Key Features
DeepSeek-LLM	Base LLM series (7B/67B)	6,954	First foundation model, Llama-style architecture
DeepSeek-V2	MoE with MLA	5,009	236B total/21B activated, MLA, DeepSeekMoE
DeepSeek-V3	Large MoE	103,641	671B total/37B activated, FP8 training, ~$5.5M cost
DeepSeek-V3.1-Exp	Experimental model	1,594	Benchmark improvements over V3
DeepSeek-V3.2-Exp	Experimental sparse attention	1,593	DeepSeek Sparse Attention (DSA), long-context optimization
DeepSeek-V3.2-Exp-Dense	Dense attention variant	1,593 supplement	Matches V3.1-Terminus benchmarks
DeepSeek-V4-Pro	Latest flagship MoE	—	1.6T total/49B activated, CSA+HCA hybrid attention, MIT license, Apr 23 2026
DeepSeek-R1	Reasoning model	92,019	Pure RL reasoning (GRPO), competes with OpenAI o1
DeepSeek-R1-Lite	Lightweight reasoning	—	Lighter R1 variant
DeepSeek-MoE	MoE research	1,934	Fine-grained expert specialization paper

2.2 Code Models

Repository	Description	Stars	Key Features
DeepSeek-Coder	Code LLM (1.3B-33B)	23,509	Code-focused training, Fill-in-Middle
DeepSeek-Coder-V2	MoE code model	6,791	236B MoE, breaks closed-source barrier in code intelligence

2.3 Math and Formal Reasoning

Repository	Description	Stars	Key Features
DeepSeek-Math	Math reasoning (7B)	3,298	GRPO introduced, mathematical reasoning
DeepSeek-Math-V2	Self-verifiable math reasoning	1,589	IMO 2025 gold, Putnam 2024 118/120, verifier-generator loop
DeepSeek-Prover-V1.5	Formal theorem prover	574	Lean 4, RL + MCTS for proof search
DeepSeek-Prover-V2	Advanced formal prover	1,267	Recursive subgoal decomposition, DeepSeek-V3 powered

2.4 Vision and Multimodal

Repository	Description	Stars	Key Features
DeepSeek-VL	Vision-Language	4,116	Real-world vision-language understanding
DeepSeek-VL2	MoE Vision-Language	5,291	MoE-based multimodal understanding
Janus	Unified multimodal	17,733	Decoupled visual encoding for understanding + generation
DeepSeek-OCR	OCR/context compression	23,186	Context optical compression
DeepSeek-OCR-2	Visual Causal Flow	2,889	arXiv: 2601.20552, next-gen OCR

2.5 Training/Inference Infrastructure

Repository	Description	Stars	Key Features
DualPipe	Bidirectional pipeline parallelism	2,955	Computation-communication overlap for V3/R1 training
DeepGEMM	FP8 GEMM kernels	7,308	Clean, efficient FP8 with fine-grained scaling
FlashMLA	Efficient MLA kernels	12,670	Optimized Multi-head Latent Attention GPU kernels
DeepEP	Expert-parallel communication	9,679	Efficient expert-parallel communication library
EPLB	Expert Parallelism Load Balancer	1,380	Production MoE load balancing
LPLB	Linear programming load balancer	505	Research-stage LP-based expert load balancing
TileKernels	Tilelang kernel library	1,559	Kernel library written in tilelang
3FS	Distributed file system	9,931	High-performance FS for AI training/inference
smallpond	Data processing framework	4,961	Lightweight, built on DuckDB and 3FS
profile-data	Overlap analysis	1,158	Analyze computation-communication overlap in V3/R1
open-infra-index	Infrastructure index	8,001	Production-tested AI infrastructure tools

2.6 Research/Training Innovations

Repository	Description	Stars	Key Features
Engram	Conditional memory sparsity	4,427	N-gram lookup as new sparsity axis beyond MoE
ESFT	Expert-Specialized Fine-Tuning	735	EMNLP 2024, tune only task-relevant MoE experts

2.7 Agent/Ecosystem

Repository	Description	Stars
awesome-deepseek-agent	Agent project curation	2,582
awesome-deepseek-integration	API integration examples	37,621
awesome-deepseek-coder	Code project curation	788

3. Key Papers (Chronological)

2024

Paper	arXiv	Key Contribution
DeepSeekMoE	2401.06066	Fine-grained MoE with shared experts
DeepSeek-Coder	2401.14196	Code-focused LLM with Fill-in-Middle
DeepSeekMath	2402.03300	GRPO algorithm, mathematical reasoning
DeepSeek-VL	2403.05525	Real-world vision-language understanding
DeepSeek-V2	2405.04434	Multi-head Latent Attention (MLA) + DeepSeekMoE
DeepSeek-Coder-V2	2406.11931	MoE code model breaking closed-source barrier
ESFT	2407.01906	Expert-Specialized Fine-Tuning for MoE
Fire-Flyer AI-HPC	2408.14158	Cost-effective software-hardware co-design
DeepSeek-Prover-V1.5	2408.08152	RL + MCTS for formal theorem proving
Janus	2410.13848	Decoupled visual encoding for unified multimodal
JanusFlow	2411.05820	Autoregression + Rectified Flow for multimodal
DeepSeek-VL2	2412.10302	MoE vision-language models
DeepSeek-V3	2412.19437	671B MoE, FP8 training, $5.5M cost

2025

Paper	arXiv	Key Contribution
DeepSeek-R1	2501.12948	Pure RL reasoning, GRPO, emergent CoT
Native Sparse Attention	2502.11089	Hardware-aligned trainable sparse attention
DeepSeek-V3.2	2512.02556	DSA + Scalable RL + Agentic Task Synthesis Pipeline
DeepSeek-OCR-2	2601.20552	Visual Causal Flow

2026

Paper	arXiv	Key Contribution
DeepSeek-Prover-V2	—	Recursive subgoal decomposition for Lean 4
DeepSeek-Math-V2	—	Self-verifiable math reasoning, IMO gold
DeepSeek-V4	DeepSeek_V4.pdf (HF)	V4-Pro (1.6T/49B) + V4-Flash (284B/13B), CSA+HCA, mHC, Muon, MIT

Undated / In Repo

Paper	Source	Key Contribution
Engram	Repo	Conditional memory via scalable lookup
DualPipe	Repo	Bidirectional pipeline parallelism

4. Agent Optimization Guide (针对 DeepCode CLI Agent 优化)

4.1 Model Selection by Task

Task	Recommended Model	Reasoning
Code generation (complex)	V4-Pro / V4-Pro-Max	80.6% SWE-bench, 93.5 LiveCodeBench
Code gen (simple/high volume)	V4-Flash	13B active, $0.14/M in, 74% cheaper than GPT-4o
Architecture design / Refactoring	V4-Pro + R1 hybrid	R1 for reasoning plan, V4-Pro for execution
Debugging / Root cause analysis	R1 / V4-Pro reasoning mode	Internal CoT, self-verification, reflection
Code review	V3.2 / V4-Pro	Strong Codeforces + Aider scores
Formal verification	DeepSeek-Prover-V2	Lean 4 theorem proving
Documentation / Explanations	V4-Flash	Cost-effective, fast
Long-context analysis (1M tokens)	V4-Pro	CSA+HCA, 10% KV cache of V3.2
Chat / Interactive	V4-Flash	Low latency, fast TTFT
Math-heavy reasoning	V3.2-Speciale / R1	IMO 2025 gold, Putnam 118/120

4.2 Prompting Strategies for Coding Agents

V4 Pro / Flash (Chat Models — Non-Thinking)

Effective patterns:
- ✅ Use system prompts with explicit persona and constraints
- ✅ Provide XML/fenced output format instructions
- ✅ Use few-shot examples in user messages
- ⚠️ Temperature: 0.3-0.6 (coding = 0.3, creative = 0.6)
- ✅ Structured outputs via JSON schema in system prompt
- ✅ Function calling / tool use natively supported

Example system prompt for coding agent:
  You are an expert software engineer. Write production-ready, well-typed code.
  Follow the project conventions. Use concise explanations.
  Output code blocks with language tags.

R1 / Reasoning Models (Thinking Mode)

Critical differences from chat models:
- ❌ System prompts are WEAK — R1 deprioritizes them during thinking
- ✅ Put ALL instructions in the USER prompt
- ✅ Explicit output format MUST come LAST (after reasoning completes)
- ✅ Temperature: 0.5-0.7 (0.6 recommended)
- ✅ Use <think> tags for internal reasoning naturally
- ⚠️ Vague prompts → excessively long thinking (waste tokens)
- ✅ Precise, well-scoped questions → concise, accurate answers
- ✅ For coding: provide exact file path, language, constraints

Example prompt for R1:
  Write a Python function that [specific task].
  Constraints: [list].
  File: [path/to/file.py].
  Output ONLY valid JSON: {"code": "...", "explanation": "..."}

Hybrid Strategy (Recommended for DeepCode CLI)

Step 1: Problem Analysis (R1)
  - Use R1 to reason about architecture, debug root cause, plan refactoring
  - Let it think freely, capture the plan

Step 2: Code Generation (V4-Pro / V4-Flash)
  - Feed R1's plan as context to V4-Pro with explicit system prompt
  - Generate actual code with tool use

Step 3: Verification (V4-Pro / Prover-V2)
  - Code review, test generation, formal verification

4.3 API Configuration for Agent Use

Parameter	V4-Pro	V4-Flash	R1
Model name	deepseek-v4-pro	deepseek-v4-flash	deepseek-reasoner
Input price /1M	$1.74	$0.14	$0.50 (V3.2 pricing)
Output price /1M	$3.48	$0.28	$2.18
Context window	1,000,000	1,000,000	164K
Max output	384K	384K	8K
Function calling	✓	✓	limited
Streaming	✓	✓	✓
Prompt caching	✓	✓	—
Assistant prefill	✓	✓	—
Vision (PDF/Image)	✓	✓	—

Cost Optimization for Agent Loops

V4-Flash for high-volume agentic tasks:
  - Triage, classification, simple edits: V4-Flash ($0.14/$0.28)
  - Complex reasoning, multi-step: V4-Pro ($1.74/$3.48)
  - Planning/pre-analysis: R1 ($0.50/$2.18)

Prompt caching strategy:
  - Long system prompt + repeated context → cached (cache hit ≈ 10% cost)
  - Prefix-aware: keep consistent prefix for repetitive agent tasks
  - 90% cache hit rate → effective cost ~$0.17/M for V4-Pro

Typical agent cost estimates:
  - V4-Flash per agent step: ~$0.001-0.005 (assuming 2K in/1K out)
  - V4-Pro per agent step: ~$0.01-0.05
  - 100-step agentic workflow with V4-Flash: ~$0.10-0.50

4.4 Context Window Strategy for Agents

V4-Pro 1M Context: Tiered Access

Sliding window (128 tokens):             Immediate local context (always dense)
CSA (4x compression, top-1024):         Project-level context (selected)
HCA (128x compression, dense):          Full repository context (compressed)

For DeepCode CLI:
  - Active file: sliding window (always precise)
  - Related files (imports, types): CSA tier (selective)
  - Full project map: HCA tier (compressed summary)
  - Git history, conventions, README: HCA tier

Context Budget Allocation

System prompt / Agent rules:    4K   (stable, cached)
Project conventions / AGENTS.md:  4K   (stable, cached)
Current file context:           16K   (sliding window)
Related files:                  32K   (CSA-selected)
Open files / tabs:              ~8K
Tool call history:              16K   (last N interactions)
Agent scratchpad:               4K
Total:                          ~84K  (well within 1M limit)

4.5 Function Calling / Tool Use

V4-Pro natively supports function calling and structured outputs.

For DeepCode CLI tools (read, edit, grep, bash, etc.):

Recommended: native OpenAI-compatible function calling
  - Define tools as JSON schema in API call
  - V4-Pro routes to appropriate experts for tool selection
  - Parallel function calling supported

Fallback: structured output mode (JSON schema)
  - Force model to output valid JSON matching a schema
  - Useful for orchestrator pattern
  - Combine with assistant prefill for guided output start

4.6 Evaluation Benchmarks Relevant to Coding Agents

Benchmark	V4-Pro-Max	V4-Pro-High	V4-Flash-Max	V3.2	R1
SWE-bench Verified	80.6%	79.4%	79.0%	~50%	—
LiveCodeBench	93.5	89.8	91.6	74.1	—
Codeforces	3206	2919	3052	2121	—
Aider-Polyglot	~80	~76	~68	74.5	—
HLE (Pass@1)	37.7	34.5	34.8	—	—
GPQA Diamond	90.1	89.1	88.1	79.9	71.5
MMLU-Pro	87.5	87.1	86.2	85.0	84.0
SimpleQA-Verified	57.9	46.2	34.1	—	—
Terminal Bench 2.0	67.9	63.3	56.9	—	—
SWE-Pro	55.4	54.4	52.6	—	—
MRCR 1M	83.5	83.3	78.7	—	—
CorpusQA 1M	62.0	56.5	60.5	—	—
BrowseComp	83.4	80.4	73.2	40.1	—
MCPAtlas Public	73.6	74.2	69.0	—	—
Toolathlon (Pass@1)	51.8	49.0	47.8	—	—

*V4-Pro matches Claude Opus 4.6 on SWE-bench (80.6 vs 80.8) at 2% of the cost.

4.7 Key Takeaways for Agent Optimization

1. V4-Pro is the best coding model as of May 2026 — $3.48/M output is unmatched value
2. V4-Flash for agentic loops — $0.14/$0.28 enables cheap multi-step workflows
3. R1 for planning/reasoning — but feed its output to V4-Pro for code generation
4. 1M context is real — with CSA+HCA tiered access, full-repo context is practical
5. OpenAI-compatible API — drop-in replacement for existing agent frameworks
6. MIT license — no restrictions on commercial use or fine-tuning
7. No system prompt for R1 — all instructions in user message
8. Temperature 0.3 for V4 coding, 0.6 for R1 reasoning

---

5. Key Technical Innovations Deep Dive

4.1 Multi-head Latent Attention (MLA)

Introduced in DeepSeek-V2. Solves KV cache memory bottleneck.

Standard MHA: cache full K and V matrices per layer per token
MLA: compress KV to low-rank latent vector c_kv, reconstruct K/V on-the-fly

c_kv = W_DKV · h_t          (compress to latent, dimension d_c << n_h · d_h)
k = W_UK · c_kv              (upsample K)
v = W_UV · c_kv              (upsample V)

Result: ~93% KV cache reduction, comparable or better performance

4.2 DeepSeekMoE

Introduced in DeepSeek-MoE, refined in V2/V3.

Standard MoE: few large experts, top-k routing
DeepSeekMoE:
  - Fine-grained segmentation: many smaller experts (e.g., 256)
  - Shared experts: always activated, capture common knowledge
  - Routed experts: specialized knowledge per token
  - Auxiliary-loss-free load balancing (V3)

4.3 GRPO (Group Relative Policy Optimization)

Introduced in DeepSeek-R1. Eliminates need for separate critic model.

Standard PPO: policy model + critic/value model
GRPO:
  - Sample group of responses per prompt
  - Compute rewards for each response
  - Group mean as baseline
  - Advantages = (reward - group_mean) / group_std
  - No separate critic model needed
  - Significantly lower memory/computation cost

Key result: R1-Zero trained with pure RL (no SFT) developed emergent reasoning behaviors — chain-of-thought, self-verification, reflection — and spontaneously learned to allocate more "thinking time" to harder problems.

4.4 DeepSeek Sparse Attention (DSA)

Introduced in DeepSeek-V3.2-Exp. Fine-grained sparse attention for long-context.

Dense attention: O(n²) for all token pairs
DSA: fine-grained sparse pattern maintaining output quality
  - First fine-grained sparse attention achieving parity with dense
  - Substantial long-context training/inference efficiency improvement
  - V3.2-Exp matches V3.1-Terminus on public benchmarks

5.5 DeepSeek V4-Pro / V4-Flash (Hybrid CSA + HCA Attention)

Released April 24, 2026. Full technical report: DeepSeek_V4.pdf (43 pages, 4.3MB, from HuggingFace).

V4-Pro:
  - Total parameters: 1.6T
  - Activated per token: 49B
  - Layers: 61 transformer layers, hidden dim: 7168
  - MoE: 1 shared expert + 384 routed experts, 6 active per token (Hash routing for first 3 layers)
  - Attention: interleaved CSA+HCA (first 2 layers HCA only)
  - CSA: m=4 compression, top-k=1024, n_h^I=64 indexer heads, c_I=128, n_h=128 query heads, c=512 head dim, d_c=1536
  - HCA: m'=128 compression
  - SWA: n_win=128 sliding window, grouped output: g=16, d_g=1024
  - mHC: n_hc=4 expansion, Sinkhorn-Knopp 20 iterations
  - Context: 1,000,000 tokens
  - Training: 33T tokens, Muon (μ=0.95, wd=0.1, γ=0.18), AdamW for embed/prediction head/RMSNorm
  - LR: 2.0e-4 → 2.0e-5, max batch: 94.4M, 4K→16K→64K→1M sequence ramp
  - Precision: FP4 (routed expert weights + indexer QK path) + FP8 (other weights)
  - Download size: ~865 GB, License: MIT
  - Optimizer: Muon (hybrid Newton-Schulz: 8 fast + 2 stable iterations)

V4-Flash:
  - Total parameters: 284B
  - Activated per token: 13B
  - Layers: 43 transformer layers, hidden dim: 4096
  - MoE: 1 shared expert + 256 routed experts, 6 active per token (Hash routing for first 3 layers)
  - Attention: interleaved CSA+HCA (first 2 layers pure SWA)
  - CSA: m=4 compression, top-k=512, n_h^I=64, c_I=128, n_h=64, c=512, d_c=1024
  - HCA: m'=128 compression
  - SWA: n_win=128, grouped output: g=8, d_g=1024
  - mHC: n_hc=4 expansion, Sinkhorn-Knopp 20 iterations
  - Context: 1,000,000 tokens
  - Training: 32T tokens, Muon (μ=0.95, wd=0.1, γ=0.18)
  - LR: 2.7e-4 → 2.7e-5, max batch: 75.5M
  - Download size: ~160 GB, License: MIT

Key architectural innovations (paper-grounded):

1. Compressed Sparse Attention (CSA):
   - Dual-stream KV compression (C^a, C^b with overlapping windows), m=4
   - Lightning Indexer: low-rank queries (W^DQ + W^IUQ), ReLU scoring, per-head weights
   - Shared KV MQA: compressed entries serve as both K and V
   - Grouped Output Projection: g groups reduce n_h·c → d overhead

2. Heavily Compressed Attention (HCA):
   - Single-stream compression, m'=128 (no overlap)
   - Dense attention on heavily compressed representation
   - No sparse selection — pure global summary

3. Combined CSA + HCA at 1M context:
   - V4-Pro: 27% FLOPs, 10% KV cache of V3.2
   - V4-Flash: 10% FLOPs, 7% KV cache of V3.2
   - vs BF16 GQA8 baseline: ~2% KV cache
   - Mixed KV precision: BF16 (RoPE 64 dims) + FP8 (remaining)

4. Manifold-Constrained Hyper-Connections (mHC):
   - Residual mapping B_l constrained to doubly stochastic manifold (Birkhoff polytope)
   - Sinkhorn-Knopp projection (20 iters, exp→row norm→col norm)
   - Dynamic parameterization: input-dependent + static components
   - A_l, C_l bounded via Sigmoid, ∥B_l∥₂ ≤ 1 ensures stability
   - Engineering overhead: only 6.7% of 1F1B pipeline stage

5. Muon Optimizer:
   - Hybrid Newton-Schulz: 8 steps (3.4445, -4.7750, 2.0315) + 2 steps (2, -1.5, 0.5)
   - Nesterov momentum (μ=0.95), BF16 Newton-Schulz stable
   - Mixed ZeRO: knapsack for dense params, per-expert for MoE
   - MoE gradients: stochastic rounding to BF16, all-to-all + FP32 local sum

Infrastructure innovations:

6. MegaMoE (Fused EP Kernel):
   - Wave-based expert scheduling (1.5-1.96× speedup)
   - Computation-communication ratio: C/B ≤ 6144 FLOPs/Byte hides communication
   - Open-sourced in DeepGEMM

7. TileLang DSL:
   - Host Codegen: Python checks → C++ (<1μs per invocation)
   - Z3 SMT solver for formal integer analysis
   - IEEE-754 + bitwise reproducibility

8. Batch-Invariant & Deterministic Kernels:
   - Dual-kernel attention (single SM + multi SM for wave-quantization)
   - DeepGEMM replaces cuBLAS end-to-end
   - Deterministic MoE backward: token order preprocessing + buffer isolation

9. KV Cache Management:
   - Heterogeneous: State Cache (SWA + uncompressed tail) + Classical Cache (CSA/HCA blocks)
   - Block size: lcm(4, 128) = 128 original tokens per block
   - On-disk storage: Full SWA / Periodic Checkpointing / Zero SWA Caching

Post-training innovations:

10. FP4 Quantization-Aware Training:
    - MoE expert weights: FP32 master→FP4→FP8 (lossless dequant)
    - Indexer QK path: full FP4
    - Index scores: FP32→BF16 (2× speedup, 99.7% recall)

11. On-Policy Distillation (OPD):
    - 10+ teacher models → single student
    - Full-vocabulary reverse KL (D_KL(π_θ ∥ π_Ei))
    - Teachers offloaded to distributed storage, ZeRO-like sharding
    - Cached last-layer hidden states → on-the-fly logits reconstruction

12. Quick Instruction:
    - Special tokens (<|action|>, <|query|>, <|domain|>, etc.)
    - Reuses existing KV cache, zero redundant prefill
    - Parallel execution of auxiliary tasks

13. Interleaved Thinking:
    - Tool-calling: full reasoning history preserved across rounds
    - General conversation: previous reasoning discarded per turn

14. DSec Sandbox (Rust):
    - 4 substrates: Function Call / Container / microVM (Firecracker) / fullVM (QEMU)
    - 3FS-backed layered storage, preemption-safe trajectory logging
    - Hundreds of thousands of concurrent instances

Training stability:

15. Anticipatory Routing:
    - Step t uses θ_t for features, θ_{t-Δt} routing indices
    - Auto-detection: loss spike → short rollback → activate → revert
    - ~20% overhead when active (negligible overall)

16. SwiGLU Clamping:
    - Linear component: clamp [-10, 10]
    - Gate component: upper bound 10

V4-Pro-Max benchmarks (selected):
  - SimpleQA: 57.9% (vs Gemini 3.1 Pro 75.6%)
  - HLE: 37.7% (vs Opus 4.6 40.0%)
  - LiveCodeBench: 93.5 (highest, vs GPT-5.4 91.7%)
  - Codeforces: 3206 (23rd human, vs GPT-5.4 3168)
  - SWE-Verified: 80.6% (vs Opus 4.6 80.8%)
  - TerminalBench 2.0: 67.9% (vs GPT-5.4 75.1%)
  - MRCR 1M: 83.5 MMR (vs Opus 4.6 92.9%)
  - CorpusQA 1M: 62.0% (vs Opus 4.6 71.7%)
  - Formal Putnam 2025: 120/120

Pricing:
  - V4-Pro: $1.74/M in, $3.48/M out
  - V4-Flash: $0.14/M in, $0.28/M out
  - V4-Pro (Max reasoning): $3.48/M in, $6.96/M out
  - vs GPT-5.5: ~$0.55-0.75/M in, ~$2-3/M out (~25-30x cheaper)
  - vs Claude Opus 4.6: ~$15/M in, ~$75/M out (~20x cheaper)

5.6 DeepSeek V3.2: DSA + Scalable RL + Agent Pipeline

arXiv: 2512.02556 (Dec 2, 2025). Not an experimental — full V3.2 release.

Three key innovations:

1. DeepSeek Sparse Attention (DSA):
   - Lightning indexer: small, multi-head scorer (FP8 efficient)
   - Fine-grained token selection: top-k KV retrieval per query
   - Implemented under MLA framework
   - Dense warm-up (1K steps, 2.1B tokens) trains indexer via KL-divergence
   - Full sparse training adapts all params to sparse pattern
   - Result: first fine-grained sparse attention matching dense quality

2. Scalable RL Framework:
   - Post-training compute scaled substantially (larger than typical)
   - V3.2-Speciale: higher compute variant
   - Achieves IMO 2025 gold + IOI gold
   - Matches GPT-5 on reasoning, Gemini-3.0-Pro on complex reasoning

3. Agentic Task Synthesis Pipeline:
   - Novel pipeline for generating training data for tool-use scenarios
   - Scalable agentic post-training
   - Improved generalization and instruction-following in interactive envs

Benchmarks:
  - Codeforces: 2121 (V3.2-Exp), higher in full V3.2
  - LiveCodeBench: 74.1 (Exp)
  - Aider-Polyglot: 74.5 (Exp)
  - MMLU-Pro: 85.0
  - GPQA-Diamond: 79.9
  - AIME 2025: 89.3
  - SWE-bench Verified: ~50 (via V3.2)
  - BrowseComp: 40.1 / BrowseComp-zh: 47.9

5.7 Engram (Conditional Memory)

Introduced as a new sparsity axis complementary to MoE.

MoE: scales capacity via conditional computation (routing)
Engram: scales knowledge via conditional memory (lookup)

Key idea: N-gram embedding lookup as O(1) knowledge primitive
  - Deterministic addressing
  - Offloadable to host memory with minimal inference overhead
  - Relieves early layers from static pattern reconstruction
  - U-shaped scaling law for optimal MoE vs Engram allocation
  - Engram-27B improves over MoE baselines on knowledge/reasoning/code/math

6. DeepSeek Model Evolution Timeline

DeepSeek-LLM (base)
  ↓
DeepSeekMoE (fine-grained experts)
  ↓
DeepSeek-V2 (MLA + DeepSeekMoE, 236B)
  ↓
DeepSeek-Coder / Coder-V2 (code focus)
  ↓
DeepSeek-Math / Math-V2 (GRPO, IMO gold)
  ↓
DeepSeek-V3 (671B MoE, FP8, $5.5M, 128K context)
  ↓
DeepSeek-R1 / R1-Zero (pure RL reasoning, emergent CoT)
  ↓
DeepSeek-VL / VL2 / Janus (multimodal)
  ↓
DeepSeek-Prover-V1.5 / V2 (Lean 4 formal reasoning)
  ↓
DeepSeek-OCR / OCR-2 (visual understanding)
  ↓
DeepSeek-V3.1-Terminus (128K context extension)
  ↓
DeepSeek-V3.2-Exp / V3.2 (DSA sparse attention + agent pipeline)
  ↓
Engram (conditional memory sparsity)
  ↓
DeepSeek-V4-Pro / V4-Flash (CSA+HCA hybrid attention, mHC, Muon, 1M context)

The V4 series (Apr 2026) represents the current frontier — CSA+HCA attention enabling practical 1M-token context at 27% FLOPs and 10% KV cache of V3.2.

7. Infrastructure Innovation Timeline

Fire-Flyer AI-HPC (hardware-software co-design)
  ↓
FlashMLA (efficient MLA GPU kernels)
  ↓
DeepGEMM (FP8 GEMM with fine-grained scaling)
  ↓
DeepEP (expert-parallel communication)
  ↓
DualPipe (bidirectional pipeline parallelism)
  ↓
EPLB / LPLB (expert load balancing)
  ↓
3FS (distributed file system for AI)
  ↓
smallpond (data processing on DuckDB + 3FS)
  ↓
TileKernels / TileLang (DSL kernel library with Host Codegen + Z3 SMT)
  ↓
MegaMoE (fused EP kernel with wave scheduling, 1.5-1.96× speedup)
  ↓
DSec Sandbox (Firecracker microVM + 3FS, 100K+ concurrent agent sandboxes)

8. Relevance to DeepCode CLI

DeepSeek's research is directly relevant to DeepCode CLI architecture:

8.1 Code Intelligence

• DeepSeek-Coder/Coder-V2: Fill-in-Middle, long-context code understanding
• DeepSeek-V3.2-Exp: Codeforces 2121, LiveCodeBench 74.1, Aider-Polyglot 74.5
• SWE-bench Verified 67.8, SWE-bench Multilingual 57.9

8.2 Reasoning for Debugging/Architecture

• DeepSeek-R1: emergent chain-of-thought, self-verification, reflection
• GRPO: efficient RL without critic model
• Applicable to training code-review/debug agents

8.3 Formal Verification

• DeepSeek-Prover-V2: Lean 4 theorem proving
• Applicable to formal verification of generated code

8.4 Efficient Inference

• MLA: 93% KV cache reduction for long code contexts
• DSA: efficient long-context processing
• MoE: 37B activated from 671B total parameters
• Engram: O(1) knowledge lookup

8.5 Multimodal Evidence

• DeepSeek-OCR/OCR-2: visual understanding for GUI evidence
• Janus: unified multimodal for screenshot/code understanding

8.6 Infrastructure

• 3FS: distributed file system for large codebase training
• DualPipe: efficient distributed training
• DeepGEMM/FlashMLA: optimized inference kernels

9. Total GitHub Stats Summary

Category	Repos	Total Stars
Foundation Models	6	~209,000
Code Models	2	~30,000
Math/Proving	4	~6,700
Vision/Multimodal	5	~53,000
Infrastructure	11	~50,000
Research	2	~5,200
Ecosystem	3	~41,000
Total	36	~395,000