AirCoding/AirPlan/docs/architecture/branch-deepcode-cli/deepseek-research-survey.md

# DeepSeek Research Survey

Date: 2026-05-28
Status: Reference survey for DeepCode CLI branch

Source: GitHub deepseek-ai organization API + README fetches, arXiv references, training data.
Note: Some 2025-2026 details may need further verification against latest arXiv publications.

## 1. DeepSeek Company Overview

DeepSeek is a Chinese AI research company backed by High-Flyer quantitative fund. They have been remarkably prolific, publishing 15+ significant open-source papers and model releases across MoE architecture, reasoning, code, math, vision, multimodal, and infrastructure — all with open-source weights and MIT/Apache 2.0 licenses.

Key architectural innovations:

- **Multi-head Latent Attention (MLA)**: KV cache compression via low-rank latent representation, ~93% cache reduction
- **DeepSeekMoE**: Fine-grained expert segmentation + shared experts + topology-aware routing
- **GRPO (Group Relative Policy Optimization)**: RL without separate critic model
- **DeepSeek Sparse Attention (DSA)**: Fine-grained sparse attention for long-context efficiency
- **Engram**: Conditional memory via N-gram embeddings as a new sparsity axis complementary to MoE

## 2. Complete Model/Repository Inventory

### 2.1 Foundation Language Models

| Repository | Description | Stars | Key Features |
|---|---|---|---|
| `DeepSeek-LLM` | Base LLM series (7B/67B) | 6,954 | First foundation model, Llama-style architecture |
| `DeepSeek-V2` | MoE with MLA | 5,009 | 236B total/21B activated, MLA, DeepSeekMoE |
| `DeepSeek-V3` | Large MoE | 103,641 | 671B total/37B activated, FP8 training, ~$5.5M cost |
| `DeepSeek-V3.1-Exp` | Experimental model | 1,594 | Benchmark improvements over V3 |
| `DeepSeek-V3.2-Exp` | Experimental sparse attention | 1,593 | DeepSeek Sparse Attention (DSA), long-context optimization |
| `DeepSeek-V3.2-Exp-Dense` | Dense attention variant | 1,593 supplement | Matches V3.1-Terminus benchmarks |
| `DeepSeek-V4-Pro` | Latest flagship MoE | — | 1.6T total/49B activated, CSA+HCA hybrid attention, MIT license, Apr 23 2026 |
| `DeepSeek-R1` | Reasoning model | 92,019 | Pure RL reasoning (GRPO), competes with OpenAI o1 |
| `DeepSeek-R1-Lite` | Lightweight reasoning | — | Lighter R1 variant |
| `DeepSeek-MoE` | MoE research | 1,934 | Fine-grained expert specialization paper |

### 2.2 Code Models

| Repository | Description | Stars | Key Features |
|---|---|---|---|
| `DeepSeek-Coder` | Code LLM (1.3B-33B) | 23,509 | Code-focused training, Fill-in-Middle |
| `DeepSeek-Coder-V2` | MoE code model | 6,791 | 236B MoE, breaks closed-source barrier in code intelligence |

### 2.3 Math and Formal Reasoning

| Repository | Description | Stars | Key Features |
|---|---|---|---|
| `DeepSeek-Math` | Math reasoning (7B) | 3,298 | GRPO introduced, mathematical reasoning |
| `DeepSeek-Math-V2` | Self-verifiable math reasoning | 1,589 | IMO 2025 gold, Putnam 2024 118/120, verifier-generator loop |
| `DeepSeek-Prover-V1.5` | Formal theorem prover | 574 | Lean 4, RL + MCTS for proof search |
| `DeepSeek-Prover-V2` | Advanced formal prover | 1,267 | Recursive subgoal decomposition, DeepSeek-V3 powered |

### 2.4 Vision and Multimodal

| Repository | Description | Stars | Key Features |
|---|---|---|---|
| `DeepSeek-VL` | Vision-Language | 4,116 | Real-world vision-language understanding |
| `DeepSeek-VL2` | MoE Vision-Language | 5,291 | MoE-based multimodal understanding |
| `Janus` | Unified multimodal | 17,733 | Decoupled visual encoding for understanding + generation |
| `DeepSeek-OCR` | OCR/context compression | 23,186 | Context optical compression |
| `DeepSeek-OCR-2` | Visual Causal Flow | 2,889 | arXiv: 2601.20552, next-gen OCR |

### 2.5 Training/Inference Infrastructure

| Repository | Description | Stars | Key Features |
|---|---|---|---|
| `DualPipe` | Bidirectional pipeline parallelism | 2,955 | Computation-communication overlap for V3/R1 training |
| `DeepGEMM` | FP8 GEMM kernels | 7,308 | Clean, efficient FP8 with fine-grained scaling |
| `FlashMLA` | Efficient MLA kernels | 12,670 | Optimized Multi-head Latent Attention GPU kernels |
| `DeepEP` | Expert-parallel communication | 9,679 | Efficient expert-parallel communication library |
| `EPLB` | Expert Parallelism Load Balancer | 1,380 | Production MoE load balancing |
| `LPLB` | Linear programming load balancer | 505 | Research-stage LP-based expert load balancing |
| `TileKernels` | Tilelang kernel library | 1,559 | Kernel library written in tilelang |
| `3FS` | Distributed file system | 9,931 | High-performance FS for AI training/inference |
| `smallpond` | Data processing framework | 4,961 | Lightweight, built on DuckDB and 3FS |
| `profile-data` | Overlap analysis | 1,158 | Analyze computation-communication overlap in V3/R1 |
| `open-infra-index` | Infrastructure index | 8,001 | Production-tested AI infrastructure tools |

### 2.6 Research/Training Innovations

| Repository | Description | Stars | Key Features |
|---|---|---|---|
| `Engram` | Conditional memory sparsity | 4,427 | N-gram lookup as new sparsity axis beyond MoE |
| `ESFT` | Expert-Specialized Fine-Tuning | 735 | EMNLP 2024, tune only task-relevant MoE experts |

### 2.7 Agent/Ecosystem

| Repository | Description | Stars |
|---|---|---|
| `awesome-deepseek-agent` | Agent project curation | 2,582 |
| `awesome-deepseek-integration` | API integration examples | 37,621 |
| `awesome-deepseek-coder` | Code project curation | 788 |

## 3. Key Papers (Chronological)

### 2024

| Paper | arXiv | Key Contribution |
|---|---|---|
| DeepSeekMoE | 2401.06066 | Fine-grained MoE with shared experts |
| DeepSeek-Coder | 2401.14196 | Code-focused LLM with Fill-in-Middle |
| DeepSeekMath | 2402.03300 | GRPO algorithm, mathematical reasoning |
| DeepSeek-VL | 2403.05525 | Real-world vision-language understanding |
| DeepSeek-V2 | 2405.04434 | Multi-head Latent Attention (MLA) + DeepSeekMoE |
| DeepSeek-Coder-V2 | 2406.11931 | MoE code model breaking closed-source barrier |
| ESFT | 2407.01906 | Expert-Specialized Fine-Tuning for MoE |
| Fire-Flyer AI-HPC | 2408.14158 | Cost-effective software-hardware co-design |
| DeepSeek-Prover-V1.5 | 2408.08152 | RL + MCTS for formal theorem proving |
| Janus | 2410.13848 | Decoupled visual encoding for unified multimodal |
| JanusFlow | 2411.05820 | Autoregression + Rectified Flow for multimodal |
| DeepSeek-VL2 | 2412.10302 | MoE vision-language models |
| DeepSeek-V3 | 2412.19437 | 671B MoE, FP8 training, $5.5M cost |

### 2025

| Paper | arXiv | Key Contribution |
|---|---|---|
| DeepSeek-R1 | 2501.12948 | Pure RL reasoning, GRPO, emergent CoT |
| Native Sparse Attention | 2502.11089 | Hardware-aligned trainable sparse attention |
| DeepSeek-V3.2 | 2512.02556 | DSA + Scalable RL + Agentic Task Synthesis Pipeline |
| DeepSeek-OCR-2 | 2601.20552 | Visual Causal Flow |

### 2026

| Paper | arXiv | Key Contribution |
|---|---|---|
| DeepSeek-Prover-V2 | — | Recursive subgoal decomposition for Lean 4 |
| DeepSeek-Math-V2 | — | Self-verifiable math reasoning, IMO gold |
| DeepSeek-V4 | DeepSeek_V4.pdf (HF) | V4-Pro (1.6T/49B) + V4-Flash (284B/13B), CSA+HCA, mHC, Muon, MIT |

### Undated / In Repo

| Paper | Source | Key Contribution |
|---|---|---|
| Engram | Repo | Conditional memory via scalable lookup |
| DualPipe | Repo | Bidirectional pipeline parallelism |

## 4. Agent Optimization Guide (针对 DeepCode CLI Agent 优化)

### 4.1 Model Selection by Task

| Task | Recommended Model | Reasoning |
|---|---|---|
| Code generation (complex) | V4-Pro / V4-Pro-Max | 80.6% SWE-bench, 93.5 LiveCodeBench |
| Code gen (simple/high volume) | V4-Flash | 13B active, $0.14/M in, 74% cheaper than GPT-4o |
| Architecture design / Refactoring | V4-Pro + R1 hybrid | R1 for reasoning plan, V4-Pro for execution |
| Debugging / Root cause analysis | R1 / V4-Pro reasoning mode | Internal CoT, self-verification, reflection |
| Code review | V3.2 / V4-Pro | Strong Codeforces + Aider scores |
| Formal verification | DeepSeek-Prover-V2 | Lean 4 theorem proving |
| Documentation / Explanations | V4-Flash | Cost-effective, fast |
| Long-context analysis (1M tokens) | V4-Pro | CSA+HCA, 10% KV cache of V3.2 |
| Chat / Interactive | V4-Flash | Low latency, fast TTFT |
| Math-heavy reasoning | V3.2-Speciale / R1 | IMO 2025 gold, Putnam 118/120 |

### 4.2 Prompting Strategies for Coding Agents

#### V4 Pro / Flash (Chat Models — Non-Thinking)
```text
Effective patterns:
- ✅ Use system prompts with explicit persona and constraints
- ✅ Provide XML/fenced output format instructions
- ✅ Use few-shot examples in user messages
- ⚠️ Temperature: 0.3-0.6 (coding = 0.3, creative = 0.6)
- ✅ Structured outputs via JSON schema in system prompt
- ✅ Function calling / tool use natively supported

Example system prompt for coding agent:
  You are an expert software engineer. Write production-ready, well-typed code.
  Follow the project conventions. Use concise explanations.
  Output code blocks with language tags.
```

#### R1 / Reasoning Models (Thinking Mode)
```text
Critical differences from chat models:
- ❌ System prompts are WEAK — R1 deprioritizes them during thinking
- ✅ Put ALL instructions in the USER prompt
- ✅ Explicit output format MUST come LAST (after reasoning completes)
- ✅ Temperature: 0.5-0.7 (0.6 recommended)
- ✅ Use <think> tags for internal reasoning naturally
- ⚠️ Vague prompts → excessively long thinking (waste tokens)
- ✅ Precise, well-scoped questions → concise, accurate answers
- ✅ For coding: provide exact file path, language, constraints

Example prompt for R1:
  Write a Python function that [specific task].
  Constraints: [list].
  File: [path/to/file.py].
  Output ONLY valid JSON: {"code": "...", "explanation": "..."}
```

#### Hybrid Strategy (Recommended for DeepCode CLI)
```text
Step 1: Problem Analysis (R1)
  - Use R1 to reason about architecture, debug root cause, plan refactoring
  - Let it think freely, capture the plan

Step 2: Code Generation (V4-Pro / V4-Flash)
  - Feed R1's plan as context to V4-Pro with explicit system prompt
  - Generate actual code with tool use

Step 3: Verification (V4-Pro / Prover-V2)
  - Code review, test generation, formal verification
```

### 4.3 API Configuration for Agent Use

| Parameter | V4-Pro | V4-Flash | R1 |
|---|---|---|---|
| Model name | `deepseek-v4-pro` | `deepseek-v4-flash` | `deepseek-reasoner` |
| Input price /1M | $1.74 | $0.14 | $0.50 (V3.2 pricing) |
| Output price /1M | $3.48 | $0.28 | $2.18 |
| Context window | 1,000,000 | 1,000,000 | 164K |
| Max output | 384K | 384K | 8K |
| Function calling | ✓ | ✓ | limited |
| Streaming | ✓ | ✓ | ✓ |
| Prompt caching | ✓ | ✓ | — |
| Assistant prefill | ✓ | ✓ | — |
| Vision (PDF/Image) | ✓ | ✓ | — |

#### Cost Optimization for Agent Loops
```text
V4-Flash for high-volume agentic tasks:
  - Triage, classification, simple edits: V4-Flash ($0.14/$0.28)
  - Complex reasoning, multi-step: V4-Pro ($1.74/$3.48)
  - Planning/pre-analysis: R1 ($0.50/$2.18)

Prompt caching strategy:
  - Long system prompt + repeated context → cached (cache hit ≈ 10% cost)
  - Prefix-aware: keep consistent prefix for repetitive agent tasks
  - 90% cache hit rate → effective cost ~$0.17/M for V4-Pro

Typical agent cost estimates:
  - V4-Flash per agent step: ~$0.001-0.005 (assuming 2K in/1K out)
  - V4-Pro per agent step: ~$0.01-0.05
  - 100-step agentic workflow with V4-Flash: ~$0.10-0.50
```

### 4.4 Context Window Strategy for Agents

#### V4-Pro 1M Context: Tiered Access
```text
Sliding window (128 tokens):             Immediate local context (always dense)
CSA (4x compression, top-1024):         Project-level context (selected)
HCA (128x compression, dense):          Full repository context (compressed)

For DeepCode CLI:
  - Active file: sliding window (always precise)
  - Related files (imports, types): CSA tier (selective)
  - Full project map: HCA tier (compressed summary)
  - Git history, conventions, README: HCA tier
```

#### Context Budget Allocation
```text
System prompt / Agent rules:    4K   (stable, cached)
Project conventions / AGENTS.md:  4K   (stable, cached)
Current file context:           16K   (sliding window)
Related files:                  32K   (CSA-selected)
Open files / tabs:              ~8K
Tool call history:              16K   (last N interactions)
Agent scratchpad:               4K
Total:                          ~84K  (well within 1M limit)
```

### 4.5 Function Calling / Tool Use

V4-Pro natively supports function calling and structured outputs.

For DeepCode CLI tools (read, edit, grep, bash, etc.):
```text
Recommended: native OpenAI-compatible function calling
  - Define tools as JSON schema in API call
  - V4-Pro routes to appropriate experts for tool selection
  - Parallel function calling supported

Fallback: structured output mode (JSON schema)
  - Force model to output valid JSON matching a schema
  - Useful for orchestrator pattern
  - Combine with assistant prefill for guided output start
```

### 4.6 Evaluation Benchmarks Relevant to Coding Agents

| Benchmark | V4-Pro-Max | V4-Pro-High | V4-Flash-Max | V3.2 | R1 |
|---|---|---|---|---|---|---|
| SWE-bench Verified | **80.6%** | 79.4% | 79.0% | ~50% | — |
| LiveCodeBench | **93.5** | 89.8 | 91.6 | 74.1 | — |
| Codeforces | **3206** | 2919 | 3052 | 2121 | — |
| Aider-Polyglot | **~80** | ~76 | ~68 | 74.5 | — |
| HLE (Pass@1) | 37.7 | 34.5 | 34.8 | — | — |
| GPQA Diamond | 90.1 | 89.1 | 88.1 | 79.9 | 71.5 |
| MMLU-Pro | 87.5 | 87.1 | 86.2 | 85.0 | 84.0 |
| SimpleQA-Verified | **57.9** | 46.2 | 34.1 | — | — |
| Terminal Bench 2.0 | 67.9 | 63.3 | 56.9 | — | — |
| SWE-Pro | 55.4 | 54.4 | 52.6 | — | — |
| MRCR 1M | 83.5 | 83.3 | 78.7 | — | — |
| CorpusQA 1M | 62.0 | 56.5 | 60.5 | — | — |
| BrowseComp | 83.4 | 80.4 | 73.2 | 40.1 | — |
| MCPAtlas Public | 73.6 | 74.2 | 69.0 | — | — |
| Toolathlon (Pass@1) | 51.8 | 49.0 | 47.8 | — | — |

*V4-Pro matches Claude Opus 4.6 on SWE-bench (80.6 vs 80.8) at 2% of the cost.

### 4.7 Key Takeaways for Agent Optimization

1. **V4-Pro is the best coding model as of May 2026** — $3.48/M output is unmatched value
2. **V4-Flash for agentic loops** — $0.14/$0.28 enables cheap multi-step workflows
3. **R1 for planning/reasoning** — but feed its output to V4-Pro for code generation
4. **1M context is real** — with CSA+HCA tiered access, full-repo context is practical
5. **OpenAI-compatible API** — drop-in replacement for existing agent frameworks
6. **MIT license** — no restrictions on commercial use or fine-tuning
7. **No system prompt for R1** — all instructions in user message
8. **Temperature 0.3 for V4 coding, 0.6 for R1 reasoning**

---

## 5. Key Technical Innovations Deep Dive

### 4.1 Multi-head Latent Attention (MLA)

Introduced in DeepSeek-V2. Solves KV cache memory bottleneck.

```text
Standard MHA: cache full K and V matrices per layer per token
MLA: compress KV to low-rank latent vector c_kv, reconstruct K/V on-the-fly

c_kv = W_DKV · h_t          (compress to latent, dimension d_c << n_h · d_h)
k = W_UK · c_kv              (upsample K)
v = W_UV · c_kv              (upsample V)

Result: ~93% KV cache reduction, comparable or better performance
```

### 4.2 DeepSeekMoE

Introduced in DeepSeek-MoE, refined in V2/V3.

```text
Standard MoE: few large experts, top-k routing
DeepSeekMoE:
  - Fine-grained segmentation: many smaller experts (e.g., 256)
  - Shared experts: always activated, capture common knowledge
  - Routed experts: specialized knowledge per token
  - Auxiliary-loss-free load balancing (V3)
```

### 4.3 GRPO (Group Relative Policy Optimization)

Introduced in DeepSeek-R1. Eliminates need for separate critic model.

```text
Standard PPO: policy model + critic/value model
GRPO:
  - Sample group of responses per prompt
  - Compute rewards for each response
  - Group mean as baseline
  - Advantages = (reward - group_mean) / group_std
  - No separate critic model needed
  - Significantly lower memory/computation cost
```

Key result: R1-Zero trained with pure RL (no SFT) developed emergent reasoning behaviors — chain-of-thought, self-verification, reflection — and spontaneously learned to allocate more "thinking time" to harder problems.

### 4.4 DeepSeek Sparse Attention (DSA)

Introduced in DeepSeek-V3.2-Exp. Fine-grained sparse attention for long-context.

```text
Dense attention: O(n²) for all token pairs
DSA: fine-grained sparse pattern maintaining output quality
  - First fine-grained sparse attention achieving parity with dense
  - Substantial long-context training/inference efficiency improvement
  - V3.2-Exp matches V3.1-Terminus on public benchmarks
```

### 5.5 DeepSeek V4-Pro / V4-Flash (Hybrid CSA + HCA Attention)

Released April 24, 2026. Full technical report: DeepSeek_V4.pdf (43 pages, 4.3MB, from HuggingFace).

```text
V4-Pro:
  - Total parameters: 1.6T
  - Activated per token: 49B
  - Layers: 61 transformer layers, hidden dim: 7168
  - MoE: 1 shared expert + 384 routed experts, 6 active per token (Hash routing for first 3 layers)
  - Attention: interleaved CSA+HCA (first 2 layers HCA only)
  - CSA: m=4 compression, top-k=1024, n_h^I=64 indexer heads, c_I=128, n_h=128 query heads, c=512 head dim, d_c=1536
  - HCA: m'=128 compression
  - SWA: n_win=128 sliding window, grouped output: g=16, d_g=1024
  - mHC: n_hc=4 expansion, Sinkhorn-Knopp 20 iterations
  - Context: 1,000,000 tokens
  - Training: 33T tokens, Muon (μ=0.95, wd=0.1, γ=0.18), AdamW for embed/prediction head/RMSNorm
  - LR: 2.0e-4 → 2.0e-5, max batch: 94.4M, 4K→16K→64K→1M sequence ramp
  - Precision: FP4 (routed expert weights + indexer QK path) + FP8 (other weights)
  - Download size: ~865 GB, License: MIT
  - Optimizer: Muon (hybrid Newton-Schulz: 8 fast + 2 stable iterations)

V4-Flash:
  - Total parameters: 284B
  - Activated per token: 13B
  - Layers: 43 transformer layers, hidden dim: 4096
  - MoE: 1 shared expert + 256 routed experts, 6 active per token (Hash routing for first 3 layers)
  - Attention: interleaved CSA+HCA (first 2 layers pure SWA)
  - CSA: m=4 compression, top-k=512, n_h^I=64, c_I=128, n_h=64, c=512, d_c=1024
  - HCA: m'=128 compression
  - SWA: n_win=128, grouped output: g=8, d_g=1024
  - mHC: n_hc=4 expansion, Sinkhorn-Knopp 20 iterations
  - Context: 1,000,000 tokens
  - Training: 32T tokens, Muon (μ=0.95, wd=0.1, γ=0.18)
  - LR: 2.7e-4 → 2.7e-5, max batch: 75.5M
  - Download size: ~160 GB, License: MIT

Key architectural innovations (paper-grounded):

1. Compressed Sparse Attention (CSA):
   - Dual-stream KV compression (C^a, C^b with overlapping windows), m=4
   - Lightning Indexer: low-rank queries (W^DQ + W^IUQ), ReLU scoring, per-head weights
   - Shared KV MQA: compressed entries serve as both K and V
   - Grouped Output Projection: g groups reduce n_h·c → d overhead

2. Heavily Compressed Attention (HCA):
   - Single-stream compression, m'=128 (no overlap)
   - Dense attention on heavily compressed representation
   - No sparse selection — pure global summary

3. Combined CSA + HCA at 1M context:
   - V4-Pro: 27% FLOPs, 10% KV cache of V3.2
   - V4-Flash: 10% FLOPs, 7% KV cache of V3.2
   - vs BF16 GQA8 baseline: ~2% KV cache
   - Mixed KV precision: BF16 (RoPE 64 dims) + FP8 (remaining)

4. Manifold-Constrained Hyper-Connections (mHC):
   - Residual mapping B_l constrained to doubly stochastic manifold (Birkhoff polytope)
   - Sinkhorn-Knopp projection (20 iters, exp→row norm→col norm)
   - Dynamic parameterization: input-dependent + static components
   - A_l, C_l bounded via Sigmoid, ∥B_l∥₂ ≤ 1 ensures stability
   - Engineering overhead: only 6.7% of 1F1B pipeline stage

5. Muon Optimizer:
   - Hybrid Newton-Schulz: 8 steps (3.4445, -4.7750, 2.0315) + 2 steps (2, -1.5, 0.5)
   - Nesterov momentum (μ=0.95), BF16 Newton-Schulz stable
   - Mixed ZeRO: knapsack for dense params, per-expert for MoE
   - MoE gradients: stochastic rounding to BF16, all-to-all + FP32 local sum

Infrastructure innovations:

6. MegaMoE (Fused EP Kernel):
   - Wave-based expert scheduling (1.5-1.96× speedup)
   - Computation-communication ratio: C/B ≤ 6144 FLOPs/Byte hides communication
   - Open-sourced in DeepGEMM

7. TileLang DSL:
   - Host Codegen: Python checks → C++ (<1μs per invocation)
   - Z3 SMT solver for formal integer analysis
   - IEEE-754 + bitwise reproducibility

8. Batch-Invariant & Deterministic Kernels:
   - Dual-kernel attention (single SM + multi SM for wave-quantization)
   - DeepGEMM replaces cuBLAS end-to-end
   - Deterministic MoE backward: token order preprocessing + buffer isolation

9. KV Cache Management:
   - Heterogeneous: State Cache (SWA + uncompressed tail) + Classical Cache (CSA/HCA blocks)
   - Block size: lcm(4, 128) = 128 original tokens per block
   - On-disk storage: Full SWA / Periodic Checkpointing / Zero SWA Caching

Post-training innovations:

10. FP4 Quantization-Aware Training:
    - MoE expert weights: FP32 master→FP4→FP8 (lossless dequant)
    - Indexer QK path: full FP4
    - Index scores: FP32→BF16 (2× speedup, 99.7% recall)

11. On-Policy Distillation (OPD):
    - 10+ teacher models → single student
    - Full-vocabulary reverse KL (D_KL(π_θ ∥ π_Ei))
    - Teachers offloaded to distributed storage, ZeRO-like sharding
    - Cached last-layer hidden states → on-the-fly logits reconstruction

12. Quick Instruction:
    - Special tokens (<|action|>, <|query|>, <|domain|>, etc.)
    - Reuses existing KV cache, zero redundant prefill
    - Parallel execution of auxiliary tasks

13. Interleaved Thinking:
    - Tool-calling: full reasoning history preserved across rounds
    - General conversation: previous reasoning discarded per turn

14. DSec Sandbox (Rust):
    - 4 substrates: Function Call / Container / microVM (Firecracker) / fullVM (QEMU)
    - 3FS-backed layered storage, preemption-safe trajectory logging
    - Hundreds of thousands of concurrent instances

Training stability:

15. Anticipatory Routing:
    - Step t uses θ_t for features, θ_{t-Δt} routing indices
    - Auto-detection: loss spike → short rollback → activate → revert
    - ~20% overhead when active (negligible overall)

16. SwiGLU Clamping:
    - Linear component: clamp [-10, 10]
    - Gate component: upper bound 10

V4-Pro-Max benchmarks (selected):
  - SimpleQA: 57.9% (vs Gemini 3.1 Pro 75.6%)
  - HLE: 37.7% (vs Opus 4.6 40.0%)
  - LiveCodeBench: 93.5 (highest, vs GPT-5.4 91.7%)
  - Codeforces: 3206 (23rd human, vs GPT-5.4 3168)
  - SWE-Verified: 80.6% (vs Opus 4.6 80.8%)
  - TerminalBench 2.0: 67.9% (vs GPT-5.4 75.1%)
  - MRCR 1M: 83.5 MMR (vs Opus 4.6 92.9%)
  - CorpusQA 1M: 62.0% (vs Opus 4.6 71.7%)
  - Formal Putnam 2025: 120/120

Pricing:
  - V4-Pro: $1.74/M in, $3.48/M out
  - V4-Flash: $0.14/M in, $0.28/M out
  - V4-Pro (Max reasoning): $3.48/M in, $6.96/M out
  - vs GPT-5.5: ~$0.55-0.75/M in, ~$2-3/M out (~25-30x cheaper)
  - vs Claude Opus 4.6: ~$15/M in, ~$75/M out (~20x cheaper)
```

### 5.6 DeepSeek V3.2: DSA + Scalable RL + Agent Pipeline

arXiv: 2512.02556 (Dec 2, 2025). Not an experimental — full V3.2 release.

```text
Three key innovations:

1. DeepSeek Sparse Attention (DSA):
   - Lightning indexer: small, multi-head scorer (FP8 efficient)
   - Fine-grained token selection: top-k KV retrieval per query
   - Implemented under MLA framework
   - Dense warm-up (1K steps, 2.1B tokens) trains indexer via KL-divergence
   - Full sparse training adapts all params to sparse pattern
   - Result: first fine-grained sparse attention matching dense quality

2. Scalable RL Framework:
   - Post-training compute scaled substantially (larger than typical)
   - V3.2-Speciale: higher compute variant
   - Achieves IMO 2025 gold + IOI gold
   - Matches GPT-5 on reasoning, Gemini-3.0-Pro on complex reasoning

3. Agentic Task Synthesis Pipeline:
   - Novel pipeline for generating training data for tool-use scenarios
   - Scalable agentic post-training
   - Improved generalization and instruction-following in interactive envs

Benchmarks:
  - Codeforces: 2121 (V3.2-Exp), higher in full V3.2
  - LiveCodeBench: 74.1 (Exp)
  - Aider-Polyglot: 74.5 (Exp)
  - MMLU-Pro: 85.0
  - GPQA-Diamond: 79.9
  - AIME 2025: 89.3
  - SWE-bench Verified: ~50 (via V3.2)
  - BrowseComp: 40.1 / BrowseComp-zh: 47.9
```

### 5.7 Engram (Conditional Memory)

Introduced as a new sparsity axis complementary to MoE.

```text
MoE: scales capacity via conditional computation (routing)
Engram: scales knowledge via conditional memory (lookup)

Key idea: N-gram embedding lookup as O(1) knowledge primitive
  - Deterministic addressing
  - Offloadable to host memory with minimal inference overhead
  - Relieves early layers from static pattern reconstruction
  - U-shaped scaling law for optimal MoE vs Engram allocation
  - Engram-27B improves over MoE baselines on knowledge/reasoning/code/math
```

## 6. DeepSeek Model Evolution Timeline

```text
DeepSeek-LLM (base)
  ↓
DeepSeekMoE (fine-grained experts)
  ↓
DeepSeek-V2 (MLA + DeepSeekMoE, 236B)
  ↓
DeepSeek-Coder / Coder-V2 (code focus)
  ↓
DeepSeek-Math / Math-V2 (GRPO, IMO gold)
  ↓
DeepSeek-V3 (671B MoE, FP8, $5.5M, 128K context)
  ↓
DeepSeek-R1 / R1-Zero (pure RL reasoning, emergent CoT)
  ↓
DeepSeek-VL / VL2 / Janus (multimodal)
  ↓
DeepSeek-Prover-V1.5 / V2 (Lean 4 formal reasoning)
  ↓
DeepSeek-OCR / OCR-2 (visual understanding)
  ↓
DeepSeek-V3.1-Terminus (128K context extension)
  ↓
DeepSeek-V3.2-Exp / V3.2 (DSA sparse attention + agent pipeline)
  ↓
Engram (conditional memory sparsity)
  ↓
DeepSeek-V4-Pro / V4-Flash (CSA+HCA hybrid attention, mHC, Muon, 1M context)
```

The V4 series (Apr 2026) represents the current frontier — CSA+HCA attention enabling practical 1M-token context at 27% FLOPs and 10% KV cache of V3.2.

## 7. Infrastructure Innovation Timeline

```text
Fire-Flyer AI-HPC (hardware-software co-design)
  ↓
FlashMLA (efficient MLA GPU kernels)
  ↓
DeepGEMM (FP8 GEMM with fine-grained scaling)
  ↓
DeepEP (expert-parallel communication)
  ↓
DualPipe (bidirectional pipeline parallelism)
  ↓
EPLB / LPLB (expert load balancing)
  ↓
3FS (distributed file system for AI)
  ↓
smallpond (data processing on DuckDB + 3FS)
  ↓
TileKernels / TileLang (DSL kernel library with Host Codegen + Z3 SMT)
  ↓
MegaMoE (fused EP kernel with wave scheduling, 1.5-1.96× speedup)
  ↓
DSec Sandbox (Firecracker microVM + 3FS, 100K+ concurrent agent sandboxes)
```

## 8. Relevance to DeepCode CLI

DeepSeek's research is directly relevant to DeepCode CLI architecture:

### 8.1 Code Intelligence

- DeepSeek-Coder/Coder-V2: Fill-in-Middle, long-context code understanding
- DeepSeek-V3.2-Exp: Codeforces 2121, LiveCodeBench 74.1, Aider-Polyglot 74.5
- SWE-bench Verified 67.8, SWE-bench Multilingual 57.9

### 8.2 Reasoning for Debugging/Architecture

- DeepSeek-R1: emergent chain-of-thought, self-verification, reflection
- GRPO: efficient RL without critic model
- Applicable to training code-review/debug agents

### 8.3 Formal Verification

- DeepSeek-Prover-V2: Lean 4 theorem proving
- Applicable to formal verification of generated code

### 8.4 Efficient Inference

- MLA: 93% KV cache reduction for long code contexts
- DSA: efficient long-context processing
- MoE: 37B activated from 671B total parameters
- Engram: O(1) knowledge lookup

### 8.5 Multimodal Evidence

- DeepSeek-OCR/OCR-2: visual understanding for GUI evidence
- Janus: unified multimodal for screenshot/code understanding

### 8.6 Infrastructure

- 3FS: distributed file system for large codebase training
- DualPipe: efficient distributed training
- DeepGEMM/FlashMLA: optimized inference kernels

## 9. Total GitHub Stats Summary

| Category | Repos | Total Stars |
|---|---|---|
| Foundation Models | 6 | ~209,000 |
| Code Models | 2 | ~30,000 |
| Math/Proving | 4 | ~6,700 |
| Vision/Multimodal | 5 | ~53,000 |
| Infrastructure | 11 | ~50,000 |
| Research | 2 | ~5,200 |
| Ecosystem | 3 | ~41,000 |
| **Total** | **36** | **~395,000** |