DMI-Lib is a system for efficiently extracting internal activations (hidden states, attention weights, etc.) from LLMs during inference without slowing down serving. It decouples observation from the main inference path using an asynchronous GPU-CPU memory abstraction that captures and stages tensors, then exports them via a policy-controlled backend. This lets you monitor model internals with only 0.4-6.8% overhead in batch inference and ~6% in online serving, a 2-15× latency improvement over existing approaches.
Main takeaways:
- Extracting model internals during inference usually tanks performance; DMI-Lib adds only 0.4-6.8% overhead (batch) or ~6% (serving)
- Uses asynchronous observability: captures tensors off the inference hot path, stages them in GPU-CPU shared memory, exports via host backend
- Works across diverse inference backends and observation points (hidden states, attention, etc.) without breaking serving optimizations
- 2-15× lower latency overhead than baselines with similar observability features
- Open source library designed as a first-class systems primitive for model-internal monitoring