The authors build a compliance-grade LLM serving stack for fraud and AML workloads, which have prefix-heavy prompts (reusable policy text, risk taxonomies) and schema-constrained outputs (JSON labels). They combine vLLM runtime tuning, PagedAttention, automatic prefix caching, multi-adapter serving, adapter-aware batching, and speculative decoding. On public-synthetic AML datasets, workload-aware tuning improved throughput from 612–650 to 3,600 requests/hour, reduced P99 latency from 31–38 seconds to 6.4–8.7 seconds, and increased GPU utilization from 12% to 78%.
Main takeaways:
- Fraud/AML compliance prompts are prefix-heavy with reusable policy instructions and schema-constrained outputs, making KV-cache efficiency and prefix reuse critical.
- Stack combines vLLM-style runtime tuning, PagedAttention, automatic prefix caching, multi-adapter serving, and speculative decoding.
- Workload-aware tuning improved throughput 5–6×, reduced P99 latency 4–5×, and raised GPU utilization from 12% to 78%.
- Includes LLM-as-judge quality gate with deterministic compliance checks and multi-judge rubric scoring.
- Demonstrates that regulated LLM workloads require specialized serving infrastructure tuned to their prompt structure and output constraints.