The conversation surrounding AI infrastructure has correctly identified the key value (KV) cache as a critical bottleneck in scaling AI inference. As models push toward longer context windows and ...