EntropyKV is an online, 100% attention-free key-value (KV) cache eviction policy designed to run long-context LLMs on consumer-grade GPUs. Traditional KV cache compression methods (e.g., H2O, SnapKV) ...
Note from Dec 2022: The code here work beautifully and I plan to continue making minor bug fixes maintaining the current functionality. But I no longer will be making any improvements to this project.
Some results have been hidden because they may be inaccessible to you
Show inaccessible results