As the intent is to provide a very thin wrapping layer and play to the strengths of the original c++ library as well as python, the approach to wrapping intentionally adopts the following guidelines: ...

In this work, we propose SlimInfer, an innovative framework designed to accelerate long-context inference for Large Language Models (LLMs) by dynamically pruning less critical prompt tokens during the ...