kv cache quantization

#1
by mahdisml - opened

it is possible to quantize kv cache in strata ? :(
In llama.cpp and Unsloth’s llama.cpp, quantizing the KV cache for this model will increase its size instead of decreasing it

Sign up or log in to comment