llama.cpp: Prefetching weights when offloading to CPU
Hello r/LocalLLaMA, I put up an experimental PR which prefetches weights when offloading to CPU. Long story short from results it helps dense + smaller MoE models for PP (prompt processing). Give it a try if you are ram-rich and gpu-poor like me. [https://github.com/ggml-org/llama.cpp/pull/21067](h
Receipts (all sources)
Hello r/LocalLLaMA, I put up an experimental PR which prefetches weights when offloading to CPU. Long story short from results it helps dense + smaller MoE models for PP (prompt processing). Give it a try if you are ram-rich and gpu-poor like me. [https://github.com/ggml-org/llama.cpp/pull/21067](h
I'm having a lot of fun playing with llama-server testing various flags, models and runtimes. I'm starting to wonder what's next to build out my homelab AI stack. Do I use Open WebUI for RAG/Search? Should I take a stab at something like LangGraph? My goal is to create as something as close to Claud