Hacker NewsArticle URL: https://prismml.com/news/bonsai-27b
Comments URL: https://news.ycombinator.com/item?id=48910545
Points: 653
# Comments: 232
Hacker News 讨论
656 points · 234 comments · 查看原帖
- SwellJoe
What I most want to see it compared to is Gemma 4 12B in the 4-bit QAT version. It's barely bigger than this at just under 7GB, so it also runs on just about any modern device and is remarkably smart for its size. It's an excellent tool user, crazy good vision for its size. I'm still trying to wrap my head around how much is lost with each step down in resolution, but the QAT versions from Google seem to prove the answer is "very little" at four bits.
- kbart
Excuse my likely stupid question, but has anybody had some success using Claude Code with frontier agents (or Junie or anything else) to invoke local LLMs for specific sub-tasks or wrapped as skills? In other words, is there a way to use expensive, frontier models as orchestrators that manage local models to do the specialised coding tasks?
- motbus3
I need help understanding this. I understood that the magic here is the quantization that allows it to use from 50G to 4G and their process retain most of the intelligence within Pareto limits of gain. And then they proceed to compare with other quantized models as in the level of intelligence per size. It gets to my attention though that the performance in tool calling is mostly affected which is a problem for other small models. How does this model compare to a recent 4G model? How do we know it retained intelligence from the parent rather then being fine tuned for the benchmarks? I am not shtng on them or anything. I'd rather find it amazing, BUT given my limited knowledge, I feel the results miss fair comparison plots and the ones might be misleading. Buy I also reckon it might be me the problem. Anyone care to explain this poor silly fellow some of those points?
- kristianp
Apparently Apple is "in talks" with the PrismML: https://www.cnbc.com/2026/07/14/apple-prismml-ai-compression...
- networked
I have benchmarked Bonsai 27B CPU inference on my computer (a Ryzen 7 5700X desktop with 48G RAM running Ubuntu 24.04) using the latest 62061f910 build of PrismML's llama.cpp fork. Binary: 9 t/s prompt, 6 t/s generation. Ternary: 0.8 t/s prompt, 0.7 t/s generation. It looks like CPU inference for ternary isn't optimized yet.
- Havoc
Got this running on my phone. Unfortunately like other small models it hallucinates quite easily. eg asked it what Signoz is. It reckoned it is a woocommerce/shopify competitor aimed at India market
- davedx
I find it super interesting that we're now in an era where we have LLM's that are quantized to binary weights - 1's and 0's. So effectively they're digital neural networks. I assume that in addition to the significant memory savings, this should also lead to much simpler matrix multiplication operations? Could models like these run on CPU's efficiently, or does the geometry of the compute mean GPU's are still a better choice?
- simonw
The models themselves are showing up on Hugging Face here: https://huggingface.co/prism-ml/models I've tried a couple in LM Studio - the GGUF one and the MLX one - but neither worked there. Anyone else get them to work? Might be that LM Studio needs to upgrade their llama.cpp or MLX engines first.