Kimi K3 Is Competitive with Fable; Kimi K3 and Fable Is SoTA

Also: Kimi K3: second only to Fable 5 on AA-Briefcase https://artificialanalysis.ai/articles/kimi-k3-agentic-knowl...

Comments URL: https://news.ycombinator.com/item?id=48999291

Points: 825

# Comments: 426

Hacker News 讨论

825 points · 426 comments · 查看原帖

nxtfari
If you haven’t been really running and testing these models yourself, they are all benchmaxxed. No matter how close they score to frontier on whatever metric, they always fall apart in real world tasks and their token efficiency is ridiculously bad. Fireworks has incredible incentive to make this claim in a headline, because Fireworks hosting K3 for you is pure profit for them, unlike when they host closed source models.
martinjc
I wonder why there's so much resistance here against chinese models. Sure at my employer claude is used, but at home? I am just happy using my z.ai sub for 20x i got in September last year, coupled with the 39 dollar tier of kimi. I use them in pi, with a collection of extensions i curated myself for this iterationm of models, and a couple glue extensions we have made. At home i feel way more productive, the speed of my queries are second to none with kimi 2.6, and handing review over to glm5.2 means i can juggle the small models in my brainstorm to commit workflow.
acd
This benchmark is probably also self promotion of services. Fireworks happens to make a router. Using the router gets better performance. https://docs.fireworks.ai/deployments/routers
JSR_FDED
Very interesting. They test Kimi K3 and Fable on a set of approx 1000 tasks grouped into 5 areas (SWE, Legal, etc). They put a router model in front that predicts whether Kimi or Fable is going to give a better cost for a correct result. (They believe that ultimately such a router model should be continuously trained on your own workloads so it makes the best decisions for you). Their router chose Kimi the majority of the time (72% in one category, all the way to 96% in another category), leading to cost savings in every category (from 1.5x to 50x depending).
thecopy
When DeepSeek was released, it had an immidiate and significant impact on the US stock-market. Now when its becoming common knowledge that China is almost at parity with US SOTA models with good momentum, why is there no sentiment change on the market?
kaycey2022
What is routing? How do they decide which is better? The only way to come up with a routing model for your workload is to send queries to all the models and then come up with a way to say which solution was better, often trying out multiple times for the same model + query to account for other statistical errors. This makes you, an ai inference user an unwitting AI company with a non scalable product. The biggest mental trap people have fallen for is the notion of “best” and, always using the frontier model. Instead you should just bite the bulet and choose the cheapest or the best. This routing dance is just tokenmaxxing in disguise. Edit: Another thing concerning me is that the models themselves are not concrete behind their endpoints. Models can be arbitrarily dumbed down by reducing their inference resources. Once Anthro/OAI release a new great model, they are fully incentivised to d
abdullahkhalids
I will accept a 5% drop in benchmarks for a model that talks to me like a human.
tracker1
Kind of cool to see.. that said, there's more to a tooling experience than benchmarks and specific models. Cursor, Claude Code, Codex, etc. add to the mix. Things like Open-Router and backend options make it easy enough to test. The tools, libraries and languages you are using can also dramatically affect results. Even on state of the art models, I find, for example, the output of SQL for complex interactions, or C# for that matter to be sub-par, where I find Rust results to be pretty great, with JS/TS falling in between. At the best, it can feel amazing and productive, at worst, time consuming and annoying that you could have done it faster yourself. YMMV in real world use. Note: I'm a proponent of human in the loop gatekeeper/reviewer usage of AI, and I'm not able to even consider Chinese models for my own use, and not able to use anything at my day job.