Rendered at 09:41:02 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
lopotubez 2 hours ago [-]
I feel like these types of benchmarks mostly test what the mainstream tells you is possible and or works. You rarely get to hear the different implementations of inference engines, different types of setups (e.g. hybrid ones or older hardware) and their pros and cons. Also it seems these were tested with very minimal token usage.
Currently on my 9 year old laptop with a GTX 1070 Max-Q 8GB vram, 16 gb ddr4 ram and i7 8750h. I am running a finetuned Qwen 3.5-35B-A3B (Tiel-Coder-35B-A3B-GGUF-MTP) with 4-bit quant on a llamacpp fork (Turboquant) with 131k context at around 16-19 tk/s and 350-500 tk/s prefil using Pi harness. It is amazing for writing scripts and doing medium coding tasks and summarizing text for me. After around 100k context it goes down to 9 tk/s but that is to be expected.
If you ever ask any frontier LLM or these types of benchmarks, they would probably say that the best model I can run on my system is a crappy 9B model or something, which is just completely false. So I really wonder how much more you can pull out from more powerful systems and much better open weight models.
testerius 14 minutes ago [-]
It sounds great and I hope local LLM/AI will be more powerful and available even on old hardware.
I tried llama.cpp with offloading "all what could be" to my GPU (GTX 750 Ti 2 GB). I could run local Qwen2.5-Coder quite fast (I do not remember but something like 25-30 t/s). However this model generates bad code (I tried Python and JavaScript). So, for my setup it is better to chat with free ChatGPT and do copying-pasting between web chat and code editor.
Is it even worth for trying something on my GPU? What do you think?
And yes, I use Claude Code (Opus 5.5) and Codex in my job but for personal stuff I would prefer trying free solutions (it is the reason I tried local LLM). I do not want to buy subs, at leats for now.
samuell 46 minutes ago [-]
Thanks for sharing! I would love for some go to source for this type of insider info from the people who actually have tried. (The best I've found so far are some guys on Twitter/X like Alex Fateev (https://x.com/superalesha) and AJ (https://x.com/ItsmeAjayKV))
I'm personally sitting on an extremely limited Geforce GTX 1660 6GB, and would like to see how far one could go even with such a crappy card.
At least it was quite amazing to see the difference between CPU-based and even this crappy GPU-based inference when trying out some tiny models in jan.ai. It's still like orders of magnitude faster at least.
Abishek_Muthian 13 minutes ago [-]
Try to use your IGPU as well if you have enough RAM, almost everyone has dual GPUs in their computers.
Use Vulkan build of llama.cpp to run low parameter MoE models in your IGPU and CUDA llama.cpp to run dense models in your DGPU.
Ofcourse you need to set your expectations right on what problem you want to solve, I've not been happy with using quantized models for coding even on a decent laptop; where as for NLP, ASR, STT etc. work great with this setup.
I might write more about this setup soon.
OtomotO 5 minutes ago [-]
I need to play around with this.
I am running on a Ryzen 5950X, 64GB DDR4 Ram and a Radeon RX 6700 XT with 12GB... I don't expect wonders, but I have between 11 and 120 (!) t/s.
I told it an open ended question: "What would you improve in this file at {location} in {bigger project}" and the context window seems to be too small. It's running in circles.
I must be doing something wrong.
Again, I am not expecting wonders and I am totally willing to accept that "I am holding it wrong".
Cool idea but I'd like to also be able to set a custom value of tok/s, ignoring all the other options
hizlikovboy27 2 hours ago [-]
around this time, i was thinking about running a local LLM for the company i work for. i'm trying to figure out what kind of hardware i'd need and what level of performance i could expect. i tried a few things on your website but how much do the results reflect real world usage?
in my use case, the model might need to analyze a given task first then generate and execute a database query and finally send the results back to the client. to what extent could your setup handle this kind of workflow and how accurate are the results?
good job.
mirekrusin 53 minutes ago [-]
It would be great if numbers weren't off, would include optimised versions (ie. speculative decoding), context size, include allowed parallel inference (single agents use parallel inference a lot) etc.
I don't think in general people care about long list of models, they just care about the best one.
Also dual gpu is common setup, there should be ie. 2x 4090.
gitowiec 2 hours ago [-]
I would like to see in such simulators:
- multi GPU configuration (hetero and homo)
- PCIe gen chooser (3.0/4.0/5.0)
- is nvlink enabled (2 cards, 2 x 2 cards, 3 x 2 cards)
- is PCIe bifurcated or CPU native (threadripper pro for example)
kristopolous 2 hours ago [-]
just go rent one for a few minutes and try it out ... it'll cost you well under $0.50 if you're just trying to get a feel for if it's right for you. There's a lot of things like context window that this doesn't capture.
Depending on who you sign up for, that's within the introductory credits (usually like 5-10$) so it's actually $0.
hbogert 3 hours ago [-]
Shame they are not including old crap like my RX580 which is doing some impressive stuff with Qwen 9B Models.
dotancohen 2 hours ago [-]
Tell us more about what your RX580 can do with the small Qwen models. What workflows are you doing, and what kind of performance are you seeing?
hbogert 1 hours ago [-]
yeah opencode just runs with it with lmstudio as llm engine.
It won't win benchmarks of course, but 20 tps generation is okay and letting it chug away on my local kubernetes cluster to troubleshoot is pretty nice. First start is pretty slow though, parsing is around 100-200 tps. Which is crap for long questions or harnesses like opencode which start around with thousands of tokens. Once cache is warm, pretty enjoyable.
I think it could have way more use cases where it's just 'good enough'. Email categorization, voice commands for the home. 1 year ago this was still a very miserable experience, but qwen 9B + better cache quantization and more tricks in the latest llm engines makes it viable.
Will it do difficult architectural refactors by just giving a one-shot prompt like opus allows, nope, not gonna happen.
internet_points 2 hours ago [-]
that's 8G vram? you're using it for coding?
hbogert 1 hours ago [-]
Yep, it's 8GB vram from 2017 :D
Yeah simple, but menial refactors where IDEs don't have clear cut refactors yet. Or languages like CUE which has awesome tooling, but not so great refactors in its lsp yet.
kinduff 1 hours ago [-]
Cool examples, this is interesting to test. I've noticed that OpenAI and Meta play a lot with animations and fades to mask the delays.
bozdemir 3 hours ago [-]
Model list is a bit limited, and whole harness affects the speed, so the libs used in harness and their versions might be useful to show for each individual model setup.
albinstman 3 hours ago [-]
I’ll look into adding harnesses. The benchmark sources usually don’t contain harness so the problem is limited data. What models would you like to see?
bozdemir 2 hours ago [-]
For example you might add Strata & Qwen 3.8 Flash Next with some quantizations. Nowadays it is a go to selection and very sought after MoE model.
albinstman 2 hours ago [-]
Qwen 3.8 flash next is already in the works and should be live soon.
cyanydeez 2 hours ago [-]
He's exagerating, I hope. The only bottleneck a harness attachs is pre-context fill and whatever tooling is installed. A proper harness reduces these to negligable startup costs.
Unless of course he's driving some weird lambo that dumps 200k of context into their chats. Who knows, I feel like we're on a yellow submarine.
bozdemir 2 hours ago [-]
Exagerating? I setup llm endpoints on corporate level in a company on H200 gpus, that serves hundreds of software engineers, I have observed differences, even on different versions of nvidia drivers. I shared my day to day experience. It is more and more important if you have very limited VRAM like 24gb on a 4090 RTX desktop or RTX 5090 Laptop gpu. I set up and use llms on my personal devices also, when I can't use cloud models on classified company materials. Here is an example problem what I daily report and deal with.
I can believe different drivers make a difference, but that wasn't the point under debate
bozdemir 2 hours ago [-]
[flagged]
pil0u 3 hours ago [-]
I don't know what basis these numbers are estimated on, but my M2 Ultra 128GB runs almost twice as fast as what's reported for the M3 Ultra 512GB.
I wished there was such a tool before picking my hardware, it's useful. But the estimates needs to be somewhat correct.
albinstman 3 hours ago [-]
Thank you for the feedback. What model/quant did you test so I can look in it?
Perenti 2 hours ago [-]
Doesn't provide for hybrid setups RTX5070 + RTX5080 for example.
friendly_chap 2 hours ago [-]
This is a great idea but I'm far less concerned about output token speed and way more about the ability to parallelize prefill - for agentic coding that's pretty much way more important than output speed.
Currently on my 9 year old laptop with a GTX 1070 Max-Q 8GB vram, 16 gb ddr4 ram and i7 8750h. I am running a finetuned Qwen 3.5-35B-A3B (Tiel-Coder-35B-A3B-GGUF-MTP) with 4-bit quant on a llamacpp fork (Turboquant) with 131k context at around 16-19 tk/s and 350-500 tk/s prefil using Pi harness. It is amazing for writing scripts and doing medium coding tasks and summarizing text for me. After around 100k context it goes down to 9 tk/s but that is to be expected.
If you ever ask any frontier LLM or these types of benchmarks, they would probably say that the best model I can run on my system is a crappy 9B model or something, which is just completely false. So I really wonder how much more you can pull out from more powerful systems and much better open weight models.
I tried llama.cpp with offloading "all what could be" to my GPU (GTX 750 Ti 2 GB). I could run local Qwen2.5-Coder quite fast (I do not remember but something like 25-30 t/s). However this model generates bad code (I tried Python and JavaScript). So, for my setup it is better to chat with free ChatGPT and do copying-pasting between web chat and code editor.
Is it even worth for trying something on my GPU? What do you think?
And yes, I use Claude Code (Opus 5.5) and Codex in my job but for personal stuff I would prefer trying free solutions (it is the reason I tried local LLM). I do not want to buy subs, at leats for now.
I'm personally sitting on an extremely limited Geforce GTX 1660 6GB, and would like to see how far one could go even with such a crappy card.
At least it was quite amazing to see the difference between CPU-based and even this crappy GPU-based inference when trying out some tiny models in jan.ai. It's still like orders of magnitude faster at least.
Use Vulkan build of llama.cpp to run low parameter MoE models in your IGPU and CUDA llama.cpp to run dense models in your DGPU.
Ofcourse you need to set your expectations right on what problem you want to solve, I've not been happy with using quantized models for coding even on a decent laptop; where as for NLP, ASR, STT etc. work great with this setup.
I might write more about this setup soon.
I am running on a Ryzen 5950X, 64GB DDR4 Ram and a Radeon RX 6700 XT with 12GB... I don't expect wonders, but I have between 11 and 120 (!) t/s.
I told it an open ended question: "What would you improve in this file at {location} in {bigger project}" and the context window seems to be too small. It's running in circles.
I must be doing something wrong.
Again, I am not expecting wonders and I am totally willing to accept that "I am holding it wrong".
llama-server \ -m ~/models/tiel-coder/Tiel-Coder-35B-A3B-MTP-UD-IQ4_XS.gguf \ -ngl 20 \ -c 16384 \ --jinja \ --alias tiel-coder
using it in opencode
{ "$schema": "https://opencode.ai/config.json", "provider": { "llamacpp": { "npm": "@ai-sdk/openai-compatible", "name": "Local llama.cpp", "options": { "baseURL": "http://127.0.0.1:8080/v1" }, "models": { "tiel-coder": { "name": "Tiel-Coder 35B A3B", "limit": { "context": 16384, "output": 4096 } } } } }, "model": "llamacpp/tiel-coder" }
in my use case, the model might need to analyze a given task first then generate and execute a database query and finally send the results back to the client. to what extent could your setup handle this kind of workflow and how accurate are the results?
good job.
I don't think in general people care about long list of models, they just care about the best one.
Also dual gpu is common setup, there should be ie. 2x 4090.
- multi GPU configuration (hetero and homo)
- PCIe gen chooser (3.0/4.0/5.0)
- is nvlink enabled (2 cards, 2 x 2 cards, 3 x 2 cards)
- is PCIe bifurcated or CPU native (threadripper pro for example)
Depending on who you sign up for, that's within the introductory credits (usually like 5-10$) so it's actually $0.
It won't win benchmarks of course, but 20 tps generation is okay and letting it chug away on my local kubernetes cluster to troubleshoot is pretty nice. First start is pretty slow though, parsing is around 100-200 tps. Which is crap for long questions or harnesses like opencode which start around with thousands of tokens. Once cache is warm, pretty enjoyable.
I think it could have way more use cases where it's just 'good enough'. Email categorization, voice commands for the home. 1 year ago this was still a very miserable experience, but qwen 9B + better cache quantization and more tricks in the latest llm engines makes it viable.
Will it do difficult architectural refactors by just giving a one-shot prompt like opus allows, nope, not gonna happen.
Yeah simple, but menial refactors where IDEs don't have clear cut refactors yet. Or languages like CUE which has awesome tooling, but not so great refactors in its lsp yet.
Unless of course he's driving some weird lambo that dumps 200k of context into their chats. Who knows, I feel like we're on a yellow submarine.
https://github.com/ggml-org/llama.cpp/issues/27330
I wished there was such a tool before picking my hardware, it's useful. But the estimates needs to be somewhat correct.