It's kind of insane how having this tech at this speed 5 years ago would have probably still been seen as insanely useful and revolutionary. If LLMs were more capable but dramatically slower, I wonder how it would impact how we use it? Dramatically more thought being put into prompts, much more preparation probably
The future of inference is likely in ASICs, so we'll get the inverse, a bit less capable than frontier but super fast models. Like this 14k tok/s beast https://chatjimmy.ai/ from Taalas (who got acquired by AMD recently).
GPT-6-astra runs at like ~40 tok/s, I have a hard time imagining what could be accomplished with that type of model at 10k+ tok/s when in the hands of the public. Will certainly make cybersecurity a challenge for older systems.
At 1t/s it's still faster than humans for a lot of tasks, basically doing overnight what could take humans half a week. Plus you can always parallelize.
This is what people forget when they see slow performance: at 1 t/s it's still roughly the equivalent of having another person work for you at no extra cost besides the initial purchase/sign-on-bonus.
Frontier models are amazing, but what will really be useful for us is having models and hardware so efficient that you can run useful LLMs locally. One of my favourite LLMs to this day is still my jail-broken gemma4 12b because it's small enough to run on my computer, but also 100% local and free as in liberty.
I don't really do anything at my job that could be done in a single prompt, and certainly not something that would take me 7 hours. A 7 hour task would take multiple iterations with how LLMs are right now.
You said it would do overnight what would take a human half a week, so I'm curious what tasks you are doing where AI is 3x faster than you even at 1t/s
Your prompts are probably very underspecified then. Frontier models one shot the majority of my prompts. UI is kind of the exception, there I do have to ask for a lot of tweaks.
Maybe. Do you have an example of a prompt you've done recently?
I don't think it's valuable at all to try and craft a specific prompt to "one-shot" a task, I'm way faster just doing small asks and guiding it. At 1t/s, how would you even know your prompt was insufficient? It would take you 7 hours to see it.
Stuff along the lines of implement controller service and tests for the following endpoints:
- list of many endpoints with the JSON they receive and return and description of what they need to achieve
Stuff you could probably do in a single work day if you lock in and enter flow state, but in a typical job takes like half a week. And the vast majority of times the AI one shots it with no bugs, where I would have copy paste errors or dumb stuff I'd need to fix before it's shippable.
Write the program, punch the cards, send the cards to another building to be loaded, program runs, printout comes out in another building, somehow this takes 2-3 days
Those machines were also manually scheduled. So you booked in your time slot to run your program in advance. That syntax error meant not being able to run the fixed version until your next scheduled session, maybe days later.
How many times have you had a model start compacting already before getting back to you? Most have 1 million context window. It's happened to me occasionally
In particular you can still get used Optane SSDs on eBay, although they’re fairly pricey. (not the bogus m.2 ones that are slower than a halfway decent consumer NVMe)
LLMs tend to care for sequential bandwidth more than random access. From that perspective, Optane probably doesn't offer much of an advantage vs other NVMe drives even from the same era while losing out greatly to newer drives with faster busses to fill. There could be something special to this particular instance but https://github.com/argonautlabsai/deltafin/blob/main/k3-publ... and the overall tok/s numbers seem to suggest it's just the sequential that's playing in.
Decode was flat from 128 to 512 generated tokens (0.926 → 0.923 tok/s drafter-off), but that's a 6-token prompt plus the output — total context under a thousand.
The current configuration admits about 4.4k tokens of context at all, and at anything like 200k the killer wouldn't be decode, it would be prefill: today it reads each layer's experts once per 64-row pass,
so 200k tokens of prompt would be measured in days, not minutes, until the scheduling fix.
Author here. Some context and the caveats up front.
The model is Kimi K3, 2.78T parameters, ~1.45 TB of expert weights. It does not fit in memory, so the experts stream from disk: one 17.5 MB file per (layer, expert), read with pread + F_NOCACHE, 16 of 896 per layer. The machine is an M5 Max MacBook Pro with 128 GB and three Thunderbolt 5 enclosures plus the internal SSD. Expert weights are untouched at their released MXFP4 precision; the resident attention trunk is int8, which upstream labels non-weight-exact, so I don't claim bit-exactness against BF16 — I claim token-identical output against my own reference on the prompt of record, checked on every promotion.
Numbers, with the unflattering ones in the same paragraph as the good ones: 1.00 tok/s steady over a 512-token completion, 1.13 over 128 tokens, and 0.96 median on the 17-token benchmark from the upstream repo's issue #15 against the 0.684 posted there. Time to first token on a 512-token prompt is about 6.3 minutes — prefill is currently read-amplified 6.2x, which is the biggest open problem in the repo and is described in the results directory.
What I think is actually interesting isn't the number, it's that four of the gains came from defects in the read path that instrumentation found and I would never have guessed. The instruments are in a second repo, ARGODRIVE — a 10 ms per-device read monitor, a per-read barrier trace that records which drive served each expert and which one landed last in every pass, and a config assertion harness that refuses to record a benchmark unless the setting under test actually fired. They're deltafin-specific today. The four findings:
• A constant capped the reader threads at 16 and bounded both the demand and prefetch pools with the same value. Separating them was +14%; demand queueing went from 70% of blocked time to 7.5%.
• Splitting each hot expert's read across two replicas on two devices was +10% — after the same knob had measured negative six times on layouts where every expert had one home and there was nothing to split against.
• The prefetch path had no balancer at all: it walked a fixed directory order and took the first hit, so on any replicated layout it dumped everything on one enclosure. Giving it least-expected-completion dispatch with in-flight counters shared with the demand path was +11% and turned every replicated layout I had previously measured as a loss into a win.
• A recorded "law" that a given draft depth was worse turned out to have been measured against a drafter that no longer existed. Re-testing it was +8%.
There's also a drive-count ladder in the repo — same layout, one to four drives: 57% / 78% / 92% / 100% of the four-drive decode rate. And a catalogue of about a thousand timed runs of things that did not work, with the numbers: RAM expert caches from 8 to 40 GB (-4% to -48%), striping a single copy (-7 to -25%), two drives sharing one Thunderbolt link (-11%), streaming the attention trunk from SSD (-60%), Metal's file-loading API (-19 to -22%). That catalogue is the part I expect to be most useful to other people.
The engine is a fork of gavamedia/deltafin, which is MIT and did the hard part; I've told the author about all of this and the upstream-relevant fixes are going back as PRs.
Two things I'd genuinely like help with: whether anyone has done expert-major prefill scheduling on an MoE (read each expert once per layer and run its kernel over all rows routed to it — it should take prefill from 6.2x amplification to about 1x), and whether the drive ladder reproduces on other hardware.
This response is so dense with numbers and special characters that it's probably about 1000 tokens. So at 1 token/s, it takes almost 17 minutes to generate this on the MacBook Pro.
And maybe author sent ~1K tokens as the starting prompt, and possibly some more stuff in the system prompt, and add on top of that that Apple hardware is famously bad at prompt processing.
I've never understood why "Hacker" News so frequently gets "But why though?" comments at the top.
The entire history of innovation is filled with people doing something just to see they can get it to work, even if badly, and then people continue to iterate on that until it works better, then works well, and then is so obvious people would never even question it. But it all starts with someone doing it to scratch an itch.
Neural networks, the foundation of our current AI revolution, used to fit well into the "neat, but practically useless" category.
Sure there are countless "but why though?" experiments that don't pan out, but that's just the cost of exploration. There can be no step-function innovation in a world where people only do things that make immediate practical sense.
Gen X here. Having grown up in the 80's I remember multiple occasions where someone would ask "what are you going to do with a computer?". As in, what could a computer possibly be useful for. Just imagine someone asking this question today. It would probably be more shaped like the comment you are responding to.
What are you going to do with a computer? I've always hated this attitude. We do these things because they are interesting to us, for the fun of exploration, because we enjoy learning, because we want to iterate and improve, to make the world better, or any plethora of reasons that involve intellectual curiosity of some sort.
It actully does the job. Example: every morning it takes 30-40 minutes to generate reports automatically and these reports are being sent as a pdf to read to Telegram.
Qwen3.8-Flash-Next ships with a 51B lookup table that can be read directly from ssd or memory, which greatly improves it's speed and intelligence. It can load at 4bit quant in ~60GB.
These demos are maybe useless, but if open models keep progressing, there's going to be some break through that continues whittling down just how much needs to be kept in VRAM, and progressive degredation to regular system ram and to ssds.
Afterall, they're not writing anything to these, so saturing all bandwidth could bring models to the masses. all without any help from Zark Muckerberg.
I've had the thought that an underexplored use case for llms is treating them as classification machines. Put in a lot of input then ask for a single token classifying the input. Like "does this code have any bugs? reply "yes" or "no""
Doing something like that at 1 token per second may be useful.
That's the one workload this setup is worst at today, unfortunately: output tokens are cheap at 1/s but input isn't — a 512-token prompt takes ~6 minutes before the first token,
because prefill currently reads each layer's experts once per 64-row pass (~9 TB of reads for a 1.4 TB model).
Fix is scheduling and it's the next thing being built; once prefill reads each expert once per layer, the one-token-out classifier pattern becomes the sweet spot rather than the worst case.
I hate seeing comments like this on HN. We used to upvote “look at this crazy thing I did” work. Not everything has to make sense or be ground breaking.
Not useful for chat, agreed — and I wouldn't pretend otherwise. It's useful for the other kind of work: scheduled, unattended jobs where nobody is waiting on the cursor. My use is day/week/month end review — go through the numbers, flag what doesn't reconcile, draft the report — and there the two things that matter are that the model is good enough to trust with the judgement (K3 is, and it's the full 2.8T model, not a cut-down one) and that the data never leaves the machine.
GPT-6-astra runs at like ~40 tok/s, I have a hard time imagining what could be accomplished with that type of model at 10k+ tok/s when in the hands of the public. Will certainly make cybersecurity a challenge for older systems.
Frontier models are amazing, but what will really be useful for us is having models and hardware so efficient that you can run useful LLMs locally. One of my favourite LLMs to this day is still my jail-broken gemma4 12b because it's small enough to run on my computer, but also 100% local and free as in liberty.
I get weird interruptions. I can only guess its related to some kind of timeouts in the harness or something.
Its not a problem of the model of course, but it seems impractical atm.
I wonder if anyone else had this kind of thing happening.
Which tasks? I think you're underestimating how token hungry current proposed workflows are.
You said it would do overnight what would take a human half a week, so I'm curious what tasks you are doing where AI is 3x faster than you even at 1t/s
I don't think it's valuable at all to try and craft a specific prompt to "one-shot" a task, I'm way faster just doing small asks and guiding it. At 1t/s, how would you even know your prompt was insufficient? It would take you 7 hours to see it.
- list of many endpoints with the JSON they receive and return and description of what they need to achieve
Stuff you could probably do in a single work day if you lock in and enter flow state, but in a typical job takes like half a week. And the vast majority of times the AI one shots it with no bugs, where I would have copy paste errors or dumb stuff I'd need to fix before it's shippable.
Maybe a dumb question.
https://github.com/argonautlabsai/deltafin/tree/main/k3-publ...
In particular you can still get used Optane SSDs on eBay, although they’re fairly pricey. (not the bogus m.2 ones that are slower than a halfway decent consumer NVMe)
Decode was flat from 128 to 512 generated tokens (0.926 → 0.923 tok/s drafter-off), but that's a 6-token prompt plus the output — total context under a thousand.
The current configuration admits about 4.4k tokens of context at all, and at anything like 200k the killer wouldn't be decode, it would be prefill: today it reads each layer's experts once per 64-row pass,
so 200k tokens of prompt would be measured in days, not minutes, until the scheduling fix.
4.4k seems... I just realized I have no sense of scale whatsoever
or 40 macbooks with each 4 ssd. to get 40 tps.
it is even worse with 40 macbooks.
if 40 macbooks is all that take to serve a 1TB model with decent speed then you would see everyone selling the models for very cheap right now.
so... enough for 1 question + 1 long answer?
The entire history of innovation is filled with people doing something just to see they can get it to work, even if badly, and then people continue to iterate on that until it works better, then works well, and then is so obvious people would never even question it. But it all starts with someone doing it to scratch an itch.
Neural networks, the foundation of our current AI revolution, used to fit well into the "neat, but practically useless" category.
Sure there are countless "but why though?" experiments that don't pan out, but that's just the cost of exploration. There can be no step-function innovation in a world where people only do things that make immediate practical sense.
What are you going to do with a computer? I've always hated this attitude. We do these things because they are interesting to us, for the fun of exploration, because we enjoy learning, because we want to iterate and improve, to make the world better, or any plethora of reasons that involve intellectual curiosity of some sort.
> Just imagine someone asking this question today.
to be fair, considering how young folks live whole life in a smartphone nowadays...
These demos are maybe useless, but if open models keep progressing, there's going to be some break through that continues whittling down just how much needs to be kept in VRAM, and progressive degredation to regular system ram and to ssds.
Afterall, they're not writing anything to these, so saturing all bandwidth could bring models to the masses. all without any help from Zark Muckerberg.
Doing something like that at 1 token per second may be useful.
because prefill currently reads each layer's experts once per 64-row pass (~9 TB of reads for a 1.4 TB model).
Fix is scheduling and it's the next thing being built; once prefill reads each expert once per layer, the one-token-out classifier pattern becomes the sweet spot rather than the worst case.
It is cool that they got it to work at all.
Such of a Claudism. Not criticizing, just noticing.