Ask HN: What hardware are you using to run local LLMs?
Figured I'd ask here because I have no doubt some of you are running some pretty cool, cost effective solutions for local models. Don't doubt some have mini data centers in their closets either haha, but just looking to get a view into what fellow HN'ers are using for their day to day work.
I have a Mac Studio M1 Ultra with 128GB of RAM, an M5 Pro MBP with 64GB of RAM, and until recently an M1 Pro MBP with 32GB of RAM. I also have two Mac Minis with 16GB of RAM, but they're mainly used for embedding models because they're so small.
I'm doing a bunch of batch work for an experiment with them right now, but my daily driver is a Gemini Pro subscription.
I have a pair of ASUS GX10 systems [0] (DGX Sparks in a different enclosure) in a cluster running DeepSeek-V4-Flash-0731 [1] with tensor parallelism. I managed to snag them early this year when prices were merely stratospheric, not mesospheric.
Even at half of today's retail price they're not remotely cost-effective compared to cloud models for my single-developer workflow, and idle power draw is quite high with the ConnectX-7 NICs powered on for clustering. (Mine idle around 45-50W each. Powering off the NICs cuts that roughly in half, but they're still very power-hungry compared to an Apple Silicon Mac or even a Framework Desktop.)
I learned a lot building the cluster, which was my main objective. I could probably sell them for a nice profit, but I'm still enjoying using them.
I have a 5090, once you start splitting weights/cache amongst devices or start having to rely on unified memory you quickly enter the land of diminishing returns and tradeoffs around cost, power consumption and speed. I'd just go for whatever the cheapest single device that will fit your chosen model weights + context at the speed and price you are comfortable with, which for most folks will probably be a single GPU and a model in the 30b neighborhood (qwen/glimmer)
I got an old Dell T5820 64GB RAM and put in 2x V100 32GB GPUs. All in was around $1800 USD.
Runs Qwen3.8-27B Q4 on one GPU at ~55 tok/s. I also put MiniMax H3 on the other GPU for generating images and video, and the kids have fun with that. I use this box for all my personal leisure coding projects, but stick to bigger models over API for real work work.
Local clusters built around old GPUs. I have had cluster of P40 GPUs, MI50s gpu, and now I have a cluster of cmp170hx. I'm always looking for the next cheap GPU dump and then I get as many as I can before it get's popular.
I'm running a local LLM on my system below. It's alright, free intelligence for simple daily tasks. I still have opus sub for deep intelligence applications.
Local LLM loaded: Qwen 3.8 27B (qwen3.8:27b), served through Ollama.
MY SYSTEM:
ASUS ROG MAXIMUS XI HERO Motherboard
Intel® Core™ i9-9900K × 16
64GB DDR5 RAM
RTX 3090 with 24GB VRAM
I have a 4080 Super with 16gb of ram and I run Ollama and have Qwen 3 8b and Qwen embeddings.
I am currently not the serious about local inference, but that could potentially change in the future if better models drop or I get a new PC with a unified memory structure.
A cluster of 4 M3 Ultra Mac Studios with 512 GB of unified memory each. Don't recommend. EXO labs is dead and trying to shard models is really hard. On their own, each Mac can generate text at usable speeds, but I assume that you trade memory for speed. I have never used an NVIDIA GPU, so I guess that you trade model capacity for speed. I would like to know what y'all Nvidia people think
I run a M5 Max macbook with 128gb of RAM, 18 cpu cores, 40 gpu cores. It is a beast for local inference, allows offloading of execution. Still use frontier for design / spec work to hand off to the local models to execute.
The person you are responding to is not rich by any means in terms of tech standards. If you work at a big tech company (not even a FAANG), you make very decent money in good times and bad.
They could have financed this purchase with their yearly bonus or selling some equity that vested.
Not sure if I qualify for the "day to day work" part, but I have an inference PC with an RTX 3060 12GB and 32GB DDR4. I use it to run Qwen3.6-35B-A3B and Gemma4-26B-A4B with llama.cpp. Just in chat mode.
I have a 3090 w/ 24G VRAM, it supports flash attention and bf16, but is a bit old. I dont run generative models on it as they would be slow and inaccurate, but I do run embedding models, OCR, and system one (decision) models on it.
I grabbed a minisforum before they got prohibitively expensive ( and before that a framework variant, but I got a dud from them ). If you can't do that you can estimate what you can run model wise and play with LMS ( some interesting options are now available -- in my recent tests GLM and Qwen are doing well for local stuff. )
Note.. you can always try to pool things ( not memory, that is super hard; devices as there are more small models now that you can squeeze onto things ).
I'm running an RTX 5090 (32gb VRAM) and Strix Halo (128gb unified memory). I haven't had much time to tinker with my AI setup so I'm still running Qwen3.6-27B (NVFP4 quant) on the 5090. The prices today are just insane.
I bought a Framework Desktop mainboard only, 128gb AMD 395+ setup, and put it in an existing case with PSU i had on hand. I had a fan for it. It was $2699 +$15 expedited shipping. The case and PSU/etc was probably $200 more (It's an overkill 850w SFX PSU and cheap but great ITX case from Deepcool).
I started with Ubuntu but pivoted to Fedora over the summer after a recommendation from a security engineer friend. And it is mostly easier.
Key changes for other distros are how to load drivers/apps and get running. Ubuntu has a recent CVE issue but worked fine. CachyOS/etc set up for more Windows like UI and transplants was fine but not focused on the Stricx Halo setup. Although I do see close to RTX 5070 performance in most games I've tested.
I was running LM Studio and some other runtimes for ease of setup and model switching. Once you settle on a runtime and model setup you like it is easy to set it up, let it run, and manage as needed.
Last week I set up GUFO, as it targets the AMD Strix Halo hardware and is efficient for Qwen Flash Next. I could probably get the work done with Qwen 3.8 27B but I like how QFN thinks and lets me follow along. Starting at about 260k context, I see 88gb of RAM usage when loaded and about 108gb when in use, depending on context overhead. I do have to restart the model every so often as Copilot Chat in VS Code starts failing.
I have also run Lemonade, llama.cpp directly, vllm, and more. Custom or bespoke runtimes for your hardware are key, like Srata or GUFO. I need to look at llama-swap. LM Studio is easy and with their Tailscale backed link feature you can stay connected from any device.
IF it supports OpenAI/etc back end you're mostly there. Get the model running and connect from your preferred harness/etc. GUFO works well with Copilot Chat in VS Code. I need to connect it to Pi and openhands/etc later this week once I get this test project in a good spot.
Qwen 3.8 27B, Gemma 4 26B A4B variants, whisper, and more run well. A Spark or bank of GPUs would do better, of course, but not for under $3k for 128gb of memory. I have tested adding an Nvidia 3060 12gb to the Strix Halo but the iGPU is turned off by Fedora and I didn't have more time to solve.
My secondary Fedora AI system has the 3060 12gb and a 5900XT (16c/32t) plus 64gb of DDR4-3200. I need to get Strata up on it and see how well medium sized models work on the mixed memory. I bet MOE models will work well. I have the hardware so will use it but renting capacity is much cheaper if you just want to mess around and get used to the tools.
Happy to answer questions about how I got here. I'm an infra guy, not a software dev, and what I'm doing with AI at home could definitely be 5x faster with a real dev and dev process.
I'm doing a bunch of batch work for an experiment with them right now, but my daily driver is a Gemini Pro subscription.
Even at half of today's retail price they're not remotely cost-effective compared to cloud models for my single-developer workflow, and idle power draw is quite high with the ConnectX-7 NICs powered on for clustering. (Mine idle around 45-50W each. Powering off the NICs cuts that roughly in half, but they're still very power-hungry compared to an Apple Silicon Mac or even a Framework Desktop.)
I learned a lot building the cluster, which was my main objective. I could probably sell them for a nice profit, but I'm still enjoying using them.
[0] https://www.asus.com/networking-iot-servers/desktop-ai-super... [1] https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731
Runs Qwen3.8-27B Q4 on one GPU at ~55 tok/s. I also put MiniMax H3 on the other GPU for generating images and video, and the kids have fun with that. I use this box for all my personal leisure coding projects, but stick to bigger models over API for real work work.
Local LLM loaded: Qwen 3.8 27B (qwen3.8:27b), served through Ollama.
MY SYSTEM: ASUS ROG MAXIMUS XI HERO Motherboard Intel® Core™ i9-9900K × 16 64GB DDR5 RAM RTX 3090 with 24GB VRAM
heavily customised agentic framework: https://github.com/connollydavid/host gets good work out of the weights
They could have financed this purchase with their yearly bonus or selling some equity that vested.
Btw, fellow 3090 user, some time ago i saw this: https://github.com/Zagitta/gpu-unlock
Can't try it personally because windows, but maybe it will help you.
Note.. you can always try to pool things ( not memory, that is super hard; devices as there are more small models now that you can squeeze onto things ).
I started with Ubuntu but pivoted to Fedora over the summer after a recommendation from a security engineer friend. And it is mostly easier.
Generic setup doc here: (not fully updated with GUFO): https://github.com/Selicos/AI_Tools_Public/blob/main/docs/Fe...
Key changes for other distros are how to load drivers/apps and get running. Ubuntu has a recent CVE issue but worked fine. CachyOS/etc set up for more Windows like UI and transplants was fine but not focused on the Stricx Halo setup. Although I do see close to RTX 5070 performance in most games I've tested.
I was running LM Studio and some other runtimes for ease of setup and model switching. Once you settle on a runtime and model setup you like it is easy to set it up, let it run, and manage as needed.
Last week I set up GUFO, as it targets the AMD Strix Halo hardware and is efficient for Qwen Flash Next. I could probably get the work done with Qwen 3.8 27B but I like how QFN thinks and lets me follow along. Starting at about 260k context, I see 88gb of RAM usage when loaded and about 108gb when in use, depending on context overhead. I do have to restart the model every so often as Copilot Chat in VS Code starts failing.
I have also run Lemonade, llama.cpp directly, vllm, and more. Custom or bespoke runtimes for your hardware are key, like Srata or GUFO. I need to look at llama-swap. LM Studio is easy and with their Tailscale backed link feature you can stay connected from any device.
IF it supports OpenAI/etc back end you're mostly there. Get the model running and connect from your preferred harness/etc. GUFO works well with Copilot Chat in VS Code. I need to connect it to Pi and openhands/etc later this week once I get this test project in a good spot.
Qwen 3.8 27B, Gemma 4 26B A4B variants, whisper, and more run well. A Spark or bank of GPUs would do better, of course, but not for under $3k for 128gb of memory. I have tested adding an Nvidia 3060 12gb to the Strix Halo but the iGPU is turned off by Fedora and I didn't have more time to solve.
My secondary Fedora AI system has the 3060 12gb and a 5900XT (16c/32t) plus 64gb of DDR4-3200. I need to get Strata up on it and see how well medium sized models work on the mixed memory. I bet MOE models will work well. I have the hardware so will use it but renting capacity is much cheaper if you just want to mess around and get used to the tools.
Happy to answer questions about how I got here. I'm an infra guy, not a software dev, and what I'm doing with AI at home could definitely be 5x faster with a real dev and dev process.