AMD Radeon R9700

Damn spell check. I typed it correctly the first time. Anyway, I have 8 r9700 GPUs racked up that can run a dozen or so sessions of qwen 3.8 flash next at once at > 100tps each. I build operating systems and fix compilers with them and reverse engineer anything I want. Never given money to Sam or Dario or anyone else for tokens. If consumer gpus and models never got any better, I am all set.

This is the conventional wisdom, but in practice what matters is how reliably the model performs on your tasks in the real world. I have an R9700 and an RTX 5060 Ti and I've been running an IQ3_S quant of 27B on the 5060 Ti vs a Q6 quant on the R9700. I still manage to get stuff done with the IQ3_S quant.

"If your documents fit the window, the advice stands: keep truncate." "Two reasons we still chunk even when the document would fit:" Which advice do you stand by? Obviously, very short content doesn't need chunking, so let's consider a document that fills 75% of the input context. When chunking, your cost overhead (per token) goes up as the number of new tokens per chunk goes down. That's an argument for longer chunks, although the averaging/smearing point argues for not going too long. Embedding calculations are effectively prefill: on my cheapo local inference system (32 GB AMD R9700 + 8 GB AMD RX 7600), the older 8 GB card goes about 80% as fast as the bigger card for Qwen3-Embedding-4B (a bit over 19 chunks/second on my usual corpus, blog posts+comments that are mostly well under 32K tokens). So I would suggest that anyone who is limited by CPU embedding models could benefit from even a small local GPU. For your blog post, I would suggest an explanation of the chunking modes, either in the blog post or as a hyperlink to the docs about them. "truncate" and "sentence" are fairly clear, whereas the others are not. (If "mean" just computes the mean of the embeddings, that seems like a poor choice. The arithmetic at https://www.johndcook.com/blog/2026/09/16/coffee-milk-latte/ might work for single words, but seems likely to break down at the document level. "recursive" and "fixed" are opaque, at least to me.) If/when I index my team's documents, I will consider a cont…

That may have been true a few months ago, but shit changes fast in this space! I run Qwen 3.8 27b on each of my six AMD AI PRO r9700 GPUs at 80tps decode each and they cranks for days with 256k context, doing complex kernel, compiler, debugging, enclave, bootstrapping, pentesting, hardening, and systems work full time. ~$1200-1500/ea on ebay. Also ~40tps on my strix halo now but with room for several sessions at once.

You will likely have to compromise on memory bandwidth or capacity under a $10k price. The Radeon R9700 has 32 GB of VRAM and is pretty cheap (~$1500 right now), which is what I primarily use. My home desktop has 128 GB RAM and my laptop has 96 GB RAM, but bandwidth limits make most models slow on those CPUs. Models with multi-token prediction are somewhat usable on them: Nemotron 3 Super runs reasonably well on my desktop but does poorly on agentic coding that I've given it; my laptop can run Qwen3.6-27B reasonably well with a version of llama.cpp that is patched for MTP support; but usually I run Qwen3.6-27B on my R9700. vLLM might support two or three R9700s on some OS, but I've not been able to get it to run at all with Ubuntu 26.04: system ROCm version is apparently different than what's in the container images, and system OpenMPI v5.0 finally removed C++ bindings that were deprecated in 2005 but are linked from some Python wheel that vLLM (probably indirectly) imports. If you are spending $800/month on tokens you are likely to notice degradation for local models compared to near-frontier models. The models I can run locally are consistently worse than Claude Sonnet 4.6 (again for the work I give them), although Qwen3.6 does feel almost like magic for its size because it can do a lot. The really big open-weight models should be better, but they want 200+GB RAM, which will need a correspondingly expensive CPU.

I love local first. I am finding that a 120B MoE is hitting the sweet spot for local hosted. Right now that takes a 2K strix halo, a 4k GB10 machine, or a 5k Mac Pro. 2 years from now I think hardware will take us back to the 2k ish range with good performance. I love my dual GPU setup (2AMD Radeon r9700 64GB vram) but it costs 5x electricity than my GX10 (GB10 chip inside) and since layers are landing in system memory my TPS is half the GX10. Now a dense model like Devstral2 24B slaps on the Dual GPU setup. I just haven’t gotten as much out of that as I have the 120 MoEs

As another data point. Running Qwen3.5 122B at 35t/s as a daily driver using Vulcan llama.cpp on kernel 7.0.0rc5 on a Framework Desktop board (Strix Halo 128). Also a pair of AMD AI Pro r9700 cards as my workhorses for zimageturbo, qwen tts/asr and other accessory functions and experiments. Finally have a Radeon 6900 XT running qwen3.5 32B at 60+t/s for a fast all arounder. If I buy anything nvidia it will be only for compatibility testing. AMD hardware is 100% the best option now for cost, freedom, and security for home users.

Bought a Radeon r9700. 32GB vram and it does a good job.

I recently switched from an NVidia card (5090) to a couple of AMD cards (R9700 32GB) for my inference server. I must say it's been a completely positive experience. The mainline Fedora kernel just worked without any need to mess with the DKMS. I just forwarded /dev/dri/* devices to my containers, and everything worked fine with ROCm. I needed to grab a different image (-rocm instead of -cuda) for Ollama, change the type of whisper build for Storyteller. And that was it! On the host, nvtop works fine to visualize the GPU state, and VAAPI provides accelerated encoding for ffmpeg. Honestly, it's been an absolutely pleasant experience compared to getting NVidia CUDA to work.

Scrappy builds aren't "affordable" but are more so than running Nvidia + optimized builds. You could put together a machine for $5000 or less that competes with frontier models from less than a year ago, albeit a little slower. And cards like the $1600 (sheesh) R9700 have actually gotten more capable through community efforts at optimization, since their launch. I think if the bubble can ever deflate/pop, and GPUs and RAM back off a bit, we'll see the dam burst for local AI. Right now it's only accessible to people who can justify the costs.

I am also willing to overpay to not be reliant on cloud things, to have control of local hardware. My biggest problem with buying a Mac Studio is that even in the case of the M5 Max models, I can’t think of any non-AI macOS applications in my creative life that have anywhere near that level of hardware needs, and nor can I forecast that I would within its ordinary supported lifespan as a macOS machine. These machines left high-end stills and modest video work behind generations ago, for example. So while I would like a local machine that I can leave running in a way that I wouldn’t want to do with my old secondhand M1 Max laptop, it’s going to have to be something a little more pragmatic. Probably based around the Radeon R9700, since the AMD/Nvidia gap for LLMs is beginning to close, and even a single R9700 runs the main models I am interested in at speeds that are acceptably faster than what I have here.

Yeah, folks should be aware that if you're filling up the memory on a Strix Halo for an inference workload, you're going to be getting uncomfortably slow token rates. Like, DS4 (a 1-bit quantization of DeepSeek V4 Flash) runs at something like 9-13 tokens/second, with a loooong time to first token. It is not a realistic interactive coding model for agentic use. I like my Strix Halo and keep it chewing on stuff, mostly non-interactive workloads (security audits of software mostly, training experiments, etc.), I get a lot of use out of it. If you want to experiment with AI, it is a good platform for that, though at $4k you can get an Nvidia-based Asus Ascend GX10, which is probably better. But, if you want a local model for interactive agentic use, you're going to be running either Qwen 3.6 or Gemma 4, which will fit comfortably on 2x64GB GPUs (even old GPUs will run them faster than the Strix Halo...I have dual Radeon Pro V620s which are faster, and they're six years old), or snugly on 32GB. A 48GB or 64GB Mac would run them well. Two Radeon AI Pro R9700 GPUs is probably the sweet spot, right now for GPUs. Not the cost of a good used car, like a 5090 or 4090, but plenty of memory and performance for local inference. Also, not finicky and weird and needing custom 3D printed fan shrouds like the old server GPUs on eBay. At the moment, there just isn't a model that works better on a 128GB inference machine like this that don't also work fine on 64GB machines, which may be faster…