Apple Mac Studio M1 Ultra

I host the models on my Mac Studio, an M1 Ultra with 64gb ram (I bought it when it came out, just happens to be good at LLMs). So when I work on my laptop, I have my oh-my-pi setup configured to use the models on my Mac over my local "bonjour" network or whatever Apple calls it. That way I have a nice cool lap, while using models that my M4 MacBook Air with its 16gb ram couldn't possibly run.

For vector generation I started using Meta-LLama-3-8B in april 2024 with Python and Transformers for each text chunk on an RTX-A6000. Wow that thing was fast but noisy and also burns 500W. So a year ago I switched to an M1 Ultra and only had to replace Transformers with Apple's MLX python library. Approximately the same speed but less heat and noise. The Llama model has 4k dimensions so at fp16 thats 8 kilobyte per chunk, which I store in a BLOB column in SQLite via numpy.save(). Between running on the RTX and M1 there is a very small difference in vector output but not enough for me to change retrieval results, regenerate the vectors or change to another LLM. For retrieval I load all the vectors from the SQlite database into a numpy.array and hand it to FAISS. Faiss-gpu was impressively fast on the RTX6000 and faiss-cpu is slower on the M1 Ultra but still fast enough for my purposes (I'm firing a few queries per day, not per minute). For 5 million chunks memory usage is around 40 GB which both fit into the A6000 and easily fits into the 128GB of the M1 Ultra. It works, I'm happy.

> Crazy amount at a crazy high speed That's 300GB/s slower than my old Mac Studio (M1 Ultra). Memory speeds in 2025 remain thouroughly unimpressive outside of high-end GPUs and fully integrated systems.

Get a Mac Studio with however much memory you need, and ideally an Ultra chip (for max memory bandwidth), and there's your workstation. I regularly run quantized 100b+ models on my M1 Ultra with 128Gb RAM.

The gap is widening largely because the maintainer left the project due to the Linux kernel rust drama. But there is still plenty of very talented contributors who have taken over. That, combined with a rapidly growing interest to run Linux on Apple Silicon instead of macOS, will likely result in faster development over the next few years. But for me, the long term momentum of new platforms as they release doesn't matter - what IS supported is excellent (including the brilliant GPU driver). My Mac Studio M1 Ultra runs Fedora Asahi Remix perfectly with full support and is exactly what I need. So those who are looking for a fast Apple Silicon system that can run Linux natively will save $$ and buy a used M1/M2 from Apple or otherwise.

This claim does not appear to hold water. Works fine for me on both an M1 Ultra Mac Studio and M2 Max MacBook Pro running macOS 15.1. Seems like it’s just a bug that the author is experiencing.

Same. I used to upgrade every 1.5 years or so. But with every Apple Silicon generation so far I have felt that there are really no good reasons to upgrade. I have a MacBook M3 Pro for work, but there are no convincing differences compared to the M1 Pro. In fact, I bought a highly discounted Mac Studio with M1 Ultra because the M1 is still so good and it gives me 10Gbit ethernet, 20 cores and a lot of memory. The only thing I am thinking about is going back to the MacBook Air again since I like the lighter form factor. But the display, 24 GiB max RAM and only 2 Thunderbolt ports would be a significant downgrade.

Compared to a GPU like a 4090 with equal vram it probably won't fare well as others point out, however it far outperforms any CPU. On an M1 Ultra MacBook Pro I was seeing like 40 tokens/second with llama3:7B vs 9 tokens/second on various Intel servers/desktops with sufficient ram.

This, and then you get to have a Mac Studio. I have one, where I selected an M1 Ultra and 128G RAM to facilitate just this sort of thing. But in practice, I'm spending much more time using it to edit 4K video, and as a recording studio/to develop audio plugins on, and to livestream while doing these things. Turns out it's good at these things, and since I have the LLAMA 70b language model at home and can run it directly unquantized (not at blinding speed, of course, but it'll run just fine), I'm naturally interested in learning how to fine tune it :)

I got my Mac Studio with M1 Ultra and 128g of RAM specifically for this reason, in case I needed it. Definitely not using its full capacities for any other purpose, though it's nice to be able to run 10 instances of a big data-grinding program at once, overnight. Still waiting on 'consumer' AI training facilities I can practically use: it's a bit arcane, not up to speed on that. I can generate text up to Llama 70b if I'm patient, and generate images and sort-of-video quite easily.

I have a M1 Ultra Mac Studio with 128GB of RAM and the ability to run LLMs like LLaMa2 locally thanks to the unified memory architecture was a big factor. Just because you don't have a use for it doesn't mean everyone else is as blinkered as you. As soon as the M3 Ultra lands, I am getting one, preferably with 256GB RAM.

I was interested in Stable Diffusion / images, and also text generation. I started playing with ComfyUI and Ollama. An M1 studio ultra would generate a 'base' 512x512 image in around 6 seconds, and ollama responses seemed easily 'quick enough'. Faster than I could read. On an I7-3930K, purely CPU only, a similar image would take around 2.5 minutes, and ollama was painful, as I would be waiting for the next word. Then I switched to a 3080ti, which I hadn't been using for gaming as it got stupidly hot and I regretted having it. Suddenly it was redeemed. On the 3080ti, the same images come out in less than a second, and ollama generation is even faster. Sure, I'm limited to 7B models for text (the mac could go much higher) and there will be limits with image size/complexity, but this thing is so much faster than I expected, and hardly generates any heat/noise at the same time - completely different to gaming. This is all a simple install under linux (pop os in this case). tl;dr - A linux PC with a high-end GPU is the best value by far unless you really need big models, in my experience.

> Linux 6.2 adds (checks notes): cpufreq to M1s, devicetree for newer hardware, HDMI out to Mac Studio (2022) and bluetooth support. It's worth noting that these features were long available since July 2022 ( https://asahilinux.org/2022/07/july-2022-release/ ) to Asahi users. And like other Asahi users, I've been using them since that time on my Mac Studio running Asahi (with HDMI output, 10G ethernet, and Bluetooth too): https://triosdevelopers.com/jason.eckert/stuff/AsahiSwayM1Ul...

Also after a few years you can sell and upgrade. A 2022 Mac Studio w/ M1 Ultra and 128gb was ~$5200 new and I see them selling for over $4k on eBay. Can’t sell your used tokens…

Mine blew through $50 of OpenRouter credits in a single day. Not worth it. Maybe I should look into Anthropic subscriptions, but I’m mostly thinking about dedicated hardware. A used Mac Studio M1 Ultra can has I impressive memory bandwidth…

More like 700 € if you are lucky. Prices are still not back down from the start of the AI boom. I am hopeful that the prices will drop a bit more with Intel's recently announced Arc Pro B60 with 24GB VRAM, which unfortunately has only half the memory bandwidth of the RTX 3090. Not sure why other hardware makers are so slow to catch up. Apple really was years ahead of the competition with the M1 Ultra with 800 GB/s memory bandwidth.