NVIDIA GeForce RTX 3060

I transcribed between 3000 to 4000 of 10s-30s short videos, every day for almost 2 years for fun. A cheap desktop linux with second hand x-mining RTX 3060 and 3080Ti, connected over home network using basic Gradio and faster-whisper, so they can be exposed as public API and called from corporate network. Relatively easy and much cheaper compared to commercial offerings at the time. These GPUs are over powered for the task and every day only spent 1 to 2 hours of actual encoding, it's so quick, and it's using the biggest whisper model with audio preprocessing and VAD to improve success rate.

It shouldn't be this way, but DDR4 is perfectly fine and will last for years. RAM generally has low failure rates; and prices are relatively better on the used market. If you want ECC, many used DDR4 ECC sticks seem like a 'relative bargain' compared to how much DDR5 ECC goes for. As for GPU, AMD hasn't raised prices by as much; and honestly my 3060 12GB from many, many years ago plays absolutely every single game I want to play at 1440p; with nice visuals. Yeah, I can't pump everything to ultra, and maybe I need some DLSS for the most demanding games, but it's far cry from "old" or "obsolete". In terms of actual experience or enjoyment I get from video games, I'm not missing out on anything. It's legitimate to feel disappointed and upset at the current state of the PC market, but slightly older desktop hardware is still more than plenty fast enough. I own both a MBP M5 Pro 48GB, and a MacBook Neo, and I use my Neo probably 2x as much as my MBP.

Sadly if you want a GPU with good AI performance you gotta go with NVIDIA. It might sound crazy but as a 7900XTX owner.. My 12GB 3060 on my linux server outperformed the 7900XTX by 40%. The 3060 only has half the vram of the AMD card. Proprietary drivers under Arch Linux. On top of the significantly worse software on AMD's side (literally didn't work on windows in particular - so the "performs as good on both systems" is a nonstarter, some GGUF library dependency just doesn't work/exist under AMD on windows). Had me running the AMD card on windows under WSL (not a problem with nvidia though, that ran just fine on windows-side directly). Aaaand also the other AMD bugs, such as the pink squares display corruption that has been an active issue for my GPU in particular (7900XTX) for over a year, maybe approaching two at this point, with no fix in sight from the AMD team (barely and ack at all - not on a single patch notes, just a bunch of reddit discussion). Really regret spending so much on an AMD gpu.

Actually budget friendly is RTX 3060 12Gb. With one you can run 9B/12B models which are fine for text tasks like chatting or summarisation. Not for precision like tool calling or code. With two of them you can run models up to Qwen 27B and 35B with a few-turn context window (8k-16k). Dense at 14t/s and MoE at 68t/s. With three of them you can run 128k context, though you'll need a large format case and the right motherboard or PCIe riser. I'm running three and even with a new case this setup cost me less than one 3090.

Very few questions are stupid, these are not. Yes, you can definitely build an entire fps game using WebGL for rendering. Typically using JavaScript to handle physics, collision, gameplay, etc. My current WebGL project is rendering high definition terrain, high-poly animated models, thousands of particles, shaders, sound and more over 150 frames-per-second on a 10 year old PC with a RTX 3060. I have found hardware acceleration is often not enabled in the browser, or Windows will default to using the integrated-graphics card when running the browser and that must be changed in the Windows Graphics Settings. WebGL is a graphics API for talking right to the graphics card, supported by The Browser. ThreeJS and BabylonJS are libraries that make it easier to render 2D and 3D graphics, both use WebGL and/or WebGPU behind the scenes for rendering. Development with HTML/CSS/JavaScript and WebGL is my favorite stack to work with. Development is fast, re-loading is quick, errors and debugging is handled directly in the browsers which have great debug information and performance tracking. No compile time and support on lots of devices. Yes, OpenGL came first. WebGL is a JavaScript binding of a subset of OpenGL functionalities.

AI video generation can fairly easily choke anything that's not NVIDIA's flagship model. Even the latest local image gen models are so large that they can be frustratingly slow with non-optimal hardware even if they fit in the VRAM. IIRC when I had an M2, it was about 4x slower at running the venerable Stable Diffusion (and SDXL) than my meager RTX 3060.

I didn't really understand the performance table until I saw the top ones were 8B models. But 5 seconds / token is quite slow yeah. I guess this is for low ram machines? I'm pretty sure my 5950x with 128 gb ram can run this faster on the CPU with some layers / prefill on the 3060 gpu I have. I also see that they claim the process is compute bound at 2 seconds/token, but that doesn't seem correct with a 3090?

Personally, I'm getting 15 tok/s on both RTX 3060 and my MacBook Air M4 (w/ 32GB, but 24 should suffice), with the default config from LMStudio. Which I find even more impressive, considering the 3060 is the most used GPU (on Steam) and that M4 Air and future SoCs are/will be commonplace too. (Q4_K_M with filesize=18GB)

my trusty RTX 3060 is gonna have its day in the sun... though I have run a bunch of 7B models fairly easily on Ollama.

If you don't need a big model and are fine with hosting locally, an RTX3060 with 12GB VRAM is going to do just fine. Can be bought for about 200-300 USD. I've been pleasantly surprised by what such a mediocre GPU and Llama3 8B can do for certain (simple) use cases. Ollama makes it all pretty easy.

I inherited a few R420's from my work, and the coolest thing about it is the iDRAC. I'm not a SRE or anything of the sort, so don't get to see stuff like that much, but the utility of the iDRAC is fantastic. The machines each have 192GB of ram, so I thought I'd set them up as LLM inference machines. I figured that with that much ram, I could load just about any model. Then I discovered how slow the CPUs on these older machines is. It was so utterly slow. I have a machine I bought from Costco a few years ago that was under $1k and came with a RTX 3060 with 12GB of GPU ram. That machine can run around 20+/tokens per second on 13B models (I actually don't know - I stream the text, and cap it at 9 tokens per second so I can actually read it). The R420? Its tokens per second were in the 0.005 to 0.01 range. So, yeah, not a good CPU for that sort of task. For other stuff, sure. I thought I'd setup a small file server with one instead, but the fans are so jet engine loud that it's intolerable to have in any part of the house, even when managing fan speeds with software.

These datacenters are already running old, inefficient, slow GPUs from five years ago in addition to newly released cards, because anything newer than that is extremely bottlenecked and they need all the compute they can get. Why should it be any different in five years' time? Even nVidia is rumored to be about to bring back the RTX 3060 which is an Ampere architecture card that got released around 2021. It's just fine.

I dunno, the kiddo went from a 1650 Super to a 3060 and it's a lot nicer looking, I don't think frame gen and what not is enabled. Sure, that's up a notch on the SKU list and tons more VRAM. The 1650 Super was working with most of the games he tried, but Marvel Rivals was terrible (haven't seen him play it with the new card though) It does help that he has a small screen and regular DPI. Seems like everyone wants to run with 4x the pixels in the same space, which needs about 4x the GPU.

You can use a lower-end GPU (like the RTX 3060), which also uses less energy. But you are right, you won't be encountering model API costs when running it locally.

Evidently not a lot- the RTX 3060 would be as low as I go for GOU horsepower with any sort of AI stuff.

Depends on what you consider vaguely useful, but any modern GPU which can fit the weights at a reasonable quantization (so, 16+ GB, maybe 12 in a pinch) will probably be tolerable. Bare minimum is probably ~3060 12GB.

The RTX 3060 graphics card is a bit outdated, but its performance is still good enough for your needs. In China, it's worth around 4,000 RMB (about $550).

2 x Nvidia Tesla P40 card for €660 is not a thing i consider to be "on a budget". People can play with "small" or "medium" models less powerfull and cheaper cards. A Nvidia Geforce RTX 3060 card with "only" 12Gb VRAM can be found around €200-250 on second hand market (and they are around 300~350 new). In my opinion, 48Gb of VRAM is overkill to call it "on a budget", for me this setup is nice but it's for semi-professional or professional usage. There is of course a trade off to use medium or small models, but being "on a budget" is also to do trade off.

A cheap Nvidia GPU with lots of VRAM, like the 3060 12gb model. About the fastest you can expect for the lowest amount of money.