NVIDIA GeForce RTX 2060

Local LLMs are pretty powerful and useful for me personally, and the best compute I have locally is a an RTX2060. I certainly have no paid AI subscriptions. Obviously depends what you're doing, and not suitable for everyone, but it's an unhelpful generalisation to say AI in its current form is purely a paid product subscription.

Yup, confirming what pamcake said, 30b with 3b active. I have a laptop with a broken screen and an RTX2060 at my disposal. I can run 12b - 14b dense usably, just, although I think 4b - 8b dense models give me the best tradeoff of speed and usefulness. Larger MOE models with more parameters (20b+) but fewer active (2 - 3b) are sometimes a little bit slower, but are often far more capable.

For a long time I thought my RTX-2060 was just not capable and the other day I did a ffmpeg GPU transcode and was surprised by how well it did. So now I am thinking about putting on some of Google's new Gemma edge models (probably the smallest will work with my 6GB VRAM + 2 GB) setup. I am not a 100% sure what that 2GB is but I think it is borrowing from the system in some manner.

This works very good on my pc! I've got as i3-12100f CPU, 16GB RAM and RTX 2060. Running it with `llm --cuda 4` answers very fast, with a lot of hallucination though :|

Bought a used RTX 2060 12GB on ebay for $250. It works fairly well for Stable Diffusion, but it's a little too slow for LLMs. I'm considering renting a something with more RAM to experiment faster.

I'm not sure of what issue you have with my statement. For me, it is a painless download + sh NVIDIA-....run. I have mostly newer GPUs, though the 3 systems (1 laptop and 2 desktops) with older GTX 750ti and GT 560m run the nouveau driver (as Nvidia dropped support for those). Its a 13 year old laptop, and still running strong (linux though). Desktops are Sandy Bridge based. The RTX2060 and RTX3060 are doing fine with the current drivers. I usually only update when CUDA changes. But yeah, its pretty simple. I can't speak to non-linux OSes generally, though my experiences with windows driver updates have always been fraught with danger. My zen2 laptop has an inbuilt Renior iGPU, and I use it with the NVidia dGPU also built (GTX 1660ti). I leverage the Linux Mint OSes packaging system there for the GPU switcher. I run the AMD on the laptop panel and the NVidia on the external display. Outside of weirdness with kernel 5.13, I've not had any problems with this setup.

I have to say these seem like hobbyist-level products. For example https://coral.ai/products/m2-accelerator-dual-edgetpu can do 8 TOPS, but a 5-year-old RTX 2060 gets you 50 TOPS. A newer H100 gets you 3958 TOPS. Nobody's going to buy 500 of those chips and stick them in 500 M.2 slots to match the performance of a single H100. Do they make any better chips that I missed?