jimbocoin on Nostr: Running LLMs at home, I’ve switched to running them on my gaming laptop. Its ...

Running LLMs at home, I’ve switched to running them on my gaming laptop. Its GeForce RTX 2060 with 6GB vRAM is sufficient to run 7B parameter models.

I’m currently running dolphin-2.9.3-mistral-7B-32K via GPT4ALL. It’s quite fast at ~6 tokens per second. Not quite as high quality as the 22B models I’ve tried, but significantly faster.