Alan Siefert on Nostr: With 64GB RAM I’d guess you could locally run the 70B version just fine with a 4000 ...
With 64GB RAM I’d guess you could locally run the 70B version just fine with a 4000 series Nvidia GPU. Not sure about the 405B version. Supposedly the 5000 series GPUs are releasing beginning of 2025. I wonder if they will have LLMs in mind in their design.
Published at
2024-07-25 19:36:34Event JSON
{
"id": "c083dabc7f8ce016d6344867c73805e5f1fb04a71957b498430b1996c00d8cde",
"pubkey": "f26606e331f128325ec99cd38782edbec134a5f911ea9fada7adce2ca9015b16",
"created_at": 1721936194,
"kind": 1,
"tags": [
[
"e",
"5d54a060fd0b988e59c401a9bfa190659a54c4be9714de8bdb2d92a4ca3b4cd2",
"wss://nostr.mom/",
"root"
],
[
"e",
"b6a5782876bd2da2038a24bc1d0684470283309039c4d2aca2b8618923724842",
"wss://nostr-pub.wellorder.net",
"reply"
],
[
"p",
"46fcbe3065eaf1ae7811465924e48923363ff3f526bd6f73d7c184b16bd8ce4d"
]
],
"content": "With 64GB RAM I’d guess you could locally run the 70B version just fine with a 4000 series Nvidia GPU. Not sure about the 405B version. Supposedly the 5000 series GPUs are releasing beginning of 2025. I wonder if they will have LLMs in mind in their design.",
"sig": "6d6091ec9c6ece5b0bc6b6b83177a0707125ef523e404877b284610629b2f5c6ab770a3d7bc22f0b639b67c924ca5a203f11765857cee23389162b3eff23dff3"
}
