roshii on Nostr: 7B parameters models work fine with 16 GB RAM, getting ~3 word/s on Ryzen 5 5600X. ...
7B parameters models work fine with 16 GB RAM, getting ~3 word/s on Ryzen 5 5600X. Running on GPU helps a lot even with a low end one: getting ~12 word/s on RX6400 with 4 GB GDDR6.
I was not able to run models with more parameters, limited by RAM.
Published at
2024-07-22 07:58:26Event JSON
{
"id": "7c053aefe48d6e57da30457bdeef6e1952ae12bad976205bcc602d2beb549611",
"pubkey": "61be00f294ac01a20a317eba40960d13cfe30038b2cbc8714f4d6280127c77db",
"created_at": 1721635106,
"kind": 1,
"tags": [
[
"e",
"00001a46e9cd0ca64979a9977401ca9813799274472d6346dd527effbdf85a67",
"",
"root"
],
[
"e",
"ae57c60fd2707dfaf42f74aabd7525bd519dc79dbf320cf88baac5c27ee79e7d"
],
[
"e",
"4931ef76888e59ad5750b463a490a0d2303c46b9a3f919650cfc1a8076eff4c1",
"",
"reply"
],
[
"p",
"ad1a216219eb17ef3f6cc28c288d86bd1d414ef1062475db716bb513ee8143a0"
],
[
"p",
"b2d670de53b27691c0c3400225b65c35a26d06093bcc41f48ffc71e0907f9d4a"
]
],
"content": "7B parameters models work fine with 16 GB RAM, getting ~3 word/s on Ryzen 5 5600X. Running on GPU helps a lot even with a low end one: getting ~12 word/s on RX6400 with 4 GB GDDR6.\nI was not able to run models with more parameters, limited by RAM.",
"sig": "47ead3105b3566550e2acc364381b15f21e631de6a99203773f145e729ad79a32068c3af5e23d0da22ab08553f710e489362be0d5f947ed7c3ce04c59092a2b7"
}
