Rm -rf on Nostr: looking into this muself. i have a rather beefy pc with 512g of ram, a30 and 96 cpus, ...
looking into this muself. i have a rather beefy pc with 512g of ram, a30 and 96 cpus, and here is something that i've learned by playing with other models: most large models don't fit into memory of a single gpu, so you will have to buy several high end gpus, or you will have to run the model in ram using cpus. the cpu option is inexpensive compared to several a100's, but I haven't found a way to run it smoothly on a desktop (zen kernel is still laggy as hell), and the response is slower than that of a gpu by an order of magnitude. personally, for me either of these things is a deal killer. Another option is you.com, they will probably introduce the model soon, but than, it's just another corpo.
what worries me about gpt and llama models way more than the privavy consideration, are biases in the training set and training to lie or refuse answers on some "sensitive" topics
Published at
2024-07-26 03:09:43Event JSON
{
"id": "d13d41b58a88051f218f0fac60ff01063edc4d98168c246d7ee6c8ae90f0a5eb",
"pubkey": "11deb685d519a1e1f2c30cad6bf30b7f6cde203440d4cef01065211cec4fa7dc",
"created_at": 1721963383,
"kind": 1,
"tags": [
[
"p",
"46fcbe3065eaf1ae7811465924e48923363ff3f526bd6f73d7c184b16bd8ce4d"
],
[
"e",
"5d54a060fd0b988e59c401a9bfa190659a54c4be9714de8bdb2d92a4ca3b4cd2",
"wss://nostr.mom",
"root"
]
],
"content": "looking into this muself. i have a rather beefy pc with 512g of ram, a30 and 96 cpus, and here is something that i've learned by playing with other models: most large models don't fit into memory of a single gpu, so you will have to buy several high end gpus, or you will have to run the model in ram using cpus. the cpu option is inexpensive compared to several a100's, but I haven't found a way to run it smoothly on a desktop (zen kernel is still laggy as hell), and the response is slower than that of a gpu by an order of magnitude. personally, for me either of these things is a deal killer. Another option is you.com, they will probably introduce the model soon, but than, it's just another corpo.\nwhat worries me about gpt and llama models way more than the privavy consideration, are biases in the training set and training to lie or refuse answers on some \"sensitive\" topics",
"sig": "bbb5b5dbf1da8e889a2d099f1fa85353eaf51543d27e843bb1e43c6d944763a4887590fb9639f5a61e6e177091d99b6ea278e88f73a891b30061999776d30e5b"
}