Simon Willison on Nostr: Anyone had much success running long context prompts through local LLMs? On my M2 ...
Anyone had much success running long context prompts through local LLMs? On my M2 64GB Mac I'm finding that longer prompts take an unreasonably long time to process, am I holding it wrong? Any models or serving platforms I should try out that might respond reasonably quickly?
Published at
2025-03-09 16:52:05Event JSON
{
"id": "89898a116b2024e3dfda7fe4a05031e4ee2da7ecabfa62a8bf3aa74d557a0910",
"pubkey": "8b0be93ed69c30e9a68159fd384fd8308ce4bbf16c39e840e0803dcb6c08720e",
"created_at": 1741539125,
"kind": 1,
"tags": [
[
"proxy",
"https://fedi.simonwillison.net/users/simon/statuses/114133508104508089",
"activitypub"
]
],
"content": "Anyone had much success running long context prompts through local LLMs? On my M2 64GB Mac I'm finding that longer prompts take an unreasonably long time to process, am I holding it wrong? Any models or serving platforms I should try out that might respond reasonably quickly?",
"sig": "471dfa9edb9a78054a9f55e50f4d62cd660ad6051d08f1ebe5e834f8a0d1dd655c15e81b0cddbdda458ade340d0e2840d7e045ab3b28f5aa742cf6e65dcb2eed"
}
