Ben Eng on Nostr: Found the most easy peasy way to self-host a LLM with Web UI. 1. Assume you first ...
Found the most easy peasy way to self-host a LLM with Web UI.
1. Assume you first have a python 3.11 venv activated.
2. https://github.com/oobabooga/text-generation-webui - download and extract.
3. sh start_linux.sh to do the setup (you can select the option to use GPU or CPU-only) and then shut it down
4. Download a model (e.g., one of the bin files from https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML/tree/main) to ./oobabooga_linux/text-generation-webui/models
5. export OOBABOOGA_FLAGS='--share' ; sh start_linux.sh to run the server for use, and it prints the URL
6. Point your browser at the URL.
7. Go to Model, select the model you want, and Load.
8. Go to Chat, and start chatting.
1. Assume you first have a python 3.11 venv activated.
2. https://github.com/oobabooga/text-generation-webui - download and extract.
3. sh start_linux.sh to do the setup (you can select the option to use GPU or CPU-only) and then shut it down
4. Download a model (e.g., one of the bin files from https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML/tree/main) to ./oobabooga_linux/text-generation-webui/models
5. export OOBABOOGA_FLAGS='--share' ; sh start_linux.sh to run the server for use, and it prints the URL
6. Point your browser at the URL.
7. Go to Model, select the model you want, and Load.
8. Go to Chat, and start chatting.