Deploying FastChat locally with litellm and GitHub models
Just some really quick notes on this, it’s pointless and redundant but I’ll need these later
- lm-sys/FastChat: An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.
- LiteLLM Proxy Server (LLM Gateway) | liteLLM
LiteLLM
Create .yaml with a github models model:
model_list:
# - model_name: github-Llama-3.2-11B-Vision-Instruct # Model Alias to use for requests
- model_name: minist # Model Alias to use for requests
litellm_params:
model: github/Ministral-3B
api_key: "os.environ/GITHUB_API_KEY" # ensure you have `GITHUB_API_KEY` in your .env
After setting GITHUB_API_KEY, litellm --config config.yaml
FastChat
- Install FastChat, start the controller:
python3 -m fastchat.serve.controller
- FastChat model config:
{
"minist": {
"model_name": "minist",
"api_base": "http://0.0.0.0:4000/v1",
"api_type": "openai",
"api_key": "whatever",
"anony_only": false
}
}
- Run the webUI
python3 -m fastchat.serve.gradio_web_server_multi --register-api-endpoint-file ../model_config.json- Direct chat now works!
Nel mezzo del deserto posso dire tutto quello che voglio.
comments powered by Disqus