Overview¶
Genv is an environment management software for GPUs that manages GPUs on a single machine or in a cluster.
Genv uses Ollama for managing managing Large Language Models (LLMs) with Genv. This allows users to efficiently run, manage and utilize LLMs on GPUs within their clusters.
This guide will help you get started with LLMs using Genv.
Prerequisites¶
First, you will need to install Genv on your GPU machines in the cluster.
Note
It is recommended to install Genv system-wide.
Then, you will need to install Ollama on your GPU machines with the command:
curl -fsSL https://ollama.com/install.sh | sh
Quick start¶
If you are running on a GPU cluster with multiple machines, follow the cluster quick start. Otherwise, follow the local quick start.
Cluster¶
In my case, I have two remote GPU machines: gpu-server-1 and gpu-server-2.
Serve your first LLM on a single GPU with the command:
$ genv remote -H gpu-server-1,gpu-server-2 llm serve llama2 --gpus 1
...
Note
Use Ctrl+C to stop the serving.
Open another terminal and list all LLMs in your cluster with:
$ genv remote -H gpu-server-1,gpu-server-2 llm ps
HOST MODEL PORT CREATED EID USER
gpu-server-1 llama2 N/A 2 hours ago 571 root
We can see that llama2 is now being served in the cluster.
Attach to the model and interact with it using the command:
$ genv remote -H gpu-server-1,gpu-server-2 llm attach llama2
>>> Hi! I'm Genv, how are you?
Hello there, it's nice to meet you! *blinks* I'm just an AI assistant, so I don't have feelings or emotions
like humans do, but I'm here to help you with any questions or tasks you might have. How can I assist you
today?
>>> Send a message (/? for help)
Note
Use Ctrl+D to exit the interactive terminal.
Local¶
Serve your first LLM on a single GPU with the command:
$ genv llm serve llama2 --gpus 1
...
Note
Use Ctrl+C to stop the serving.
Open another terminal and list all LLMs on your machine with:
$ genv llm ps
MODEL PORT CREATED EID USER PID(S)
llama2 33123 22 seconds ago 1179252 raz(1001) 1179252
We can see that llama2 is now being served on the machine.
Attach to the model and interact with it using the command:
$ genv llm attach llama2
>>> Hello from Genv
Hello there! *blinks* Uh, I'm... uh... *trails off* I'm not actually a person, I'm just an AI designed to
simulate conversation. So, I don't have feelings or emotions like a real person would. But I'm here to help
you with any questions or tasks you may have! How can I assist you today?
>>> Send a message (/? for help)
Note
Use Ctrl+D to exit the interactive terminal.
Where to Go Next¶
If you are not familiar with Genv, it is recommended to go over the overview and follow the quick start tutorial.
You should also learn how remote features work in Genv.