Usage

All remote features of Genv are available as a subcommand to the command genv remote.

List Environments

You can list active environments on all remote hosts using the command envs. For example:

$ genv remote -H gpu-server-1,gpu-server-1 envs
HOST               ID      USER            NAME            CREATED
gpu-server-1       31739   raz(1003)       project-1       42 minutes ago
gpu-server-2       13811   john(1002)      project-2       2 hours ago

Total 2 environments on 2 hosts

Device Information

You can see device information and availability on all remote hosts using the command devices. For example:

$ genv remote -H gpu-server-1,gpu-server-1 devices
HOST                     TOTAL    AVAILABLE
core-server-1            1        0
core-server-2            2        1

Total 3 devices with 1 available on 2 hosts

Activating an Environment

You can activate an environment on a remote host using the command activate.

You can also specify GPU resource requirements such as device count. Genv will then look for a remote host with enough available resources, connect to it with SSH and automatically create and configure an environment there. For example:

$ genv remote -H gpu-server-1,gpu-server-2 activate --gpus 1 --name my-env
...
(genv) gpu-server-2 $

If Genv can’t find a remote host with enough available resources, the genv remote activate command will fail with the following message:

$ genv remote -H gpu-server-1,gpu-server-2 activate --gpus 4
Cannot find a host with enough available resources

Note

You can see all available resource specification options with the command genv remote activate --help

Enforcement

Genv supports enforcement features on multiple hosts. Check out this document for more information.

Monitoring

Genv supports monitoring features on multiple hosts. Check out this document for more information.

Specifying Remote Hosts

genv remote commands connect to multiple remote hosts and run genv commands on them. You can specify the remote hosts in two ways: using explicit hostnames or using a hostfile.

You can pass multiple hostnames and IP addresses using a comma-separated string to the argument -H or --host. For example:

genv remote -H gpu-server-1,gpu-server-2,192.168.1.42 envs

Alternatively, you can create a text file with the hostnames and IP addresses. For example:

$ cat /etc/genv/hostfile
gpu-server-1
gpu-server-2
192.168.1.42

Then you can pass it to the argument --hostfile. For example:

genv remote --hostfile /etc/genv/hostfile envs

Specifying User Names

It is possible to specify usernames for the SSH connections.

If you are using the same username for all machines pass it as the argument -l or --username. For example:

genv remote -l root -H gpu-server-1,gpu-server-2,192.168.1.42 envs

If you are using different usernames for different machines you can pass them as part of the hostname. For example:

genv remote -H root@gpu-server-1,user@gpu-server-2,192.168.1.42 envs

Note

You can use the user@hostname syntax in hostfiles as well.