Resources

Hugging Face: RL environments and GPU jobs

Maritime runs your reinforcement learning (RL) environments and your agents. Hugging Face Jobs runs the GPU work: the trainer and the model you train. This page connects the two, step by step. Each command on it was run on November 16, 2026.

Made for the NeurIPS 2026 papers with ML Intern hackathon.

Environments on Maritime

Any OpenEnv image, with a public URL, from one API call.

Training on Hugging Face

TRL GRPO on a GPU Job. You pay per second of GPU time.

Agents on Maritime

Always-on agents that start GPU Jobs and report the results.

How it connects

Hugging Face Job

Trainer and policy model

  • TRL GRPO on a GPU (A10G, A100, H100)
  • Billed by Hugging Face, per second
WebSocketreset, then one step per tool call. The reward comes back.
Maritime

RL environment server

  • Any OpenEnv image from the Hub
  • api.maritime.sh/a/<id>
Optional: your agent on Maritime. It is always on. It starts Hugging Face Jobs with your HF token, watches them and reports to you. Its model calls use your Maritime AI credits.

One training step goes like this:

  1. The trainer on the Hugging Face GPU starts one episode per generation (for example 8).
  2. Each episode opens a WebSocket session to the environment URL on Maritime and calls reset.
  3. The model calls tools (for Wordle, guess). Each call is one step on Maritime. The environment returns the feedback and the reward.
  4. The trainer scores each episode and updates the model on the GPU.

Works with any OpenEnv environment

The environment is an ordinary OpenEnv server. Each OpenEnv Space on the Hub has a Docker image at registry.hf.space/<owner>-<space>:latest. To see it, open the Space page and click ⋮, then Run locally.

Before you start

1.Maritime plan

Sign up at maritime.sh, then subscribe to the Growth plan at maritime.sh/billing. Growth runs up to 25 machines awake at once.

2.Maritime API key

Create one at maritime.sh/settings/api-keys, then run export MARITIME_API_KEY=mk_...

3.Hugging Face credits

Jobs run for any user or organization with a positive balance at huggingface.co/settings/billing.

4.Tools on your laptop

pip install -U huggingface_hub, then hf auth login. Also uv and jq.

Run the environment on Maritime

Deploy the environment image as a Maritime service with a public URL. This example uses Wordle, registry.hf.space/openenv-wordle:latest.

deploy.sh
curl -sS -X POST https://api.maritime.sh/api/agents \
  -H "Authorization: Bearer $MARITIME_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "wordle-env",
    "framework": "custom",
    "imageName": "registry.hf.space/openenv-wordle:latest",
    "exposedPort": 8000,
    "publicWeb": true,
    "healthCheckPath": "/health",
    "hasWebUi": false,
    "initialEnvVars": [
      { "key": "MAX_CONCURRENT_ENVS", "value": "64", "isSecret": false }
    ]
  }' | jq '{id, status, publicUrl}'

# {
#   "id": "5dfd9d4d-...",
#   "status": "deploying",
#   "publicUrl": "https://api.maritime.sh/a/5dfd9d4d-..."
# }
  • exposedPort is the port the server listens on. OpenEnv images use 8000.
  • MAX_CONCURRENT_ENVS is how many sessions one copy accepts at once. Set it to the number of games the trainer plays at once (per_device_train_batch_size × gradient_accumulation_steps), or more.
  • Keep the id and the publicUrl. The next steps use them.

Wait until the environment is ready. The first start pulls the image and takes about 2 minutes. Until then, /health returns 503 ("Agent is starting or asleep").

terminal
export ENV_URL=https://api.maritime.sh/a/<id>

# 503 "Agent is starting or asleep" until the image is pulled and booted.
until curl -sf -o /dev/null $ENV_URL/health; do
  sleep 5
done

curl -s $ENV_URL/health
# {"status":"healthy"}

Test the environment

Play a few games before you spend GPU time. This script plays 16 games at the same time.

smoke_test.py
# /// script
# dependencies = [
#     "openenv-textarena @ git+https://huggingface.co/spaces/openenv/wordle",
# ]
# ///
"""Play 16 Wordle games at once against the environment on Maritime."""
import os
from concurrent.futures import ThreadPoolExecutor

from textarena_env import TextArenaAction, TextArenaEnv

ENV_URL = os.environ["ENV_URL"]


def play(i: int) -> str:
    with TextArenaEnv(base_url=ENV_URL).sync() as env:
        env.reset()
        result = env.step(TextArenaAction(message="[crane]"))
        return result.observation.messages[0].content[-60:]


with ThreadPoolExecutor(16) as pool:
    for feedback in pool.map(play, range(16)):
        print(feedback.replace("\n", " | "))
terminal
ENV_URL=$ENV_URL uv run smoke_test.py
# ... Feedback: | C R A N E | X G G Y X | You have 5 guesses left.

WebSocket HTTP 403 or 503?

The environment is not ready yet. Wait for /health to return 200, then run the test again.

Train on a Hugging Face GPU

Save this as train_wordle.py. It is the TRL Wordle recipe, with the environment URL read from ENV_URL. hf jobs uv run installs the dependencies listed at the top of the file.

train_wordle.py
# /// script
# requires-python = ">=3.11"
# dependencies = [
#     "trl",
#     "datasets",
#     "openenv-textarena @ git+https://huggingface.co/spaces/openenv/wordle",
# ]
# ///
"""GRPO on a Hugging Face GPU. The Wordle environment runs on Maritime."""

import os
import re

import torch
from datasets import Dataset
from textarena_env import TextArenaAction, TextArenaEnv
from trl import GRPOConfig, GRPOTrainer

ENV_URL = os.environ["ENV_URL"]  # the Maritime public URL of the env
MODEL = os.environ.get("MODEL", "Qwen/Qwen3-0.6B")
MAX_STEPS = int(os.environ.get("MAX_STEPS", "20"))
FEEDBACK = re.compile(r"Feedback:\n([A-Z ]+)\n([GYX ]+)")

gpu = torch.cuda.get_device_name() if torch.cuda.is_available() else "none"
print("GPU:", gpu)
print("Environment:", ENV_URL)


class WordleEnv:
    """One Wordle game on the Maritime server per generation."""

    def __init__(self):
        self.client = None

    def reset(self, **kwargs) -> str | None:
        # A new connection per episode, so one dropped socket loses one game.
        if self.client is not None:
            self.client.close()
        self.client = TextArenaEnv(base_url=ENV_URL).sync()
        result = self.client.reset()
        self.last = result.observation.messages[0].content
        self.done = False
        self.won = False
        self.best_greens = 0
        return self.last

    def guess(self, guess: str) -> str:
        """
        Make a guess in the Wordle environment.

        Args:
            guess: The guessed word, formatted as '[abcde]'

        Returns:
            The feedback message from the environment.
        """
        if self.done:
            raise ValueError("Game over.")
        result = self.client.step(TextArenaAction(message=guess))
        full = result.observation.messages[0].content
        feedback, self.last = full[len(self.last):], full
        match = FEEDBACK.search(feedback)
        if match:
            greens = match.group(2).count("G")
            self.best_greens = max(self.best_greens, greens)
        self.won = result.done and (result.reward or 0) >= 1.0
        self.done = result.done
        return feedback


def reward(environments, **kwargs) -> list[float]:
    return [
        1.0 if env.won else 0.1 * env.best_greens
        for env in environments
    ]


prompt = (
    "You are playing Wordle. Guess the secret 5-letter word in 6 tries. "
    "Call the tool `guess` with a word in brackets, for example [crane]."
)
messages = [{"role": "user", "content": prompt}]
dataset = Dataset.from_dict({"prompt": [messages] * 256})

trainer = GRPOTrainer(
    model=MODEL,
    reward_funcs=reward,
    train_dataset=dataset,
    args=GRPOConfig(
        output_dir="/tmp/wordle-grpo",
        chat_template_kwargs={"enable_thinking": False},
        num_generations=8,
        per_device_train_batch_size=8,
        max_completion_length=1024,
        max_tool_calling_iterations=6,
        max_steps=MAX_STEPS,
        logging_steps=1,
        report_to="none",
    ),
    environment_factory=WordleEnv,
)
trainer.train()
print("done")

Start it as a Job on an A10G GPU:

terminal
hf jobs uv run \
  --flavor a10g-small \
  --timeout 2h \
  --name wordle-grpo \
  -e ENV_URL=$ENV_URL \
  -e MAX_STEPS=20 \
  --detach \
  train_wordle.py

# Follow the logs: "GPU: NVIDIA A10G", then one line per step
hf jobs logs -f <job_id>

# The status of your Jobs
hf jobs ps
OptionWhat it does
--timeout 2hJobs stop after 30 minutes by default. Set it for every training run.
--flavor a10g-smallThe GPU. hf jobs hardware lists them with prices: a10g-small is $1.00 an hour, a100-large $2.50.
--namespace <org>Runs the Job on an organization’s credits.
--secrets MARITIME_API_KEYLets the Job deploy and delete its own environments. It reads the value from your shell.

For a real Wordle run

The script above is a short test that proves the setup works. For real training, follow the TRL OpenEnv guide: start from openenv/Qwen3-1.7B-Wordle-SFT, and use vLLM in colocate mode on an a100-large GPU.

Run more environments

One environment copy serves MAX_CONCURRENT_ENVS sessions. For more parallel games, or for heavy environments (browsers, desktops), deploy more copies and give each episode a different URL:

train_wordle.py
import itertools
import os

# ENV_URLS="https://api.maritime.sh/a/<id1>,https://api.maritime.sh/a/<id2>"
ENV_URLS = itertools.cycle(os.environ["ENV_URLS"].split(","))


class WordleEnv:
    def reset(self, **kwargs):
        ...
        self.client = TextArenaEnv(base_url=next(ENV_URLS)).sync()

The Growth plan runs up to 25 machines awake at once. Each environment copy and each agent is one machine. GET https://api.maritime.sh/api/v1/account/limits shows your live limits.

Clean up

Delete each environment when you are done. Its disk is deleted too.

terminal
# Returns 204 when the environment is deleted
curl -sS -X DELETE https://api.maritime.sh/api/agents/<id> \
  -H "Authorization: Bearer $MARITIME_API_KEY"

Jobs end by themselves when the script exits. hf jobs cancel <job_id> stops one early.

Agents on Maritime that use Hugging Face GPUs

An agent on Maritime is always on, so it can run the whole loop for you: start a GPU Job, wait for it, read the logs, start the next run, and tell you the result in Telegram, WhatsApp or the dashboard chat. Hugging Face only sees a Job request with your token. The GPU bill goes to your Hugging Face account. The agent and its model calls go to Maritime.

1. Give the agent your Hugging Face token as a secret

terminal
# Give your Maritime agent a Hugging Face token (stored as a secret)
maritime env set my-agent HF_TOKEN=hf_...

2. In the agent (or any Maritime sandbox), launch Jobs

agent shell
# Inside the agent or sandbox (Ubuntu Python needs a venv for pip)
python3 -m venv ~/hf && ~/hf/bin/pip install -q huggingface_hub

# Launch a GPU Job and wait. Exit code 0 means it completed.
~/hf/bin/hf jobs uv run --flavor a10g-small --timeout 2h \
  -e ENV_URL=$ENV_URL train_wordle.py

Or from Python

launch.py
from huggingface_hub import run_uv_job, wait_for_job

job = run_uv_job(
    "train_wordle.py",
    flavor="a10g-small",
    timeout="2h",
    env={"ENV_URL": env_url},
)
print(job.url)

# COMPLETED, ERROR or CANCELED
print(wait_for_job(job_id=job.id).status.stage)
  • Template agents (OpenClaw, Hermes, Claude Code and others) already have OPENAI_BASE_URL and OPENAI_API_KEY for Maritime LLM, so their model calls use your AI credits. See Maritime LLM.
  • To let the agent also deploy environments, give it a Maritime API key: maritime env set my-agent MARITIME_API_KEY=mk_.... Agents do not get one by themselves.

Computer-use environments on desktops

For agents that see a screen and click, use a Maritime desktop as the environment: POST https://api.maritime.sh/api/v1/computers with {"os": "linux"}, then /actions to click and type and /screenshot for the observation. See Linux desktop.

Linux desktops are on the Growth plan. Windows desktops are on Growth and up, once the plan is paid.

Troubleshooting

What you seeWhyWhat to do
503 Agent is starting or asleepThe environment is booting or waking.Poll /health until 200.
WebSocket HTTP 403 right after deployThe environment is not ready yet.Wait for /health to return 200, then connect.
Sessions refused during trainingMore games at once than MAX_CONCURRENT_ENVS.Raise it, or deploy more copies (step 4).
Job ends after 30 minutesThe default Job timeout.Pass --timeout 2h (or longer).
error: externally-managed-environmentUbuntu Python on Maritime.Use a venv: python3 -m venv ~/hf.