Resources
Hugging Face: RL environments and GPU jobs
Maritime runs your reinforcement learning (RL) environments and your agents. Hugging Face Jobs runs the GPU work: the trainer and the model you train. This page connects the two, step by step. Each command on it was run on November 16, 2026.
Made for the NeurIPS 2026 papers with ML Intern hackathon.
Environments on Maritime
Any OpenEnv image, with a public URL, from one API call.
Training on Hugging Face
TRL GRPO on a GPU Job. You pay per second of GPU time.
Agents on Maritime
Always-on agents that start GPU Jobs and report the results.
How it connects
Trainer and policy model
- TRL GRPO on a GPU (A10G, A100, H100)
- Billed by Hugging Face, per second
reset, then one step per tool call. The reward comes back.RL environment server
- Any OpenEnv image from the Hub
- api.maritime.sh/a/<id>
One training step goes like this:
- The trainer on the Hugging Face GPU starts one episode per generation (for example 8).
- Each episode opens a WebSocket session to the environment URL on Maritime and calls
reset. - The model calls tools (for Wordle,
guess). Each call is onestepon Maritime. The environment returns the feedback and the reward. - The trainer scores each episode and updates the model on the GPU.
Works with any OpenEnv environment
registry.hf.space/<owner>-<space>:latest. To see it, open the Space page and click ⋮, then Run locally.Before you start
1.Maritime plan
Sign up at maritime.sh, then subscribe to the Growth plan at maritime.sh/billing. Growth runs up to 25 machines awake at once.
2.Maritime API key
Create one at maritime.sh/settings/api-keys, then run export MARITIME_API_KEY=mk_...
3.Hugging Face credits
Jobs run for any user or organization with a positive balance at huggingface.co/settings/billing.
4.Tools on your laptop
pip install -U huggingface_hub, then hf auth login. Also uv and jq.
Run the environment on Maritime
Deploy the environment image as a Maritime service with a public URL. This example uses Wordle, registry.hf.space/openenv-wordle:latest.
curl -sS -X POST https://api.maritime.sh/api/agents \
-H "Authorization: Bearer $MARITIME_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "wordle-env",
"framework": "custom",
"imageName": "registry.hf.space/openenv-wordle:latest",
"exposedPort": 8000,
"publicWeb": true,
"healthCheckPath": "/health",
"hasWebUi": false,
"initialEnvVars": [
{ "key": "MAX_CONCURRENT_ENVS", "value": "64", "isSecret": false }
]
}' | jq '{id, status, publicUrl}'
# {
# "id": "5dfd9d4d-...",
# "status": "deploying",
# "publicUrl": "https://api.maritime.sh/a/5dfd9d4d-..."
# }exposedPortis the port the server listens on. OpenEnv images use 8000.MAX_CONCURRENT_ENVSis how many sessions one copy accepts at once. Set it to the number of games the trainer plays at once (per_device_train_batch_size×gradient_accumulation_steps), or more.- Keep the
idand thepublicUrl. The next steps use them.
Wait until the environment is ready. The first start pulls the image and takes about 2 minutes. Until then, /health returns 503 ("Agent is starting or asleep").
export ENV_URL=https://api.maritime.sh/a/<id>
# 503 "Agent is starting or asleep" until the image is pulled and booted.
until curl -sf -o /dev/null $ENV_URL/health; do
sleep 5
done
curl -s $ENV_URL/health
# {"status":"healthy"}Test the environment
Play a few games before you spend GPU time. This script plays 16 games at the same time.
# /// script
# dependencies = [
# "openenv-textarena @ git+https://huggingface.co/spaces/openenv/wordle",
# ]
# ///
"""Play 16 Wordle games at once against the environment on Maritime."""
import os
from concurrent.futures import ThreadPoolExecutor
from textarena_env import TextArenaAction, TextArenaEnv
ENV_URL = os.environ["ENV_URL"]
def play(i: int) -> str:
with TextArenaEnv(base_url=ENV_URL).sync() as env:
env.reset()
result = env.step(TextArenaAction(message="[crane]"))
return result.observation.messages[0].content[-60:]
with ThreadPoolExecutor(16) as pool:
for feedback in pool.map(play, range(16)):
print(feedback.replace("\n", " | "))ENV_URL=$ENV_URL uv run smoke_test.py
# ... Feedback: | C R A N E | X G G Y X | You have 5 guesses left.WebSocket HTTP 403 or 503?
/health to return 200, then run the test again.Train on a Hugging Face GPU
Save this as train_wordle.py. It is the TRL Wordle recipe, with the environment URL read from ENV_URL. hf jobs uv run installs the dependencies listed at the top of the file.
# /// script
# requires-python = ">=3.11"
# dependencies = [
# "trl",
# "datasets",
# "openenv-textarena @ git+https://huggingface.co/spaces/openenv/wordle",
# ]
# ///
"""GRPO on a Hugging Face GPU. The Wordle environment runs on Maritime."""
import os
import re
import torch
from datasets import Dataset
from textarena_env import TextArenaAction, TextArenaEnv
from trl import GRPOConfig, GRPOTrainer
ENV_URL = os.environ["ENV_URL"] # the Maritime public URL of the env
MODEL = os.environ.get("MODEL", "Qwen/Qwen3-0.6B")
MAX_STEPS = int(os.environ.get("MAX_STEPS", "20"))
FEEDBACK = re.compile(r"Feedback:\n([A-Z ]+)\n([GYX ]+)")
gpu = torch.cuda.get_device_name() if torch.cuda.is_available() else "none"
print("GPU:", gpu)
print("Environment:", ENV_URL)
class WordleEnv:
"""One Wordle game on the Maritime server per generation."""
def __init__(self):
self.client = None
def reset(self, **kwargs) -> str | None:
# A new connection per episode, so one dropped socket loses one game.
if self.client is not None:
self.client.close()
self.client = TextArenaEnv(base_url=ENV_URL).sync()
result = self.client.reset()
self.last = result.observation.messages[0].content
self.done = False
self.won = False
self.best_greens = 0
return self.last
def guess(self, guess: str) -> str:
"""
Make a guess in the Wordle environment.
Args:
guess: The guessed word, formatted as '[abcde]'
Returns:
The feedback message from the environment.
"""
if self.done:
raise ValueError("Game over.")
result = self.client.step(TextArenaAction(message=guess))
full = result.observation.messages[0].content
feedback, self.last = full[len(self.last):], full
match = FEEDBACK.search(feedback)
if match:
greens = match.group(2).count("G")
self.best_greens = max(self.best_greens, greens)
self.won = result.done and (result.reward or 0) >= 1.0
self.done = result.done
return feedback
def reward(environments, **kwargs) -> list[float]:
return [
1.0 if env.won else 0.1 * env.best_greens
for env in environments
]
prompt = (
"You are playing Wordle. Guess the secret 5-letter word in 6 tries. "
"Call the tool `guess` with a word in brackets, for example [crane]."
)
messages = [{"role": "user", "content": prompt}]
dataset = Dataset.from_dict({"prompt": [messages] * 256})
trainer = GRPOTrainer(
model=MODEL,
reward_funcs=reward,
train_dataset=dataset,
args=GRPOConfig(
output_dir="/tmp/wordle-grpo",
chat_template_kwargs={"enable_thinking": False},
num_generations=8,
per_device_train_batch_size=8,
max_completion_length=1024,
max_tool_calling_iterations=6,
max_steps=MAX_STEPS,
logging_steps=1,
report_to="none",
),
environment_factory=WordleEnv,
)
trainer.train()
print("done")Start it as a Job on an A10G GPU:
hf jobs uv run \
--flavor a10g-small \
--timeout 2h \
--name wordle-grpo \
-e ENV_URL=$ENV_URL \
-e MAX_STEPS=20 \
--detach \
train_wordle.py
# Follow the logs: "GPU: NVIDIA A10G", then one line per step
hf jobs logs -f <job_id>
# The status of your Jobs
hf jobs ps| Option | What it does |
|---|---|
--timeout 2h | Jobs stop after 30 minutes by default. Set it for every training run. |
--flavor a10g-small | The GPU. hf jobs hardware lists them with prices: a10g-small is $1.00 an hour, a100-large $2.50. |
--namespace <org> | Runs the Job on an organization’s credits. |
--secrets MARITIME_API_KEY | Lets the Job deploy and delete its own environments. It reads the value from your shell. |
For a real Wordle run
openenv/Qwen3-1.7B-Wordle-SFT, and use vLLM in colocate mode on an a100-large GPU.Run more environments
One environment copy serves MAX_CONCURRENT_ENVS sessions. For more parallel games, or for heavy environments (browsers, desktops), deploy more copies and give each episode a different URL:
import itertools
import os
# ENV_URLS="https://api.maritime.sh/a/<id1>,https://api.maritime.sh/a/<id2>"
ENV_URLS = itertools.cycle(os.environ["ENV_URLS"].split(","))
class WordleEnv:
def reset(self, **kwargs):
...
self.client = TextArenaEnv(base_url=next(ENV_URLS)).sync()The Growth plan runs up to 25 machines awake at once. Each environment copy and each agent is one machine. GET https://api.maritime.sh/api/v1/account/limits shows your live limits.
Clean up
Delete each environment when you are done. Its disk is deleted too.
# Returns 204 when the environment is deleted
curl -sS -X DELETE https://api.maritime.sh/api/agents/<id> \
-H "Authorization: Bearer $MARITIME_API_KEY"Jobs end by themselves when the script exits. hf jobs cancel <job_id> stops one early.
Agents on Maritime that use Hugging Face GPUs
An agent on Maritime is always on, so it can run the whole loop for you: start a GPU Job, wait for it, read the logs, start the next run, and tell you the result in Telegram, WhatsApp or the dashboard chat. Hugging Face only sees a Job request with your token. The GPU bill goes to your Hugging Face account. The agent and its model calls go to Maritime.
1. Give the agent your Hugging Face token as a secret
# Give your Maritime agent a Hugging Face token (stored as a secret)
maritime env set my-agent HF_TOKEN=hf_...2. In the agent (or any Maritime sandbox), launch Jobs
# Inside the agent or sandbox (Ubuntu Python needs a venv for pip)
python3 -m venv ~/hf && ~/hf/bin/pip install -q huggingface_hub
# Launch a GPU Job and wait. Exit code 0 means it completed.
~/hf/bin/hf jobs uv run --flavor a10g-small --timeout 2h \
-e ENV_URL=$ENV_URL train_wordle.pyOr from Python
from huggingface_hub import run_uv_job, wait_for_job
job = run_uv_job(
"train_wordle.py",
flavor="a10g-small",
timeout="2h",
env={"ENV_URL": env_url},
)
print(job.url)
# COMPLETED, ERROR or CANCELED
print(wait_for_job(job_id=job.id).status.stage)- Template agents (OpenClaw, Hermes, Claude Code and others) already have
OPENAI_BASE_URLandOPENAI_API_KEYfor Maritime LLM, so their model calls use your AI credits. See Maritime LLM. - To let the agent also deploy environments, give it a Maritime API key:
maritime env set my-agent MARITIME_API_KEY=mk_.... Agents do not get one by themselves.
Computer-use environments on desktops
For agents that see a screen and click, use a Maritime desktop as the environment: POST https://api.maritime.sh/api/v1/computers with {"os": "linux"}, then /actions to click and type and /screenshot for the observation. See Linux desktop.
Linux desktops are on the Growth plan. Windows desktops are on Growth and up, once the plan is paid.
Troubleshooting
| What you see | Why | What to do |
|---|---|---|
503 Agent is starting or asleep | The environment is booting or waking. | Poll /health until 200. |
| WebSocket HTTP 403 right after deploy | The environment is not ready yet. | Wait for /health to return 200, then connect. |
| Sessions refused during training | More games at once than MAX_CONCURRENT_ENVS. | Raise it, or deploy more copies (step 4). |
| Job ends after 30 minutes | The default Job timeout. | Pass --timeout 2h (or longer). |
error: externally-managed-environment | Ubuntu Python on Maritime. | Use a venv: python3 -m venv ~/hf. |