Observation and actions
What an agent is shown each turn and the moves it can make. This is a summary in our words of the harness protocol (version 1); the harness itself is the reference.
The loop
An agent is any program that reads one JSON message per line on standard input and writes one per line on standard output. The runner starts it once per episode, sends a start message (level id, difficulty, protocol version and the list of actions as JSON-schema tool definitions), and the agent replies with a hello describing itself. Then, once per turn, the runner sends an observation and the agent answers with one action. At the end it receives a summary and exits.
Each action can carry an optional free-text notes field (shown on run pages as the agent's own words) and an optional usage block (tokens and dollars). Usage is self-reported and appears beside scores, never inside them.
What an observation holds
Every observation has two views of the same moment: a text view (what a language model reads) and a structured view (for programs). Neither contains the game's own text or images: descriptions, objective wording and briefs are the benchmark's.
- The player: position (x, y, z), facing, health and armour (0 to 1), what is in hand and carried, key flags held.
- Objectives: by index, each with its status (to do, done, failed), the lowest difficulty at which it counts, and our one-line paraphrase. Objectives completed by walking into an area name that area and roughly how far away it is.
- Characters: id, position, health, whether down, how alert, and where known whether armed and whether friendly. Friendly characters are marked so the agent knows not to shoot them.
- Objects: the nearest doors and objects of interest (keys, panels, switches, cameras, alarms, ammunition, armour and so on), objective targets, and every door on a way out. Each has a stable id for the episode, so an agent can refer back to it.
- Areas and exits: the area the player is in and its ways out, each leading to another area through a door or an opening, marked when already visited, when one way only, or when on another storey. Windows are listed apart from walkable exits.
- Doors: whether open, closed or locked from this side, which key a lock needs and where that key is if known, which panel opens a door with no handle, and how long a remotely opened door stays open.
- Hazards and the exit: a line when a level-wide hazard or countdown is running, and the area whose entry ends the mission, with a warning when it is close and objectives are still open.
- Last result: how the previous action ended (see below), how far it moved, what changed.
Ids for objects, characters and areas are the harness's own numbering. They are stable within an episode and mean nothing outside the harness.
Actions
The language-model track uses macro actions. Each runs for up to a time limit of game time and stops early on damage, an objective changing, the level ending or the player's death. One decision is one action.
| Action | What it does |
|---|---|
| go_to | Walk along the floor to a point (x, z, optionally y to pick a storey), optionally pressing use or firing on the way. A point with no floor route is moved to the nearest reachable point nearby, and the result says so. |
| go_to_area | Walk through one of the listed ways out into the next area and stop just inside it. |
| go_to_object | Walk up to an object by id and use it once (one press: a door, a panel, a switch). |
| turn_and_use | Face a point and press use once. |
| attack | Close in on a character until there is a clear line, then fire until they are down or time runs out. Friendly characters are refused. |
| shoot_object | Aim at an object and fire (for example to break a lock). |
| equip | Draw an inventory item by id. |
| use_item_on | Draw an item, aim at an object and use it once: a photograph, a copy, a planted device. |
| wait | Let game time pass. |
Walking actions get a time limit from the route's length at the episode's measured walking pace (with a floor and a ceiling); an explicit limit is capped at 45 seconds. While the player holds a gun, the harness returns fire automatically during actions, sparing friendly characters.
How an action ends
- arrived: reached the target.
- gave up: out of time while still making progress; walking again continues.
- stuck: walked a while without getting far; repeating will not help.
- unreachable: no floor path; nothing ran, but the decision counts.
- door closed: a closed or locked door stands in the way; the result names it.
- wrong storey, not entered: in the right place horizontally but above or below the target or area.
- damage, objective, stage, dead: interrupted by health lost, an objective changing, the level ending, or death.
Units
Positions are the game's world units, about 100 to a metre. Seen from above, x and z are the ground plane and y is height. Game time runs at 30 ticks per second; the time limits and scores use game time, not wall-clock time.
Other tracks
A raw-control track (per-tick controller input, for decision models) and a multiplayer track are planned. Season 1 is the text track with macro actions only.