Agentick Leaderboard

Universal benchmark for evaluating AI agents.

ONS (Oracle-Normalized Score) = (agent − random) / (oracle − random), where 0.0 = random baseline and 1.0 = oracle upper bound.

Oracle-Normalized Score (ONS)

Category ONS Breakdown

Rank	Agent	Type	Modality	Harness	Score	95% CI	Open	Date
1	Oracle Agent	other	–	–	0.895	0.811–0.969	No	2026-03-17
2	GPT-6 Astra (high, 1 seed)	llm	ascii	markovian_reasoner	0.634	0.480–0.766	No	2026-09-15
3	Claude Opus 5 (high, 1 seed)	llm	ascii	markovian_reasoner	0.628	0.501–0.734	No	2026-09-16
4	GPT-5.6 Sol (high, 1 seed)	llm	ascii	markovian_reasoner	0.578	0.462–0.695	No	2026-09-16
5	GPT-5.6 Terra (high, 1 seed)	llm	ascii	markovian_reasoner	0.566	0.473–0.658	No	2026-09-16
6	GPT-5.6 Luna (high, 1 seed)	llm	ascii	markovian_reasoner	0.553	0.479–0.627	No	2026-09-16
7	Qwen3.5-4B (SFT-250k)	llm	ascii	markovian_zero_shot	0.447	0.380–0.518	Yes	2026-05-11
8	Qwen3.5-4B (SFT-250k)	llm	ascii	markovian_reasoner	0.444	0.372–0.518	Yes	2026-05-11
9	Qwen3.5-4B (SFT-120k)	llm	ascii	markovian_zero_shot	0.354	0.294–0.408	Yes	2026-05-11
10	Qwen3.5-4B (SFT-120k)	llm	ascii	markovian_reasoner	0.349	0.290–0.402	Yes	2026-05-11
11	GPT-5 mini	llm	ascii	MarkovianReasoner	0.309	0.000–0.000	No	2026-03-20
12	PPO Dense (2M)	rl	rgb_array	–	0.287	0.212–0.367	Yes	2026-03-22
13	Qwen3.5-4B	llm	ascii	markovian_reasoner	0.228	0.161–0.292	Yes	2026-03-25
14	PPO Dense (500k)	rl	rgb_array	–	0.226	0.166–0.287	Yes	2026-03-20
15	Gemini 2.5 Flash Lite	llm	ascii	markovian_reasoner	0.187	0.126–0.247	No	2026-03-17
16	Qwen3.5-4B	llm	language	markovian_reasoner	0.181	0.085–0.272	Yes	2026-03-25
17	Qwen3.5-2B	llm	ascii	markovian_reasoner	0.133	0.069–0.195	Yes	2026-03-25
18	Qwen3.5-2B	llm	language	markovian_reasoner	0.122	0.056–0.183	Yes	2026-03-25
19	Qwen3.5-0.8B	llm	ascii	markovian_reasoner	0.094	0.048–0.140	Yes	2026-03-22
20	Qwen3.5-4B	vlm	rgb_array	markovian_zero_shot	0.093	0.046–0.137	Yes	2026-07-24
21	Qwen3-4B	llm	ascii	markovian_reasoner	0.085	0.000–0.000	Yes	2026-03-21
22	Random Agent	other	–	–	0.082	0.031–0.130	No	2026-03-17
23	PPO Sparse (500k)	rl	rgb_array	–	0.074	0.051–0.097	Yes	2026-03-20
24	Gemini 2.5 Flash Lite	llm	language	markovian_zero_shot	0.064	0.025–0.102	No	2026-03-17
25	Qwen3.5-2B	llm	ascii	markovian_zero_shot	0.062	0.032–0.093	Yes	2026-03-22
26	Qwen3.5-0.8B	llm	language	markovian_reasoner	0.061	0.021–0.100	Yes	2026-03-22
27	Gemini 2.5 Flash Lite	llm	ascii	markovian_zero_shot	0.053	0.022–0.080	No	2026-03-17
28	Qwen3-4B	llm	language	markovian_reasoner	0.050	0.000–0.000	Yes	2026-03-21
29	Qwen3.5-2B	llm	language	markovian_zero_shot	0.031	0.012–0.050	Yes	2026-03-22
30	Qwen3.5-4B	llm	ascii	markovian_zero_shot	0.023	0.007–0.042	Yes	2026-03-22
31	Qwen3.5-4B	llm	language	markovian_zero_shot	0.020	0.006–0.038	Yes	2026-03-22
32	Qwen3.5-0.8B	llm	ascii	markovian_zero_shot	0.020	0.006–0.035	Yes	2026-03-25
33	Qwen3-4B	llm	ascii	markovian_zero_shot	0.020	0.000–0.000	Yes	2026-03-21
34	Qwen3-4B	llm	language	markovian_zero_shot	0.019	0.000–0.000	Yes	2026-03-21
35	Qwen3.5-0.8B	llm	language	markovian_zero_shot	0.016	0.002–0.035	Yes	2026-03-22

Generalization ONS

Rank	Agent	Type	Modality	Harness	Score
1	Oracle Agent	other	–	–	0.837
2	GPT-6 Astra (high, 1 seed)	llm	ascii	markovian_reasoner	0.583
3	Claude Opus 5 (high, 1 seed)	llm	ascii	markovian_reasoner	0.583
4	GPT-5.6 Sol (high, 1 seed)	llm	ascii	markovian_reasoner	0.417
5	GPT-5.6 Terra (high, 1 seed)	llm	ascii	markovian_reasoner	0.500
6	GPT-5.6 Luna (high, 1 seed)	llm	ascii	markovian_reasoner	0.500
7	Qwen3.5-4B (SFT-250k)	llm	ascii	markovian_zero_shot	0.350
8	Qwen3.5-4B (SFT-250k)	llm	ascii	markovian_reasoner	0.337
9	Qwen3.5-4B (SFT-120k)	llm	ascii	markovian_zero_shot	0.357
10	Qwen3.5-4B (SFT-120k)	llm	ascii	markovian_reasoner	0.347
11	GPT-5 mini	llm	ascii	MarkovianReasoner	0.437
12	PPO Dense (2M)	rl	rgb_array	–	0.163
13	Qwen3.5-4B	llm	ascii	markovian_reasoner	0.327
14	PPO Dense (500k)	rl	rgb_array	–	0.130
15	Gemini 2.5 Flash Lite	llm	ascii	markovian_reasoner	0.287
16	Qwen3.5-4B	llm	language	markovian_reasoner	0.340
17	Qwen3.5-2B	llm	ascii	markovian_reasoner	0.133
18	Qwen3.5-2B	llm	language	markovian_reasoner	0.170
19	Qwen3.5-0.8B	llm	ascii	markovian_reasoner	0.143
20	Qwen3.5-4B	vlm	rgb_array	markovian_zero_shot	0.160
21	Qwen3-4B	llm	ascii	markovian_reasoner	0.133
22	Random Agent	other	–	–	0.150
23	PPO Sparse (500k)	rl	rgb_array	–	0.040
24	Gemini 2.5 Flash Lite	llm	language	markovian_zero_shot	0.123
25	Qwen3.5-2B	llm	ascii	markovian_zero_shot	0.120
26	Qwen3.5-0.8B	llm	language	markovian_reasoner	0.130
27	Gemini 2.5 Flash Lite	llm	ascii	markovian_zero_shot	0.087
28	Qwen3-4B	llm	language	markovian_reasoner	0.120
29	Qwen3.5-2B	llm	language	markovian_zero_shot	0.067
30	Qwen3.5-4B	llm	ascii	markovian_zero_shot	0.020
31	Qwen3.5-4B	llm	language	markovian_zero_shot	0.027
32	Qwen3.5-0.8B	llm	ascii	markovian_zero_shot	0.043
33	Qwen3-4B	llm	ascii	markovian_zero_shot	0.017
34	Qwen3-4B	llm	language	markovian_zero_shot	0.030
35	Qwen3.5-0.8B	llm	language	markovian_zero_shot	0.033

Memory ONS

Rank	Agent	Type	Modality	Harness	Score
1	Oracle Agent	other	–	–	0.980
2	GPT-6 Astra (high, 1 seed)	llm	ascii	markovian_reasoner	0.375
3	Claude Opus 5 (high, 1 seed)	llm	ascii	markovian_reasoner	0.500
4	GPT-5.6 Sol (high, 1 seed)	llm	ascii	markovian_reasoner	0.438
5	GPT-5.6 Terra (high, 1 seed)	llm	ascii	markovian_reasoner	0.438
6	GPT-5.6 Luna (high, 1 seed)	llm	ascii	markovian_reasoner	0.438
7	Qwen3.5-4B (SFT-250k)	llm	ascii	markovian_zero_shot	0.412
8	Qwen3.5-4B (SFT-250k)	llm	ascii	markovian_reasoner	0.422
9	Qwen3.5-4B (SFT-120k)	llm	ascii	markovian_zero_shot	0.287
10	Qwen3.5-4B (SFT-120k)	llm	ascii	markovian_reasoner	0.292
11	GPT-5 mini	llm	ascii	MarkovianReasoner	0.347
12	PPO Dense (2M)	rl	rgb_array	–	0.282
13	Qwen3.5-4B	llm	ascii	markovian_reasoner	0.247
14	PPO Dense (500k)	rl	rgb_array	–	0.228
15	Gemini 2.5 Flash Lite	llm	ascii	markovian_reasoner	0.163
16	Qwen3.5-4B	llm	language	markovian_reasoner	0.295
17	Qwen3.5-2B	llm	ascii	markovian_reasoner	0.212
18	Qwen3.5-2B	llm	language	markovian_reasoner	0.135
19	Qwen3.5-0.8B	llm	ascii	markovian_reasoner	0.133
20	Qwen3.5-4B	vlm	rgb_array	markovian_zero_shot	0.135
21	Qwen3-4B	llm	ascii	markovian_reasoner	0.152
22	Random Agent	other	–	–	0.117
23	PPO Sparse (500k)	rl	rgb_array	–	0.102
24	Gemini 2.5 Flash Lite	llm	language	markovian_zero_shot	0.105
25	Qwen3.5-2B	llm	ascii	markovian_zero_shot	0.065
26	Qwen3.5-0.8B	llm	language	markovian_reasoner	0.090
27	Gemini 2.5 Flash Lite	llm	ascii	markovian_zero_shot	0.083
28	Qwen3-4B	llm	language	markovian_reasoner	0.050
29	Qwen3.5-2B	llm	language	markovian_zero_shot	0.055
30	Qwen3.5-4B	llm	ascii	markovian_zero_shot	0.037
31	Qwen3.5-4B	llm	language	markovian_zero_shot	0.020
32	Qwen3.5-0.8B	llm	ascii	markovian_zero_shot	0.030
33	Qwen3-4B	llm	ascii	markovian_zero_shot	0.025
34	Qwen3-4B	llm	language	markovian_zero_shot	0.015
35	Qwen3.5-0.8B	llm	language	markovian_zero_shot	0.003

Multi Agent ONS

Rank	Agent	Type	Modality	Harness	Score
1	Oracle Agent	other	–	–	0.692
2	GPT-6 Astra (high, 1 seed)	llm	ascii	markovian_reasoner	0.450
3	Claude Opus 5 (high, 1 seed)	llm	ascii	markovian_reasoner	0.400
4	GPT-5.6 Sol (high, 1 seed)	llm	ascii	markovian_reasoner	0.400
5	GPT-5.6 Terra (high, 1 seed)	llm	ascii	markovian_reasoner	0.450
6	GPT-5.6 Luna (high, 1 seed)	llm	ascii	markovian_reasoner	0.500
7	Qwen3.5-4B (SFT-250k)	llm	ascii	markovian_zero_shot	0.348
8	Qwen3.5-4B (SFT-250k)	llm	ascii	markovian_reasoner	0.338
9	Qwen3.5-4B (SFT-120k)	llm	ascii	markovian_zero_shot	0.246
10	Qwen3.5-4B (SFT-120k)	llm	ascii	markovian_reasoner	0.238
11	GPT-5 mini	llm	ascii	MarkovianReasoner	0.150
12	PPO Dense (2M)	rl	rgb_array	–	0.432
13	Qwen3.5-4B	llm	ascii	markovian_reasoner	0.134
14	PPO Dense (500k)	rl	rgb_array	–	0.352
15	Gemini 2.5 Flash Lite	llm	ascii	markovian_reasoner	0.098
16	Qwen3.5-4B	llm	language	markovian_reasoner	0.072
17	Qwen3.5-2B	llm	ascii	markovian_reasoner	0.032
18	Qwen3.5-2B	llm	language	markovian_reasoner	0.050
19	Qwen3.5-0.8B	llm	ascii	markovian_reasoner	0.036
20	Qwen3.5-4B	vlm	rgb_array	markovian_zero_shot	0.056
21	Qwen3-4B	llm	ascii	markovian_reasoner	0.038
22	Random Agent	other	–	–	0.030
23	PPO Sparse (500k)	rl	rgb_array	–	0.066
24	Gemini 2.5 Flash Lite	llm	language	markovian_zero_shot	0.022
25	Qwen3.5-2B	llm	ascii	markovian_zero_shot	0.022
26	Qwen3.5-0.8B	llm	language	markovian_reasoner	0.014
27	Gemini 2.5 Flash Lite	llm	ascii	markovian_zero_shot	0.020
28	Qwen3-4B	llm	language	markovian_reasoner	0.020
29	Qwen3.5-2B	llm	language	markovian_zero_shot	0.006
30	Qwen3.5-4B	llm	ascii	markovian_zero_shot	0.008
31	Qwen3.5-4B	llm	language	markovian_zero_shot	0.006
32	Qwen3.5-0.8B	llm	ascii	markovian_zero_shot	0.004
33	Qwen3-4B	llm	ascii	markovian_zero_shot	0.008
34	Qwen3-4B	llm	language	markovian_zero_shot	0.006
35	Qwen3.5-0.8B	llm	language	markovian_zero_shot	0.004

Navigation ONS

Rank	Agent	Type	Modality	Harness	Score
1	Oracle Agent	other	–	–	0.975
2	GPT-6 Astra (high, 1 seed)	llm	ascii	markovian_reasoner	0.781
3	Claude Opus 5 (high, 1 seed)	llm	ascii	markovian_reasoner	0.781
4	GPT-5.6 Sol (high, 1 seed)	llm	ascii	markovian_reasoner	0.719
5	GPT-5.6 Terra (high, 1 seed)	llm	ascii	markovian_reasoner	0.750
6	GPT-5.6 Luna (high, 1 seed)	llm	ascii	markovian_reasoner	0.688
7	Qwen3.5-4B (SFT-250k)	llm	ascii	markovian_zero_shot	0.568
8	Qwen3.5-4B (SFT-250k)	llm	ascii	markovian_reasoner	0.545
9	Qwen3.5-4B (SFT-120k)	llm	ascii	markovian_zero_shot	0.427
10	Qwen3.5-4B (SFT-120k)	llm	ascii	markovian_reasoner	0.425
11	GPT-5 mini	llm	ascii	MarkovianReasoner	0.456
12	PPO Dense (2M)	rl	rgb_array	–	0.250
13	Qwen3.5-4B	llm	ascii	markovian_reasoner	0.223
14	PPO Dense (500k)	rl	rgb_array	–	0.193
15	Gemini 2.5 Flash Lite	llm	ascii	markovian_reasoner	0.237
16	Qwen3.5-4B	llm	language	markovian_reasoner	0.136
17	Qwen3.5-2B	llm	ascii	markovian_reasoner	0.136
18	Qwen3.5-2B	llm	language	markovian_reasoner	0.128
19	Qwen3.5-0.8B	llm	ascii	markovian_reasoner	0.069
20	Qwen3.5-4B	vlm	rgb_array	markovian_zero_shot	0.050
21	Qwen3-4B	llm	ascii	markovian_reasoner	0.072
22	Random Agent	other	–	–	0.040
23	PPO Sparse (500k)	rl	rgb_array	–	0.089
24	Gemini 2.5 Flash Lite	llm	language	markovian_zero_shot	0.037
25	Qwen3.5-2B	llm	ascii	markovian_zero_shot	0.055
26	Qwen3.5-0.8B	llm	language	markovian_reasoner	0.033
27	Gemini 2.5 Flash Lite	llm	ascii	markovian_zero_shot	0.036
28	Qwen3-4B	llm	language	markovian_reasoner	0.040
29	Qwen3.5-2B	llm	language	markovian_zero_shot	0.024
30	Qwen3.5-4B	llm	ascii	markovian_zero_shot	0.013
31	Qwen3.5-4B	llm	language	markovian_zero_shot	0.007
32	Qwen3.5-0.8B	llm	ascii	markovian_zero_shot	0.005
33	Qwen3-4B	llm	ascii	markovian_zero_shot	0.007
34	Qwen3-4B	llm	language	markovian_zero_shot	0.007
35	Qwen3.5-0.8B	llm	language	markovian_zero_shot	0.001

Planning ONS

Rank	Agent	Type	Modality	Harness	Score
1	Oracle Agent	other	–	–	0.928
2	GPT-6 Astra (high, 1 seed)	llm	ascii	markovian_reasoner	0.833
3	Claude Opus 5 (high, 1 seed)	llm	ascii	markovian_reasoner	0.722
4	GPT-5.6 Sol (high, 1 seed)	llm	ascii	markovian_reasoner	0.778
5	GPT-5.6 Terra (high, 1 seed)	llm	ascii	markovian_reasoner	0.694
6	GPT-5.6 Luna (high, 1 seed)	llm	ascii	markovian_reasoner	0.694
7	Qwen3.5-4B (SFT-250k)	llm	ascii	markovian_zero_shot	0.557
8	Qwen3.5-4B (SFT-250k)	llm	ascii	markovian_reasoner	0.573
9	Qwen3.5-4B (SFT-120k)	llm	ascii	markovian_zero_shot	0.459
10	Qwen3.5-4B (SFT-120k)	llm	ascii	markovian_reasoner	0.450
11	GPT-5 mini	llm	ascii	MarkovianReasoner	0.334
12	PPO Dense (2M)	rl	rgb_array	–	0.402
13	Qwen3.5-4B	llm	ascii	markovian_reasoner	0.313
14	PPO Dense (500k)	rl	rgb_array	–	0.300
15	Gemini 2.5 Flash Lite	llm	ascii	markovian_reasoner	0.249
16	Qwen3.5-4B	llm	language	markovian_reasoner	0.219
17	Qwen3.5-2B	llm	ascii	markovian_reasoner	0.237
18	Qwen3.5-2B	llm	language	markovian_reasoner	0.244
19	Qwen3.5-0.8B	llm	ascii	markovian_reasoner	0.164
20	Qwen3.5-4B	vlm	rgb_array	markovian_zero_shot	0.147
21	Qwen3-4B	llm	ascii	markovian_reasoner	0.106
22	Random Agent	other	–	–	0.150
23	PPO Sparse (500k)	rl	rgb_array	–	0.114
24	Gemini 2.5 Flash Lite	llm	language	markovian_zero_shot	0.098
25	Qwen3.5-2B	llm	ascii	markovian_zero_shot	0.090
26	Qwen3.5-0.8B	llm	language	markovian_reasoner	0.100
27	Gemini 2.5 Flash Lite	llm	ascii	markovian_zero_shot	0.092
28	Qwen3-4B	llm	language	markovian_reasoner	0.071
29	Qwen3.5-2B	llm	language	markovian_zero_shot	0.033
30	Qwen3.5-4B	llm	ascii	markovian_zero_shot	0.061
31	Qwen3.5-4B	llm	language	markovian_zero_shot	0.061
32	Qwen3.5-0.8B	llm	ascii	markovian_zero_shot	0.038
33	Qwen3-4B	llm	ascii	markovian_zero_shot	0.061
34	Qwen3-4B	llm	language	markovian_zero_shot	0.057
35	Qwen3.5-0.8B	llm	language	markovian_zero_shot	0.057

Reasoning ONS

Rank	Agent	Type	Modality	Harness	Score
1	Oracle Agent	other	–	–	0.961
2	GPT-6 Astra (high, 1 seed)	llm	ascii	markovian_reasoner	0.781
3	Claude Opus 5 (high, 1 seed)	llm	ascii	markovian_reasoner	0.781
4	GPT-5.6 Sol (high, 1 seed)	llm	ascii	markovian_reasoner	0.719
5	GPT-5.6 Terra (high, 1 seed)	llm	ascii	markovian_reasoner	0.562
6	GPT-5.6 Luna (high, 1 seed)	llm	ascii	markovian_reasoner	0.500
7	Qwen3.5-4B (SFT-250k)	llm	ascii	markovian_zero_shot	0.445
8	Qwen3.5-4B (SFT-250k)	llm	ascii	markovian_reasoner	0.446
9	Qwen3.5-4B (SFT-120k)	llm	ascii	markovian_zero_shot	0.350
10	Qwen3.5-4B (SFT-120k)	llm	ascii	markovian_reasoner	0.341
11	GPT-5 mini	llm	ascii	MarkovianReasoner	0.131
12	PPO Dense (2M)	rl	rgb_array	–	0.191
13	Qwen3.5-4B	llm	ascii	markovian_reasoner	0.124
14	PPO Dense (500k)	rl	rgb_array	–	0.152
15	Gemini 2.5 Flash Lite	llm	ascii	markovian_reasoner	0.090
16	Qwen3.5-4B	llm	language	markovian_reasoner	0.024
17	Qwen3.5-2B	llm	ascii	markovian_reasoner	0.048
18	Qwen3.5-2B	llm	language	markovian_reasoner	0.007
19	Qwen3.5-0.8B	llm	ascii	markovian_reasoner	0.021
20	Qwen3.5-4B	vlm	rgb_array	markovian_zero_shot	0.013
21	Qwen3-4B	llm	ascii	markovian_reasoner	0.005
22	Random Agent	other	–	–	0.003
23	PPO Sparse (500k)	rl	rgb_array	–	0.035
24	Gemini 2.5 Flash Lite	llm	language	markovian_zero_shot	0.000
25	Qwen3.5-2B	llm	ascii	markovian_zero_shot	0.018
26	Qwen3.5-0.8B	llm	language	markovian_reasoner	0.003
27	Gemini 2.5 Flash Lite	llm	ascii	markovian_zero_shot	0.000
28	Qwen3-4B	llm	language	markovian_reasoner	0.000
29	Qwen3.5-2B	llm	language	markovian_zero_shot	0.001
30	Qwen3.5-4B	llm	ascii	markovian_zero_shot	0.000
31	Qwen3.5-4B	llm	language	markovian_zero_shot	0.000
32	Qwen3.5-0.8B	llm	ascii	markovian_zero_shot	0.001
33	Qwen3-4B	llm	ascii	markovian_zero_shot	0.000
34	Qwen3-4B	llm	language	markovian_zero_shot	0.000
35	Qwen3.5-0.8B	llm	language	markovian_zero_shot	0.000

DistributionShift-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	100%	48%
GPT-6 Astra (high, 1 seed)	llm	0%	100%	0%	0%
Claude Opus 5 (high, 1 seed)	llm	0%	100%	0%	0%
GPT-5.6 Sol (high, 1 seed)	llm	0%	100%	0%	0%
GPT-5.6 Terra (high, 1 seed)	llm	0%	100%	0%	0%
GPT-5.6 Luna (high, 1 seed)	llm	0%	100%	0%	0%
Qwen3.5-4B (SFT-250k)	llm	8%	4%	0%	0%
Qwen3.5-4B (SFT-250k)	llm	8%	0%	0%	0%
Qwen3.5-4B (SFT-120k)	llm	0%	0%	0%	0%
Qwen3.5-4B (SFT-120k)	llm	4%	0%	0%	0%
GPT-5 mini	llm	8%	4%	4%	0%
PPO Dense (2M)	rl	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
PPO Dense (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3.5-4B	vlm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Random Agent	other	0%	4%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	4%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%

Show all agents

Reach all 3 goals across shifting maze phases.

FewShotAdaptation-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	92%	80%	72%	80%
GPT-6 Astra (high, 1 seed)	llm	100%	0%	100%	0%
Claude Opus 5 (high, 1 seed)	llm	100%	0%	100%	0%
GPT-5.6 Sol (high, 1 seed)	llm	0%	0%	100%	0%
GPT-5.6 Terra (high, 1 seed)	llm	0%	100%	0%	0%
GPT-5.6 Luna (high, 1 seed)	llm	100%	0%	100%	0%
Qwen3.5-4B (SFT-250k)	llm	40%	20%	16%	20%
Qwen3.5-4B (SFT-250k)	llm	24%	20%	12%	28%
Qwen3.5-4B (SFT-120k)	llm	28%	32%	20%	24%
Qwen3.5-4B (SFT-120k)	llm	36%	28%	8%	24%
GPT-5 mini	llm	56%	44%	28%	44%
PPO Dense (2M)	rl	0%	0%	0%	0%
Qwen3.5-4B	llm	36%	44%	28%	24%
PPO Dense (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	28%	44%	32%	24%
Qwen3.5-4B	llm	56%	28%	32%	28%
Qwen3.5-2B	llm	28%	28%	0%	0%
Qwen3.5-2B	llm	8%	32%	24%	0%
Qwen3.5-0.8B	llm	24%	24%	24%	0%
Qwen3.5-4B	vlm	36%	28%	24%	28%
Qwen3-4B	llm	12%	24%	8%	12%
Random Agent	other	28%	20%	12%	12%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	12%	12%	8%	12%
Qwen3.5-2B	llm	24%	12%	16%	0%
Qwen3.5-0.8B	llm	12%	32%	8%	16%
Gemini 2.5 Flash Lite	llm	20%	16%	20%	0%
Qwen3-4B	llm	16%	24%	8%	24%
Qwen3.5-2B	llm	24%	4%	12%	4%
Qwen3.5-4B	llm	0%	4%	4%	0%
Qwen3.5-4B	llm	4%	4%	0%	0%
Qwen3.5-0.8B	llm	16%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	8%	0%	4%
Qwen3.5-0.8B	llm	0%	0%	0%	0%

Show all agents

Watch demo trials to infer the hidden rule, then navigate to the correct candidate object in the test trial.

NoisyObservation-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	80%	84%	68%
GPT-6 Astra (high, 1 seed)	llm	100%	100%	100%	100%
Claude Opus 5 (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Sol (high, 1 seed)	llm	100%	100%	0%	100%
GPT-5.6 Terra (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Luna (high, 1 seed)	llm	100%	100%	100%	0%
Qwen3.5-4B (SFT-250k)	llm	92%	72%	84%	64%
Qwen3.5-4B (SFT-250k)	llm	92%	72%	88%	60%
Qwen3.5-4B (SFT-120k)	llm	96%	80%	92%	56%
Qwen3.5-4B (SFT-120k)	llm	92%	80%	88%	56%
GPT-5 mini	llm	100%	84%	84%	68%
PPO Dense (2M)	rl	100%	56%	24%	16%
Qwen3.5-4B	llm	88%	68%	60%	44%
PPO Dense (500k)	rl	96%	28%	8%	24%
Gemini 2.5 Flash Lite	llm	88%	68%	32%	28%
Qwen3.5-4B	llm	92%	72%	68%	32%
Qwen3.5-2B	llm	60%	36%	8%	0%
Qwen3.5-2B	llm	76%	40%	20%	4%
Qwen3.5-0.8B	llm	48%	36%	8%	8%
Qwen3.5-4B	vlm	52%	20%	0%	4%
Qwen3-4B	llm	56%	32%	4%	12%
Random Agent	other	68%	28%	8%	0%
PPO Sparse (500k)	rl	28%	4%	12%	4%
Gemini 2.5 Flash Lite	llm	60%	28%	8%	8%
Qwen3.5-2B	llm	48%	28%	8%	8%
Qwen3.5-0.8B	llm	44%	32%	4%	8%
Gemini 2.5 Flash Lite	llm	20%	8%	8%	8%
Qwen3-4B	llm	44%	16%	12%	0%
Qwen3.5-2B	llm	0%	24%	4%	8%
Qwen3.5-4B	llm	12%	4%	0%	0%
Qwen3.5-4B	llm	12%	4%	4%	4%
Qwen3.5-0.8B	llm	36%	0%	0%	0%
Qwen3-4B	llm	12%	4%	0%	4%
Qwen3-4B	llm	16%	8%	0%	0%
Qwen3.5-0.8B	llm	20%	20%	0%	0%

Show all agents

Locate and reach the true GOAL amid visual noise.

DelayedGratification-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	100%	100%
GPT-6 Astra (high, 1 seed)	llm	100%	100%	100%	100%
Claude Opus 5 (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Sol (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Terra (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Luna (high, 1 seed)	llm	100%	100%	100%	100%
Qwen3.5-4B (SFT-250k)	llm	96%	84%	64%	100%
Qwen3.5-4B (SFT-250k)	llm	96%	80%	56%	100%
Qwen3.5-4B (SFT-120k)	llm	76%	60%	32%	56%
Qwen3.5-4B (SFT-120k)	llm	76%	56%	36%	48%
GPT-5 mini	llm	100%	100%	56%	100%
PPO Dense (2M)	rl	100%	88%	20%	0%
Qwen3.5-4B	llm	84%	36%	20%	72%
PPO Dense (500k)	rl	100%	28%	0%	100%
Gemini 2.5 Flash Lite	llm	88%	60%	24%	64%
Qwen3.5-4B	llm	64%	24%	0%	52%
Qwen3.5-2B	llm	60%	52%	20%	36%
Qwen3.5-2B	llm	36%	8%	0%	20%
Qwen3.5-0.8B	llm	16%	16%	0%	16%
Qwen3.5-4B	vlm	40%	24%	16%	32%
Qwen3-4B	llm	56%	28%	8%	32%
Random Agent	other	8%	0%	0%	0%
PPO Sparse (500k)	rl	48%	16%	0%	8%
Gemini 2.5 Flash Lite	llm	20%	0%	0%	24%
Qwen3.5-2B	llm	12%	0%	0%	24%
Qwen3.5-0.8B	llm	20%	4%	0%	4%
Gemini 2.5 Flash Lite	llm	20%	0%	4%	32%
Qwen3-4B	llm	16%	8%	0%	0%
Qwen3.5-2B	llm	16%	4%	0%	12%
Qwen3.5-4B	llm	12%	4%	0%	8%
Qwen3.5-4B	llm	12%	4%	0%	8%
Qwen3.5-0.8B	llm	16%	0%	4%	0%
Qwen3-4B	llm	16%	4%	0%	8%
Qwen3-4B	llm	4%	0%	0%	0%
Qwen3.5-0.8B	llm	4%	0%	0%	0%

Show all agents

Reach the distant true GOAL without collecting any decoy KEY.

FogOfWarExploration-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	92%	76%
GPT-6 Astra (high, 1 seed)	llm	100%	100%	0%	0%
Claude Opus 5 (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Sol (high, 1 seed)	llm	100%	100%	100%	0%
GPT-5.6 Terra (high, 1 seed)	llm	100%	100%	100%	0%
GPT-5.6 Luna (high, 1 seed)	llm	100%	100%	100%	0%
Qwen3.5-4B (SFT-250k)	llm	96%	76%	52%	44%
Qwen3.5-4B (SFT-250k)	llm	100%	72%	60%	56%
Qwen3.5-4B (SFT-120k)	llm	72%	72%	36%	24%
Qwen3.5-4B (SFT-120k)	llm	72%	76%	40%	24%
GPT-5 mini	llm	96%	56%	12%	12%
PPO Dense (2M)	rl	92%	36%	12%	28%
Qwen3.5-4B	llm	84%	60%	12%	12%
PPO Dense (500k)	rl	56%	24%	12%	8%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	88%	96%	76%	64%
Qwen3.5-2B	llm	72%	52%	0%	24%
Qwen3.5-2B	llm	48%	44%	28%	16%
Qwen3.5-0.8B	llm	56%	40%	20%	20%
Qwen3.5-4B	vlm	48%	24%	4%	12%
Qwen3-4B	llm	64%	20%	16%	16%
Random Agent	other	52%	56%	12%	16%
PPO Sparse (500k)	rl	40%	20%	0%	0%
Gemini 2.5 Flash Lite	llm	52%	36%	16%	12%
Qwen3.5-2B	llm	36%	24%	4%	4%
Qwen3.5-0.8B	llm	44%	36%	12%	0%
Gemini 2.5 Flash Lite	llm	36%	24%	8%	0%
Qwen3-4B	llm	28%	12%	0%	4%
Qwen3.5-2B	llm	0%	28%	8%	16%
Qwen3.5-4B	llm	16%	8%	12%	0%
Qwen3.5-4B	llm	0%	0%	4%	0%
Qwen3.5-0.8B	llm	20%	4%	0%	4%
Qwen3-4B	llm	4%	0%	4%	0%
Qwen3-4B	llm	12%	8%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%

Show all agents

Find and reach the GOAL despite incomplete map information.

SequenceMemory-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	100%	100%
GPT-6 Astra (high, 1 seed)	llm	0%	0%	0%	0%
Claude Opus 5 (high, 1 seed)	llm	0%	0%	0%	0%
GPT-5.6 Sol (high, 1 seed)	llm	0%	0%	0%	0%
GPT-5.6 Terra (high, 1 seed)	llm	0%	0%	0%	0%
GPT-5.6 Luna (high, 1 seed)	llm	0%	0%	0%	0%
Qwen3.5-4B (SFT-250k)	llm	24%	12%	0%	4%
Qwen3.5-4B (SFT-250k)	llm	32%	16%	0%	0%
Qwen3.5-4B (SFT-120k)	llm	16%	0%	4%	0%
Qwen3.5-4B (SFT-120k)	llm	28%	12%	0%	0%
GPT-5 mini	llm	0%	0%	0%	0%
PPO Dense (2M)	rl	76%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
PPO Dense (500k)	rl	32%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	8%	0%	0%	0%
Qwen3.5-2B	llm	4%	4%	0%	0%
Qwen3.5-0.8B	llm	4%	0%	0%	0%
Qwen3.5-4B	vlm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Random Agent	other	12%	0%	0%	0%
PPO Sparse (500k)	rl	32%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	12%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%

Show all agents

Memorize shown GEM positions, then visit them in exact order during reproduce phase.

TreasureHunt-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	100%	100%
GPT-6 Astra (high, 1 seed)	llm	0%	0%	0%	0%
Claude Opus 5 (high, 1 seed)	llm	0%	0%	0%	0%
GPT-5.6 Sol (high, 1 seed)	llm	0%	0%	0%	0%
GPT-5.6 Terra (high, 1 seed)	llm	0%	0%	0%	0%
GPT-5.6 Luna (high, 1 seed)	llm	0%	0%	0%	0%
Qwen3.5-4B (SFT-250k)	llm	8%	0%	0%	0%
Qwen3.5-4B (SFT-250k)	llm	8%	0%	0%	0%
Qwen3.5-4B (SFT-120k)	llm	8%	4%	0%	0%
Qwen3.5-4B (SFT-120k)	llm	0%	0%	0%	0%
GPT-5 mini	llm	20%	4%	0%	0%
PPO Dense (2M)	rl	0%	0%	0%	0%
Qwen3.5-4B	llm	16%	0%	0%	0%
PPO Dense (500k)	rl	4%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	16%	8%	0%	0%
Qwen3.5-4B	llm	8%	0%	0%	0%
Qwen3.5-2B	llm	8%	8%	0%	0%
Qwen3.5-2B	llm	8%	0%	0%	0%
Qwen3.5-0.8B	llm	24%	0%	0%	0%
Qwen3.5-4B	vlm	16%	0%	0%	0%
Qwen3-4B	llm	4%	0%	0%	0%
Random Agent	other	28%	4%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	8%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	8%	4%	0%	0%
Gemini 2.5 Flash Lite	llm	8%	0%	0%	0%
Qwen3-4B	llm	12%	0%	0%	0%
Qwen3.5-2B	llm	0%	4%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	4%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	4%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%

Show all agents

Read scroll clues, triangulate hidden treasure positions, and step on each treasure cell to collect all treasures.

ChaseEvade-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	84%	84%	72%	16%
GPT-6 Astra (high, 1 seed)	llm	0%	100%	0%	0%
Claude Opus 5 (high, 1 seed)	llm	0%	100%	0%	0%
GPT-5.6 Sol (high, 1 seed)	llm	0%	100%	0%	0%
GPT-5.6 Terra (high, 1 seed)	llm	0%	0%	0%	0%
GPT-5.6 Luna (high, 1 seed)	llm	0%	100%	0%	0%
Qwen3.5-4B (SFT-250k)	llm	76%	48%	16%	16%
Qwen3.5-4B (SFT-250k)	llm	64%	32%	8%	12%
Qwen3.5-4B (SFT-120k)	llm	32%	48%	12%	0%
Qwen3.5-4B (SFT-120k)	llm	36%	40%	12%	0%
GPT-5 mini	llm	4%	0%	0%	0%
PPO Dense (2M)	rl	100%	40%	52%	8%
Qwen3.5-4B	llm	0%	0%	0%	0%
PPO Dense (500k)	rl	96%	32%	60%	12%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3.5-4B	vlm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Random Agent	other	0%	0%	0%	0%
PPO Sparse (500k)	rl	40%	60%	20%	8%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%

Show all agents

Survive the required steps without enemy collision.

CooperativeTransport-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	84%	64%	52%
GPT-6 Astra (high, 1 seed)	llm	100%	100%	100%	100%
Claude Opus 5 (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Sol (high, 1 seed)	llm	100%	100%	100%	0%
GPT-5.6 Terra (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Luna (high, 1 seed)	llm	100%	100%	100%	100%
Qwen3.5-4B (SFT-250k)	llm	32%	0%	0%	0%
Qwen3.5-4B (SFT-250k)	llm	40%	0%	0%	0%
Qwen3.5-4B (SFT-120k)	llm	20%	0%	0%	0%
Qwen3.5-4B (SFT-120k)	llm	16%	0%	0%	0%
GPT-5 mini	llm	8%	0%	0%	0%
PPO Dense (2M)	rl	0%	0%	0%	0%
Qwen3.5-4B	llm	48%	0%	0%	0%
PPO Dense (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	12%	0%	0%	0%
Qwen3.5-2B	llm	4%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3.5-4B	vlm	0%	0%	0%	0%
Qwen3-4B	llm	4%	0%	0%	0%
Random Agent	other	0%	0%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	4%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	4%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%

Show all agents

Push all heavy boxes into holes with NPC cooperation.

EmergentStrategy-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	88%	92%	52%	48%
GPT-6 Astra (high, 1 seed)	llm	100%	0%	0%	0%
Claude Opus 5 (high, 1 seed)	llm	100%	0%	0%	0%
GPT-5.6 Sol (high, 1 seed)	llm	100%	0%	0%	100%
GPT-5.6 Terra (high, 1 seed)	llm	100%	100%	0%	100%
GPT-5.6 Luna (high, 1 seed)	llm	100%	100%	0%	100%
Qwen3.5-4B (SFT-250k)	llm	60%	32%	16%	0%
Qwen3.5-4B (SFT-250k)	llm	56%	24%	4%	0%
Qwen3.5-4B (SFT-120k)	llm	52%	36%	4%	0%
Qwen3.5-4B (SFT-120k)	llm	36%	40%	0%	8%
GPT-5 mini	llm	92%	8%	16%	0%
PPO Dense (2M)	rl	96%	0%	0%	0%
Qwen3.5-4B	llm	52%	4%	0%	0%
PPO Dense (500k)	rl	4%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	76%	8%	4%	4%
Qwen3.5-4B	llm	20%	8%	0%	0%
Qwen3.5-2B	llm	32%	0%	0%	0%
Qwen3.5-2B	llm	24%	4%	0%	0%
Qwen3.5-0.8B	llm	36%	4%	0%	0%
Qwen3.5-4B	vlm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Random Agent	other	20%	4%	4%	0%
PPO Sparse (500k)	rl	4%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	4%	0%	0%	0%
Qwen3.5-2B	llm	20%	8%	0%	0%
Qwen3.5-0.8B	llm	4%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	12%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%

Show all agents

Exploit NPC behaviors to lure/scare them onto locking pressure plates, permanently opening barriers to reach the GOAL...

Herding-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	80%	76%	44%	36%
GPT-6 Astra (high, 1 seed)	llm	100%	0%	0%	0%
Claude Opus 5 (high, 1 seed)	llm	100%	0%	0%	0%
GPT-5.6 Sol (high, 1 seed)	llm	0%	0%	0%	0%
GPT-5.6 Terra (high, 1 seed)	llm	0%	0%	0%	0%
GPT-5.6 Luna (high, 1 seed)	llm	0%	0%	0%	0%
Qwen3.5-4B (SFT-250k)	llm	60%	36%	4%	4%
Qwen3.5-4B (SFT-250k)	llm	76%	32%	4%	12%
Qwen3.5-4B (SFT-120k)	llm	20%	12%	0%	4%
Qwen3.5-4B (SFT-120k)	llm	32%	0%	0%	0%
GPT-5 mini	llm	8%	4%	0%	0%
PPO Dense (2M)	rl	100%	52%	32%	0%
Qwen3.5-4B	llm	12%	4%	0%	0%
PPO Dense (500k)	rl	28%	40%	44%	76%
Gemini 2.5 Flash Lite	llm	4%	4%	0%	0%
Qwen3.5-4B	llm	16%	4%	0%	0%
Qwen3.5-2B	llm	0%	4%	0%	0%
Qwen3.5-2B	llm	12%	8%	0%	0%
Qwen3.5-0.8B	llm	0%	4%	0%	0%
Qwen3.5-4B	vlm	12%	4%	0%	0%
Qwen3-4B	llm	4%	8%	0%	0%
Random Agent	other	8%	4%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	4%	4%	8%	0%
Qwen3.5-2B	llm	8%	8%	0%	0%
Qwen3.5-0.8B	llm	8%	4%	0%	0%
Gemini 2.5 Flash Lite	llm	4%	8%	0%	0%
Qwen3-4B	llm	4%	4%	0%	0%
Qwen3.5-2B	llm	8%	4%	0%	0%
Qwen3.5-4B	llm	4%	8%	0%	0%
Qwen3.5-4B	llm	4%	4%	0%	0%
Qwen3.5-0.8B	llm	4%	4%	0%	0%
Qwen3-4B	llm	4%	8%	0%	0%
Qwen3-4B	llm	4%	4%	0%	0%
Qwen3.5-0.8B	llm	4%	4%	0%	0%

Show all agents

Move all SHEEP into the pen zone (TARGET cells).

TagHunt-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	96%	84%	68%	64%
GPT-6 Astra (high, 1 seed)	llm	100%	100%	0%	0%
Claude Opus 5 (high, 1 seed)	llm	100%	0%	0%	0%
GPT-5.6 Sol (high, 1 seed)	llm	100%	100%	0%	0%
GPT-5.6 Terra (high, 1 seed)	llm	100%	100%	0%	0%
GPT-5.6 Luna (high, 1 seed)	llm	100%	100%	0%	0%
Qwen3.5-4B (SFT-250k)	llm	92%	96%	80%	28%
Qwen3.5-4B (SFT-250k)	llm	100%	100%	84%	28%
Qwen3.5-4B (SFT-120k)	llm	100%	88%	48%	16%
Qwen3.5-4B (SFT-120k)	llm	100%	72%	64%	20%
GPT-5 mini	llm	72%	60%	20%	8%
PPO Dense (2M)	rl	100%	96%	96%	92%
Qwen3.5-4B	llm	88%	48%	8%	4%
PPO Dense (500k)	rl	100%	80%	84%	48%
Gemini 2.5 Flash Lite	llm	92%	4%	0%	0%
Qwen3.5-4B	llm	76%	8%	0%	0%
Qwen3.5-2B	llm	24%	0%	0%	0%
Qwen3.5-2B	llm	52%	0%	0%	0%
Qwen3.5-0.8B	llm	24%	4%	0%	0%
Qwen3.5-4B	vlm	88%	8%	0%	0%
Qwen3-4B	llm	48%	12%	0%	0%
Random Agent	other	20%	0%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	20%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	12%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	16%	0%	0%	0%
Qwen3-4B	llm	28%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	4%	0%	0%	0%
Qwen3.5-4B	llm	4%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	4%	0%	0%	0%
Qwen3-4B	llm	4%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%

Show all agents

Tag all NPCs by stepping onto them.

CuriosityMaze-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	100%	100%
GPT-6 Astra (high, 1 seed)	llm	0%	0%	0%	0%
Claude Opus 5 (high, 1 seed)	llm	0%	0%	0%	0%
GPT-5.6 Sol (high, 1 seed)	llm	0%	0%	0%	0%
GPT-5.6 Terra (high, 1 seed)	llm	0%	0%	0%	0%
GPT-5.6 Luna (high, 1 seed)	llm	0%	0%	0%	0%
Qwen3.5-4B (SFT-250k)	llm	16%	0%	0%	0%
Qwen3.5-4B (SFT-250k)	llm	12%	0%	0%	0%
Qwen3.5-4B (SFT-120k)	llm	4%	0%	0%	0%
Qwen3.5-4B (SFT-120k)	llm	0%	0%	0%	0%
GPT-5 mini	llm	0%	0%	0%	0%
PPO Dense (2M)	rl	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
PPO Dense (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3.5-4B	vlm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Random Agent	other	0%	0%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	0%	–
Gemini 3.1 Flash Lite	llm	–	–	0%	–

Show all agents

Visit at least the required percentage of all reachable cells before the step budget runs out.

DynamicObstacles-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	96%	88%	92%	72%
GPT-6 Astra (high, 1 seed)	llm	100%	100%	100%	100%
Claude Opus 5 (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Sol (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Terra (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Luna (high, 1 seed)	llm	100%	100%	100%	0%
Qwen3.5-4B (SFT-250k)	llm	92%	88%	64%	48%
Qwen3.5-4B (SFT-250k)	llm	92%	92%	60%	40%
Qwen3.5-4B (SFT-120k)	llm	88%	72%	64%	36%
Qwen3.5-4B (SFT-120k)	llm	80%	84%	52%	36%
GPT-5 mini	llm	80%	92%	60%	28%
PPO Dense (2M)	rl	96%	72%	12%	4%
Qwen3.5-4B	llm	64%	60%	36%	0%
PPO Dense (500k)	rl	76%	20%	4%	0%
Gemini 2.5 Flash Lite	llm	64%	68%	32%	8%
Qwen3.5-4B	llm	44%	16%	24%	0%
Qwen3.5-2B	llm	52%	36%	12%	0%
Qwen3.5-2B	llm	36%	32%	8%	0%
Qwen3.5-0.8B	llm	36%	28%	8%	0%
Qwen3.5-4B	vlm	0%	0%	0%	0%
Qwen3-4B	llm	24%	8%	0%	0%
Random Agent	other	8%	0%	0%	0%
PPO Sparse (500k)	rl	60%	12%	12%	0%
Gemini 2.5 Flash Lite	llm	4%	0%	0%	0%
Qwen3.5-2B	llm	16%	4%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	4%	12%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	24%	–
Gemini 3.1 Flash Lite	llm	–	–	44%	–

Show all agents

Reach GOAL without colliding with any NPC.

GoToGoal-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	100%	92%
GPT-6 Astra (high, 1 seed)	llm	100%	100%	100%	100%
Claude Opus 5 (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Sol (high, 1 seed)	llm	100%	100%	100%	0%
GPT-5.6 Terra (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Luna (high, 1 seed)	llm	100%	100%	100%	100%
Qwen3.5-4B (SFT-250k)	llm	100%	92%	76%	68%
Qwen3.5-4B (SFT-250k)	llm	100%	88%	56%	44%
Qwen3.5-4B (SFT-120k)	llm	100%	76%	44%	40%
Qwen3.5-4B (SFT-120k)	llm	100%	76%	28%	28%
GPT-5 mini	llm	100%	88%	56%	44%
PPO Dense (2M)	rl	100%	24%	16%	8%
Qwen3.5-4B	llm	88%	52%	28%	8%
PPO Dense (500k)	rl	100%	40%	12%	0%
Gemini 2.5 Flash Lite	llm	100%	64%	8%	4%
Qwen3.5-4B	llm	68%	20%	0%	4%
Qwen3.5-2B	llm	52%	32%	0%	0%
Qwen3.5-2B	llm	52%	16%	0%	0%
Qwen3.5-0.8B	llm	32%	12%	0%	0%
Qwen3.5-4B	vlm	40%	8%	8%	4%
Qwen3-4B	llm	40%	12%	0%	0%
Random Agent	other	44%	0%	0%	0%
PPO Sparse (500k)	rl	100%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	8%	4%	0%	0%
Qwen3.5-2B	llm	40%	24%	0%	0%
Qwen3.5-0.8B	llm	36%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	28%	16%	8%	4%
Qwen3-4B	llm	40%	4%	0%	4%
Qwen3.5-2B	llm	0%	8%	0%	0%
Qwen3.5-4B	llm	20%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	4%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	20%	–
Gemini 3.1 Flash Lite	llm	–	–	20%	–

Show all agents

Reach the GOAL position.

InstructionFollowing-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	100%	92%
GPT-6 Astra (high, 1 seed)	llm	100%	100%	100%	100%
Claude Opus 5 (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Sol (high, 1 seed)	llm	100%	100%	100%	0%
GPT-5.6 Terra (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Luna (high, 1 seed)	llm	100%	100%	0%	100%
Qwen3.5-4B (SFT-250k)	llm	72%	60%	72%	20%
Qwen3.5-4B (SFT-250k)	llm	76%	76%	72%	24%
Qwen3.5-4B (SFT-120k)	llm	60%	56%	24%	16%
Qwen3.5-4B (SFT-120k)	llm	68%	52%	20%	12%
GPT-5 mini	llm	96%	96%	80%	40%
PPO Dense (2M)	rl	20%	0%	0%	0%
Qwen3.5-4B	llm	64%	68%	4%	0%
PPO Dense (500k)	rl	16%	4%	0%	0%
Gemini 2.5 Flash Lite	llm	52%	56%	0%	0%
Qwen3.5-4B	llm	48%	40%	0%	0%
Qwen3.5-2B	llm	32%	28%	0%	0%
Qwen3.5-2B	llm	16%	16%	0%	0%
Qwen3.5-0.8B	llm	12%	24%	0%	0%
Qwen3.5-4B	vlm	36%	8%	0%	0%
Qwen3-4B	llm	28%	36%	0%	0%
Random Agent	other	12%	20%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	16%	24%	0%	0%
Qwen3.5-2B	llm	4%	8%	0%	0%
Qwen3.5-0.8B	llm	28%	24%	0%	0%
Gemini 2.5 Flash Lite	llm	28%	12%	0%	0%
Qwen3-4B	llm	32%	32%	0%	0%
Qwen3.5-2B	llm	0%	8%	0%	0%
Qwen3.5-4B	llm	12%	4%	0%	0%
Qwen3.5-4B	llm	12%	8%	0%	0%
Qwen3.5-0.8B	llm	4%	8%	0%	0%
Qwen3-4B	llm	12%	8%	0%	0%
Qwen3-4B	llm	12%	4%	0%	0%
Qwen3.5-0.8B	llm	4%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	8%	–
Gemini 3.1 Flash Lite	llm	–	–	0%	–

Show all agents

Reach the unique target object without touching any distractor.

MazeNavigation-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	100%	100%
GPT-6 Astra (high, 1 seed)	llm	100%	100%	100%	100%
Claude Opus 5 (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Sol (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Terra (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Luna (high, 1 seed)	llm	100%	100%	100%	0%
Qwen3.5-4B (SFT-250k)	llm	100%	28%	24%	4%
Qwen3.5-4B (SFT-250k)	llm	100%	20%	24%	8%
Qwen3.5-4B (SFT-120k)	llm	100%	28%	4%	4%
Qwen3.5-4B (SFT-120k)	llm	100%	20%	4%	4%
GPT-5 mini	llm	80%	16%	4%	0%
PPO Dense (2M)	rl	100%	0%	0%	0%
Qwen3.5-4B	llm	84%	12%	0%	0%
PPO Dense (500k)	rl	100%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	68%	24%	4%	0%
Qwen3.5-4B	llm	72%	4%	0%	0%
Qwen3.5-2B	llm	96%	8%	0%	0%
Qwen3.5-2B	llm	76%	8%	0%	0%
Qwen3.5-0.8B	llm	8%	4%	0%	0%
Qwen3.5-4B	vlm	8%	4%	0%	0%
Qwen3-4B	llm	72%	0%	0%	0%
Random Agent	other	4%	4%	0%	0%
PPO Sparse (500k)	rl	100%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	24%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	4%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	0%	–
Gemini 3.1 Flash Lite	llm	–	–	4%	–

Show all agents

Navigate the maze to reach the GOAL exit.

RecursiveRooms-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	100%	100%
GPT-6 Astra (high, 1 seed)	llm	100%	100%	100%	100%
Claude Opus 5 (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Sol (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Terra (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Luna (high, 1 seed)	llm	100%	100%	100%	100%
Qwen3.5-4B (SFT-250k)	llm	80%	72%	36%	28%
Qwen3.5-4B (SFT-250k)	llm	84%	68%	40%	28%
Qwen3.5-4B (SFT-120k)	llm	80%	24%	8%	16%
Qwen3.5-4B (SFT-120k)	llm	72%	36%	20%	24%
GPT-5 mini	llm	48%	12%	12%	8%
PPO Dense (2M)	rl	12%	0%	0%	0%
Qwen3.5-4B	llm	80%	32%	0%	0%
PPO Dense (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	40%	4%	8%	8%
Qwen3.5-4B	llm	52%	12%	4%	0%
Qwen3.5-2B	llm	36%	16%	0%	0%
Qwen3.5-2B	llm	28%	12%	0%	12%
Qwen3.5-0.8B	llm	12%	0%	8%	8%
Qwen3.5-4B	vlm	20%	8%	8%	0%
Qwen3-4B	llm	8%	0%	0%	0%
Random Agent	other	12%	0%	0%	4%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	20%	8%	4%	16%
Qwen3.5-0.8B	llm	12%	4%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	4%	0%	0%
Qwen3-4B	llm	8%	4%	4%	0%
Qwen3.5-2B	llm	8%	4%	4%	16%
Qwen3.5-4B	llm	4%	0%	0%	0%
Qwen3.5-4B	llm	4%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	4%	0%	0%	0%
Qwen3-4B	llm	4%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	0%	–
Gemini 3.1 Flash Lite	llm	–	–	12%	–

Show all agents

Navigate through nested rooms to reach GOAL in the deepest room.

ShortestPath-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	96%	96%
GPT-6 Astra (high, 1 seed)	llm	100%	100%	100%	100%
Claude Opus 5 (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Sol (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Terra (high, 1 seed)	llm	0%	100%	100%	100%
GPT-5.6 Luna (high, 1 seed)	llm	100%	100%	100%	100%
Qwen3.5-4B (SFT-250k)	llm	72%	72%	64%	64%
Qwen3.5-4B (SFT-250k)	llm	72%	76%	52%	52%
Qwen3.5-4B (SFT-120k)	llm	68%	64%	24%	36%
Qwen3.5-4B (SFT-120k)	llm	56%	68%	40%	40%
GPT-5 mini	llm	100%	88%	72%	28%
PPO Dense (2M)	rl	100%	72%	0%	0%
Qwen3.5-4B	llm	12%	8%	0%	4%
PPO Dense (500k)	rl	72%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	56%	20%	0%	0%
Qwen3.5-4B	llm	12%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3.5-4B	vlm	8%	0%	0%	0%
Qwen3-4B	llm	4%	0%	0%	0%
Random Agent	other	0%	0%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	4%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	4%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	0%	–
Gemini 3.1 Flash Lite	llm	–	–	16%	–

Show all agents

Visit all real GOAL objects within the step budget (optimal path × budget multiplier).

TimingChallenge-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	96%	100%
GPT-6 Astra (high, 1 seed)	llm	0%	0%	0%	100%
Claude Opus 5 (high, 1 seed)	llm	0%	0%	0%	100%
GPT-5.6 Sol (high, 1 seed)	llm	0%	0%	0%	100%
GPT-5.6 Terra (high, 1 seed)	llm	0%	0%	0%	100%
GPT-5.6 Luna (high, 1 seed)	llm	0%	0%	0%	100%
Qwen3.5-4B (SFT-250k)	llm	96%	68%	60%	80%
Qwen3.5-4B (SFT-250k)	llm	80%	68%	72%	68%
Qwen3.5-4B (SFT-120k)	llm	52%	48%	64%	68%
Qwen3.5-4B (SFT-120k)	llm	44%	60%	72%	64%
GPT-5 mini	llm	36%	24%	36%	36%
PPO Dense (2M)	rl	52%	32%	44%	36%
Qwen3.5-4B	llm	4%	4%	0%	0%
PPO Dense (500k)	rl	56%	40%	48%	28%
Gemini 2.5 Flash Lite	llm	8%	16%	28%	20%
Qwen3.5-4B	llm	0%	8%	8%	0%
Qwen3.5-2B	llm	8%	4%	12%	12%
Qwen3.5-2B	llm	28%	36%	16%	16%
Qwen3.5-0.8B	llm	0%	12%	12%	4%
Qwen3.5-4B	vlm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Random Agent	other	0%	8%	0%	12%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	44%	12%	0%	4%
Qwen3.5-2B	llm	0%	4%	4%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	16%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	4%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	4%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	32%	–
Gemini 3.1 Flash Lite	llm	–	–	32%	–

Show all agents

Cross the patrol zone without collision, then reach GOAL.

BacktrackPuzzle-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	100%	100%
GPT-6 Astra (high, 1 seed)	llm	100%	100%	100%	100%
Claude Opus 5 (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Sol (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Terra (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Luna (high, 1 seed)	llm	100%	100%	100%	100%
Qwen3.5-4B (SFT-250k)	llm	100%	88%	80%	40%
Qwen3.5-4B (SFT-250k)	llm	100%	96%	76%	56%
Qwen3.5-4B (SFT-120k)	llm	100%	84%	64%	24%
Qwen3.5-4B (SFT-120k)	llm	100%	76%	68%	16%
GPT-5 mini	llm	100%	60%	20%	4%
PPO Dense (2M)	rl	96%	52%	0%	0%
Qwen3.5-4B	llm	88%	32%	20%	24%
PPO Dense (500k)	rl	100%	8%	0%	0%
Gemini 2.5 Flash Lite	llm	16%	8%	8%	8%
Qwen3.5-4B	llm	76%	16%	8%	0%
Qwen3.5-2B	llm	4%	0%	0%	0%
Qwen3.5-2B	llm	4%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3.5-4B	vlm	16%	4%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Random Agent	other	0%	4%	0%	0%
PPO Sparse (500k)	rl	92%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	4%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	16%	4%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	0%	–
Gemini 3.1 Flash Lite	llm	–	–	60%	–

Show all agents

Activate the correct SWITCH to open the gate, then backtrack to reach GOAL.

KeyDoorPuzzle-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	88%	76%
GPT-6 Astra (high, 1 seed)	llm	100%	100%	100%	100%
Claude Opus 5 (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Sol (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Terra (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Luna (high, 1 seed)	llm	100%	100%	100%	100%
Qwen3.5-4B (SFT-250k)	llm	96%	20%	0%	0%
Qwen3.5-4B (SFT-250k)	llm	96%	8%	0%	0%
Qwen3.5-4B (SFT-120k)	llm	76%	12%	0%	0%
Qwen3.5-4B (SFT-120k)	llm	64%	0%	0%	0%
GPT-5 mini	llm	100%	44%	4%	0%
PPO Dense (2M)	rl	84%	0%	0%	0%
Qwen3.5-4B	llm	60%	44%	0%	0%
PPO Dense (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	32%	4%	0%	0%
Qwen3.5-4B	llm	36%	28%	0%	0%
Qwen3.5-2B	llm	20%	0%	0%	0%
Qwen3.5-2B	llm	20%	0%	0%	0%
Qwen3.5-0.8B	llm	4%	0%	0%	0%
Qwen3.5-4B	vlm	16%	0%	0%	0%
Qwen3-4B	llm	4%	0%	0%	0%
Random Agent	other	0%	0%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	0%	–
Gemini 3.1 Flash Lite	llm	–	–	0%	–

Show all agents

Reach GOAL after unlocking ALL doors with matching keys.

PackingPuzzle-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	96%	96%	96%	96%
GPT-6 Astra (high, 1 seed)	llm	0%	0%	0%	0%
Claude Opus 5 (high, 1 seed)	llm	0%	0%	0%	0%
GPT-5.6 Sol (high, 1 seed)	llm	0%	0%	0%	0%
GPT-5.6 Terra (high, 1 seed)	llm	0%	0%	0%	0%
GPT-5.6 Luna (high, 1 seed)	llm	0%	0%	0%	0%
Qwen3.5-4B (SFT-250k)	llm	36%	20%	12%	0%
Qwen3.5-4B (SFT-250k)	llm	32%	20%	24%	4%
Qwen3.5-4B (SFT-120k)	llm	28%	24%	0%	0%
Qwen3.5-4B (SFT-120k)	llm	12%	8%	0%	0%
GPT-5 mini	llm	0%	0%	0%	0%
PPO Dense (2M)	rl	64%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
PPO Dense (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3.5-4B	vlm	4%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Random Agent	other	0%	0%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	4%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	0%	–
Gemini 3.1 Flash Lite	llm	–	–	0%	–

Show all agents

Push each piece onto its matching-type target slot.

PreciseNavigation-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	56%	32%
GPT-6 Astra (high, 1 seed)	llm	100%	100%	100%	100%
Claude Opus 5 (high, 1 seed)	llm	100%	100%	0%	100%
GPT-5.6 Sol (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Terra (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Luna (high, 1 seed)	llm	100%	100%	100%	100%
Qwen3.5-4B (SFT-250k)	llm	84%	44%	24%	8%
Qwen3.5-4B (SFT-250k)	llm	92%	48%	36%	8%
Qwen3.5-4B (SFT-120k)	llm	92%	32%	20%	0%
Qwen3.5-4B (SFT-120k)	llm	92%	28%	12%	4%
GPT-5 mini	llm	88%	12%	12%	12%
PPO Dense (2M)	rl	64%	0%	8%	4%
Qwen3.5-4B	llm	92%	88%	36%	40%
PPO Dense (500k)	rl	88%	4%	4%	0%
Gemini 2.5 Flash Lite	llm	88%	28%	36%	20%
Qwen3.5-4B	llm	80%	48%	16%	20%
Qwen3.5-2B	llm	76%	28%	20%	16%
Qwen3.5-2B	llm	76%	40%	40%	24%
Qwen3.5-0.8B	llm	40%	12%	20%	4%
Qwen3.5-4B	vlm	92%	36%	32%	8%
Qwen3-4B	llm	48%	4%	4%	8%
Random Agent	other	84%	16%	44%	24%
PPO Sparse (500k)	rl	20%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	36%	4%	8%	0%
Qwen3.5-2B	llm	48%	12%	32%	16%
Qwen3.5-0.8B	llm	52%	16%	24%	8%
Gemini 2.5 Flash Lite	llm	44%	4%	16%	4%
Qwen3-4B	llm	16%	12%	4%	0%
Qwen3.5-2B	llm	0%	12%	4%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	16%	16%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	4%
Claude Haiku 4.5	llm	–	–	32%	–
Gemini 3.1 Flash Lite	llm	–	–	48%	–

Show all agents

Slide across ice to reach the GOAL by planning trajectories through stopping points.

RecipeAssembly-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	100%	100%
GPT-6 Astra (high, 1 seed)	llm	100%	100%	100%	100%
Claude Opus 5 (high, 1 seed)	llm	0%	100%	0%	0%
GPT-5.6 Sol (high, 1 seed)	llm	0%	100%	100%	0%
GPT-5.6 Terra (high, 1 seed)	llm	0%	100%	0%	0%
GPT-5.6 Luna (high, 1 seed)	llm	0%	0%	0%	0%
Qwen3.5-4B (SFT-250k)	llm	88%	80%	52%	72%
Qwen3.5-4B (SFT-250k)	llm	88%	88%	52%	60%
Qwen3.5-4B (SFT-120k)	llm	56%	48%	16%	20%
Qwen3.5-4B (SFT-120k)	llm	76%	60%	12%	52%
GPT-5 mini	llm	24%	0%	0%	0%
PPO Dense (2M)	rl	88%	0%	0%	0%
Qwen3.5-4B	llm	4%	0%	0%	0%
PPO Dense (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	16%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3.5-4B	vlm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Random Agent	other	0%	0%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	0%	–
Gemini 3.1 Flash Lite	llm	–	–	0%	–

Show all agents

Collect and deliver all ingredients in recipe order to the crafting station.

ResourceManagement-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	100%	96%
GPT-6 Astra (high, 1 seed)	llm	100%	100%	100%	100%
Claude Opus 5 (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Sol (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Terra (high, 1 seed)	llm	100%	100%	100%	0%
GPT-5.6 Luna (high, 1 seed)	llm	100%	100%	100%	0%
Qwen3.5-4B (SFT-250k)	llm	100%	100%	100%	60%
Qwen3.5-4B (SFT-250k)	llm	100%	100%	100%	68%
Qwen3.5-4B (SFT-120k)	llm	100%	100%	92%	20%
Qwen3.5-4B (SFT-120k)	llm	100%	100%	76%	8%
GPT-5 mini	llm	100%	100%	12%	0%
PPO Dense (2M)	rl	100%	100%	100%	100%
Qwen3.5-4B	llm	100%	100%	0%	0%
PPO Dense (500k)	rl	100%	100%	100%	100%
Gemini 2.5 Flash Lite	llm	100%	100%	0%	0%
Qwen3.5-4B	llm	100%	100%	0%	0%
Qwen3.5-2B	llm	100%	100%	0%	0%
Qwen3.5-2B	llm	100%	100%	0%	0%
Qwen3.5-0.8B	llm	100%	100%	0%	0%
Qwen3.5-4B	vlm	100%	100%	0%	0%
Qwen3-4B	llm	100%	100%	0%	0%
Random Agent	other	100%	100%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	100%	100%	0%	0%
Qwen3.5-2B	llm	100%	100%	0%	0%
Qwen3.5-0.8B	llm	100%	100%	0%	0%
Gemini 2.5 Flash Lite	llm	100%	100%	0%	0%
Qwen3-4B	llm	100%	100%	0%	0%
Qwen3.5-2B	llm	0%	100%	0%	0%
Qwen3.5-4B	llm	100%	100%	0%	0%
Qwen3.5-4B	llm	100%	100%	0%	0%
Qwen3.5-0.8B	llm	100%	0%	0%	0%
Qwen3-4B	llm	100%	100%	0%	0%
Qwen3-4B	llm	100%	100%	0%	0%
Qwen3.5-0.8B	llm	100%	100%	0%	0%
Claude Haiku 4.5	llm	–	–	76%	–
Gemini 3.1 Flash Lite	llm	–	–	48%	–

Show all agents

Keep ALL stations above 0 energy for the entire episode (survive max_steps).

SokobanPush-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	84%	88%	36%
GPT-6 Astra (high, 1 seed)	llm	100%	0%	100%	0%
Claude Opus 5 (high, 1 seed)	llm	100%	0%	100%	0%
GPT-5.6 Sol (high, 1 seed)	llm	100%	0%	100%	0%
GPT-5.6 Terra (high, 1 seed)	llm	100%	100%	0%	0%
GPT-5.6 Luna (high, 1 seed)	llm	0%	0%	100%	100%
Qwen3.5-4B (SFT-250k)	llm	48%	8%	4%	0%
Qwen3.5-4B (SFT-250k)	llm	48%	12%	0%	0%
Qwen3.5-4B (SFT-120k)	llm	40%	0%	0%	0%
Qwen3.5-4B (SFT-120k)	llm	24%	4%	0%	0%
GPT-5 mini	llm	12%	0%	0%	0%
PPO Dense (2M)	rl	92%	4%	0%	0%
Qwen3.5-4B	llm	20%	4%	0%	0%
PPO Dense (500k)	rl	36%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	12%	0%	0%	0%
Qwen3.5-4B	llm	8%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	8%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3.5-4B	vlm	12%	0%	0%	0%
Qwen3-4B	llm	4%	0%	0%	0%
Random Agent	other	0%	0%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	16%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	8%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	4%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	0%	–
Gemini 3.1 Flash Lite	llm	–	–	0%	–

Show all agents

Push all BOX objects onto matching TARGET positions.

TileSorting-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	100%	100%
GPT-6 Astra (high, 1 seed)	llm	100%	100%	100%	100%
Claude Opus 5 (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Sol (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Terra (high, 1 seed)	llm	100%	100%	100%	0%
GPT-5.6 Luna (high, 1 seed)	llm	100%	100%	100%	100%
Qwen3.5-4B (SFT-250k)	llm	100%	84%	44%	12%
Qwen3.5-4B (SFT-250k)	llm	100%	88%	56%	8%
Qwen3.5-4B (SFT-120k)	llm	84%	64%	56%	0%
Qwen3.5-4B (SFT-120k)	llm	100%	80%	48%	0%
GPT-5 mini	llm	88%	0%	12%	0%
PPO Dense (2M)	rl	100%	96%	0%	0%
Qwen3.5-4B	llm	100%	72%	0%	0%
PPO Dense (500k)	rl	92%	84%	56%	0%
Gemini 2.5 Flash Lite	llm	76%	32%	16%	0%
Qwen3.5-4B	llm	96%	4%	0%	0%
Qwen3.5-2B	llm	80%	12%	0%	0%
Qwen3.5-2B	llm	100%	4%	0%	0%
Qwen3.5-0.8B	llm	68%	12%	0%	0%
Qwen3.5-4B	vlm	100%	8%	0%	0%
Qwen3-4B	llm	72%	20%	8%	0%
Random Agent	other	80%	8%	0%	0%
PPO Sparse (500k)	rl	72%	4%	12%	0%
Gemini 2.5 Flash Lite	llm	72%	8%	0%	0%
Qwen3.5-2B	llm	0%	12%	4%	0%
Qwen3.5-0.8B	llm	20%	12%	0%	0%
Gemini 2.5 Flash Lite	llm	64%	0%	0%	0%
Qwen3-4B	llm	20%	0%	0%	0%
Qwen3.5-2B	llm	0%	4%	0%	0%
Qwen3.5-4B	llm	16%	4%	0%	0%
Qwen3.5-4B	llm	16%	4%	0%	0%
Qwen3.5-0.8B	llm	4%	0%	0%	0%
Qwen3-4B	llm	16%	4%	0%	0%
Qwen3-4B	llm	0%	4%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	44%	–
Gemini 3.1 Flash Lite	llm	–	–	24%	–

Show all agents

Arrange tiles to goal configuration (1,2,3...N-1 in row-major order).

ToolUse-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	100%	100%
GPT-6 Astra (high, 1 seed)	llm	100%	100%	100%	100%
Claude Opus 5 (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Sol (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Terra (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Luna (high, 1 seed)	llm	100%	100%	100%	100%
Qwen3.5-4B (SFT-250k)	llm	100%	100%	100%	100%
Qwen3.5-4B (SFT-250k)	llm	100%	100%	100%	100%
Qwen3.5-4B (SFT-120k)	llm	100%	100%	100%	100%
Qwen3.5-4B (SFT-120k)	llm	100%	100%	100%	100%
GPT-5 mini	llm	100%	100%	100%	100%
PPO Dense (2M)	rl	100%	100%	96%	0%
Qwen3.5-4B	llm	36%	52%	84%	32%
PPO Dense (500k)	rl	56%	52%	100%	0%
Gemini 2.5 Flash Lite	llm	100%	92%	64%	56%
Qwen3.5-4B	llm	52%	56%	0%	28%
Qwen3.5-2B	llm	100%	100%	96%	100%
Qwen3.5-2B	llm	80%	88%	100%	96%
Qwen3.5-0.8B	llm	56%	52%	72%	52%
Qwen3.5-4B	vlm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	8%
Random Agent	other	16%	24%	16%	24%
PPO Sparse (500k)	rl	100%	68%	40%	4%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	100%	–
Gemini 3.1 Flash Lite	llm	–	–	96%	–

Show all agents

Collect all SCROLLs to spawn the ORB, pick up the ORB, cross the river, and reach the GOAL for full reward (1.0).

DeceptiveReward-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	100%	100%
GPT-6 Astra (high, 1 seed)	llm	100%	100%	100%	100%
Claude Opus 5 (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Sol (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Terra (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Luna (high, 1 seed)	llm	100%	100%	100%	100%
Qwen3.5-4B (SFT-250k)	llm	100%	100%	100%	92%
Qwen3.5-4B (SFT-250k)	llm	100%	100%	100%	92%
Qwen3.5-4B (SFT-120k)	llm	100%	100%	72%	72%
Qwen3.5-4B (SFT-120k)	llm	100%	100%	100%	0%
GPT-5 mini	llm	100%	100%	100%	84%
PPO Dense (2M)	rl	100%	100%	100%	100%
Qwen3.5-4B	llm	88%	52%	20%	16%
PPO Dense (500k)	rl	100%	100%	100%	100%
Gemini 2.5 Flash Lite	llm	100%	92%	60%	24%
Qwen3.5-4B	llm	0%	0%	40%	4%
Qwen3.5-2B	llm	68%	56%	24%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	16%	8%	24%	4%
Qwen3.5-4B	vlm	4%	0%	0%	0%
Qwen3-4B	llm	16%	0%	0%	0%
Random Agent	other	4%	0%	4%	0%
PPO Sparse (500k)	rl	100%	0%	12%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	28%	28%	0%	0%
Qwen3.5-0.8B	llm	0%	4%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	4%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	80%	–
Gemini 3.1 Flash Lite	llm	–	–	56%	–

Show all agents

Resist the coin reward gradient.

GraphColoring-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	100%	100%
GPT-6 Astra (high, 1 seed)	llm	100%	0%	100%	100%
Claude Opus 5 (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Sol (high, 1 seed)	llm	100%	0%	100%	100%
GPT-5.6 Terra (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Luna (high, 1 seed)	llm	100%	0%	100%	100%
Qwen3.5-4B (SFT-250k)	llm	76%	16%	0%	4%
Qwen3.5-4B (SFT-250k)	llm	60%	24%	4%	0%
Qwen3.5-4B (SFT-120k)	llm	24%	0%	0%	0%
Qwen3.5-4B (SFT-120k)	llm	20%	8%	0%	0%
GPT-5 mini	llm	0%	0%	0%	0%
PPO Dense (2M)	rl	32%	0%	0%	0%
Qwen3.5-4B	llm	16%	4%	4%	0%
PPO Dense (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	4%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3.5-4B	vlm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Random Agent	other	0%	0%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	0%	–
Gemini 3.1 Flash Lite	llm	–	–	0%	–

Show all agents

Color all nodes so no two adjacent nodes share the same color.

LightsOut-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	88%	92%	100%
GPT-6 Astra (high, 1 seed)	llm	0%	100%	100%	100%
Claude Opus 5 (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Sol (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Terra (high, 1 seed)	llm	100%	0%	0%	0%
GPT-5.6 Luna (high, 1 seed)	llm	100%	0%	0%	0%
Qwen3.5-4B (SFT-250k)	llm	48%	24%	72%	40%
Qwen3.5-4B (SFT-250k)	llm	40%	32%	56%	56%
Qwen3.5-4B (SFT-120k)	llm	36%	12%	16%	4%
Qwen3.5-4B (SFT-120k)	llm	40%	4%	16%	8%
GPT-5 mini	llm	4%	0%	0%	0%
PPO Dense (2M)	rl	100%	12%	0%	0%
Qwen3.5-4B	llm	36%	4%	0%	0%
PPO Dense (500k)	rl	56%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	16%	0%	0%	0%
Qwen3.5-4B	llm	12%	4%	0%	0%
Qwen3.5-2B	llm	4%	0%	0%	0%
Qwen3.5-2B	llm	8%	0%	0%	0%
Qwen3.5-0.8B	llm	8%	0%	0%	0%
Qwen3.5-4B	vlm	4%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Random Agent	other	0%	0%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	4%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	0%	–
Gemini 3.1 Flash Lite	llm	–	–	0%	–

Show all agents

Turn all lights OFF by toggling switches.

ProgramSynthesis-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	100%	100%
GPT-6 Astra (high, 1 seed)	llm	100%	100%	100%	100%
Claude Opus 5 (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Sol (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Terra (high, 1 seed)	llm	100%	100%	0%	100%
GPT-5.6 Luna (high, 1 seed)	llm	100%	0%	100%	0%
Qwen3.5-4B (SFT-250k)	llm	12%	4%	4%	0%
Qwen3.5-4B (SFT-250k)	llm	8%	16%	4%	0%
Qwen3.5-4B (SFT-120k)	llm	12%	0%	0%	0%
Qwen3.5-4B (SFT-120k)	llm	0%	0%	0%	0%
GPT-5 mini	llm	0%	0%	0%	0%
PPO Dense (2M)	rl	4%	0%	0%	0%
Qwen3.5-4B	llm	8%	0%	0%	0%
PPO Dense (500k)	rl	4%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3.5-4B	vlm	4%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Random Agent	other	0%	0%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	0%	–
Gemini 3.1 Flash Lite	llm	–	–	0%	–

Show all agents

Push all GEM objects so they form the same relative pattern as the reference SCROLLs.

RuleInduction-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	88%	96%	100%
GPT-6 Astra (high, 1 seed)	llm	0%	100%	0%	0%
Claude Opus 5 (high, 1 seed)	llm	0%	0%	0%	0%
GPT-5.6 Sol (high, 1 seed)	llm	0%	0%	100%	0%
GPT-5.6 Terra (high, 1 seed)	llm	0%	0%	0%	0%
GPT-5.6 Luna (high, 1 seed)	llm	0%	0%	0%	0%
Qwen3.5-4B (SFT-250k)	llm	12%	36%	36%	40%
Qwen3.5-4B (SFT-250k)	llm	24%	16%	48%	28%
Qwen3.5-4B (SFT-120k)	llm	28%	24%	40%	24%
Qwen3.5-4B (SFT-120k)	llm	40%	48%	44%	28%
GPT-5 mini	llm	0%	0%	0%	0%
PPO Dense (2M)	rl	0%	0%	0%	0%
Qwen3.5-4B	llm	28%	20%	0%	0%
PPO Dense (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	4%	0%	0%	0%
Qwen3.5-4B	llm	0%	4%	4%	4%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	12%	0%	0%
Qwen3.5-0.8B	llm	4%	0%	4%	0%
Qwen3.5-4B	vlm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Random Agent	other	0%	0%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	4%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	0%	–
Gemini 3.1 Flash Lite	llm	–	–	32%	–

Show all agents

Identify real switches via ICE pattern, INTERACT all real ones, pass barrier to GOAL.

SwitchCircuit-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	80%	80%	80%	76%
GPT-6 Astra (high, 1 seed)	llm	100%	100%	100%	100%
Claude Opus 5 (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Sol (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Terra (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Luna (high, 1 seed)	llm	100%	100%	100%	100%
Qwen3.5-4B (SFT-250k)	llm	64%	40%	0%	0%
Qwen3.5-4B (SFT-250k)	llm	72%	48%	0%	0%
Qwen3.5-4B (SFT-120k)	llm	48%	24%	0%	0%
Qwen3.5-4B (SFT-120k)	llm	52%	24%	0%	0%
GPT-5 mini	llm	4%	0%	0%	0%
PPO Dense (2M)	rl	24%	4%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
PPO Dense (500k)	rl	4%	0%	4%	4%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3.5-4B	vlm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Random Agent	other	0%	0%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	0%	–
Gemini 3.1 Flash Lite	llm	–	–	0%	–

Show all agents

Plan switch activation order to open all barriers blocking the path to GOAL.

SymbolMatching-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	100%	100%
GPT-6 Astra (high, 1 seed)	llm	100%	100%	0%	0%
Claude Opus 5 (high, 1 seed)	llm	100%	0%	0%	0%
GPT-5.6 Sol (high, 1 seed)	llm	100%	0%	0%	0%
GPT-5.6 Terra (high, 1 seed)	llm	100%	0%	0%	0%
GPT-5.6 Luna (high, 1 seed)	llm	100%	0%	0%	0%
Qwen3.5-4B (SFT-250k)	llm	64%	20%	8%	12%
Qwen3.5-4B (SFT-250k)	llm	68%	12%	8%	12%
Qwen3.5-4B (SFT-120k)	llm	64%	24%	4%	4%
Qwen3.5-4B (SFT-120k)	llm	60%	8%	0%	0%
GPT-5 mini	llm	0%	0%	0%	0%
PPO Dense (2M)	rl	32%	4%	0%	0%
Qwen3.5-4B	llm	48%	0%	0%	0%
PPO Dense (500k)	rl	16%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	8%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	4%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3.5-4B	vlm	28%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Random Agent	other	4%	0%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	0%	–
Gemini 3.1 Flash Lite	llm	–	–	0%	–

Show all agents

Pick up each symbol item and deliver it to the matching target of the same type on the right side of the grid.

TaskInterference-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	100%	100%
GPT-6 Astra (high, 1 seed)	llm	100%	100%	100%	100%
Claude Opus 5 (high, 1 seed)	llm	100%	100%	100%	100%
GPT-5.6 Sol (high, 1 seed)	llm	100%	100%	0%	0%
GPT-5.6 Terra (high, 1 seed)	llm	0%	100%	0%	0%
GPT-5.6 Luna (high, 1 seed)	llm	100%	0%	0%	0%
Qwen3.5-4B (SFT-250k)	llm	100%	100%	100%	100%
Qwen3.5-4B (SFT-250k)	llm	100%	100%	100%	100%
Qwen3.5-4B (SFT-120k)	llm	100%	100%	100%	88%
Qwen3.5-4B (SFT-120k)	llm	100%	100%	100%	92%
GPT-5 mini	llm	16%	8%	4%	0%
PPO Dense (2M)	rl	0%	0%	0%	0%
Qwen3.5-4B	llm	52%	0%	0%	0%
PPO Dense (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	4%	0%	0%	0%
Qwen3.5-4B	llm	4%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3.5-4B	vlm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Random Agent	other	0%	0%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	12%	–
Gemini 3.1 Flash Lite	llm	–	–	0%	–

Show all agents

Raise both GEM and ORB meters to >= threshold simultaneously.