Agentick Leaderboard

Universal benchmark for evaluating AI agents.

ONS (Oracle-Normalized Score) = (agent − random) / (oracle − random), where 0.0 = random baseline and 1.0 = oracle upper bound.

Oracle-Normalized Score (ONS)

Category ONS Breakdown

Rank	Agent	Type	Modality	Harness	Score	95% CI	Open	Date
1	Oracle Agent	other	–	–	0.895	0.811–0.969	No	2026-03-17
2	Qwen3.5-4B (SFT-250k)	llm	ascii	markovian_zero_shot	0.447	0.380–0.518	Yes	2026-05-11
3	Qwen3.5-4B (SFT-250k)	llm	ascii	markovian_reasoner	0.444	0.372–0.518	Yes	2026-05-11
4	Qwen3.5-4B (SFT-120k)	llm	ascii	markovian_zero_shot	0.354	0.294–0.408	Yes	2026-05-11
5	Qwen3.5-4B (SFT-120k)	llm	ascii	markovian_reasoner	0.349	0.290–0.402	Yes	2026-05-11
6	GPT-5 mini	llm	ascii	MarkovianReasoner	0.309	0.000–0.000	No	2026-03-20
7	PPO Dense (2M)	rl	rgb_array	–	0.287	0.212–0.367	Yes	2026-03-22
8	Qwen3.5-4B	llm	ascii	markovian_reasoner	0.228	0.161–0.292	Yes	2026-03-25
9	PPO Dense (500k)	rl	rgb_array	–	0.226	0.166–0.287	Yes	2026-03-20
10	Gemini 2.5 Flash Lite	llm	ascii	markovian_reasoner	0.187	0.126–0.247	No	2026-03-17
11	Qwen3.5-4B	llm	language	markovian_reasoner	0.181	0.085–0.272	Yes	2026-03-25
12	Qwen3.5-2B	llm	ascii	markovian_reasoner	0.133	0.069–0.195	Yes	2026-03-25
13	Qwen3.5-2B	llm	language	markovian_reasoner	0.122	0.056–0.183	Yes	2026-03-25
14	Qwen3.5-0.8B	llm	ascii	markovian_reasoner	0.094	0.048–0.140	Yes	2026-03-22
15	Qwen3-4B	llm	ascii	markovian_reasoner	0.085	0.000–0.000	Yes	2026-03-21
16	Random Agent	other	–	–	0.082	0.031–0.130	No	2026-03-17
17	PPO Sparse (500k)	rl	rgb_array	–	0.074	0.051–0.097	Yes	2026-03-20
18	Gemini 2.5 Flash Lite	llm	language	markovian_zero_shot	0.064	0.025–0.102	No	2026-03-17
19	Qwen3.5-2B	llm	ascii	markovian_zero_shot	0.062	0.032–0.093	Yes	2026-03-22
20	Qwen3.5-0.8B	llm	language	markovian_reasoner	0.061	0.021–0.100	Yes	2026-03-22
21	Gemini 2.5 Flash Lite	llm	ascii	markovian_zero_shot	0.053	0.022–0.080	No	2026-03-17
22	Qwen3-4B	llm	language	markovian_reasoner	0.050	0.000–0.000	Yes	2026-03-21
23	Qwen3.5-2B	llm	language	markovian_zero_shot	0.031	0.012–0.050	Yes	2026-03-22
24	Qwen3.5-4B	llm	ascii	markovian_zero_shot	0.023	0.007–0.042	Yes	2026-03-22
25	Qwen3.5-4B	llm	language	markovian_zero_shot	0.020	0.006–0.038	Yes	2026-03-22
26	Qwen3.5-0.8B	llm	ascii	markovian_zero_shot	0.020	0.006–0.035	Yes	2026-03-25
27	Qwen3-4B	llm	ascii	markovian_zero_shot	0.020	0.000–0.000	Yes	2026-03-21
28	Qwen3-4B	llm	language	markovian_zero_shot	0.019	0.000–0.000	Yes	2026-03-21
29	Qwen3.5-0.8B	llm	language	markovian_zero_shot	0.016	0.002–0.035	Yes	2026-03-22

Generalization ONS

Rank	Agent	Type	Modality	Harness	Score
1	Oracle Agent	other	–	–	0.837
2	Qwen3.5-4B (SFT-250k)	llm	ascii	markovian_zero_shot	0.350
3	Qwen3.5-4B (SFT-250k)	llm	ascii	markovian_reasoner	0.337
4	Qwen3.5-4B (SFT-120k)	llm	ascii	markovian_zero_shot	0.357
5	Qwen3.5-4B (SFT-120k)	llm	ascii	markovian_reasoner	0.347
6	GPT-5 mini	llm	ascii	MarkovianReasoner	0.437
7	PPO Dense (2M)	rl	rgb_array	–	0.163
8	Qwen3.5-4B	llm	ascii	markovian_reasoner	0.327
9	PPO Dense (500k)	rl	rgb_array	–	0.130
10	Gemini 2.5 Flash Lite	llm	ascii	markovian_reasoner	0.287
11	Qwen3.5-4B	llm	language	markovian_reasoner	0.340
12	Qwen3.5-2B	llm	ascii	markovian_reasoner	0.133
13	Qwen3.5-2B	llm	language	markovian_reasoner	0.170
14	Qwen3.5-0.8B	llm	ascii	markovian_reasoner	0.143
15	Qwen3-4B	llm	ascii	markovian_reasoner	0.133
16	Random Agent	other	–	–	0.150
17	PPO Sparse (500k)	rl	rgb_array	–	0.040
18	Gemini 2.5 Flash Lite	llm	language	markovian_zero_shot	0.123
19	Qwen3.5-2B	llm	ascii	markovian_zero_shot	0.120
20	Qwen3.5-0.8B	llm	language	markovian_reasoner	0.130
21	Gemini 2.5 Flash Lite	llm	ascii	markovian_zero_shot	0.087
22	Qwen3-4B	llm	language	markovian_reasoner	0.120
23	Qwen3.5-2B	llm	language	markovian_zero_shot	0.067
24	Qwen3.5-4B	llm	ascii	markovian_zero_shot	0.020
25	Qwen3.5-4B	llm	language	markovian_zero_shot	0.027
26	Qwen3.5-0.8B	llm	ascii	markovian_zero_shot	0.043
27	Qwen3-4B	llm	ascii	markovian_zero_shot	0.017
28	Qwen3-4B	llm	language	markovian_zero_shot	0.030
29	Qwen3.5-0.8B	llm	language	markovian_zero_shot	0.033

Memory ONS

Rank	Agent	Type	Modality	Harness	Score
1	Oracle Agent	other	–	–	0.980
2	Qwen3.5-4B (SFT-250k)	llm	ascii	markovian_zero_shot	0.412
3	Qwen3.5-4B (SFT-250k)	llm	ascii	markovian_reasoner	0.422
4	Qwen3.5-4B (SFT-120k)	llm	ascii	markovian_zero_shot	0.287
5	Qwen3.5-4B (SFT-120k)	llm	ascii	markovian_reasoner	0.292
6	GPT-5 mini	llm	ascii	MarkovianReasoner	0.347
7	PPO Dense (2M)	rl	rgb_array	–	0.282
8	Qwen3.5-4B	llm	ascii	markovian_reasoner	0.247
9	PPO Dense (500k)	rl	rgb_array	–	0.228
10	Gemini 2.5 Flash Lite	llm	ascii	markovian_reasoner	0.163
11	Qwen3.5-4B	llm	language	markovian_reasoner	0.295
12	Qwen3.5-2B	llm	ascii	markovian_reasoner	0.212
13	Qwen3.5-2B	llm	language	markovian_reasoner	0.135
14	Qwen3.5-0.8B	llm	ascii	markovian_reasoner	0.133
15	Qwen3-4B	llm	ascii	markovian_reasoner	0.152
16	Random Agent	other	–	–	0.117
17	PPO Sparse (500k)	rl	rgb_array	–	0.102
18	Gemini 2.5 Flash Lite	llm	language	markovian_zero_shot	0.105
19	Qwen3.5-2B	llm	ascii	markovian_zero_shot	0.065
20	Qwen3.5-0.8B	llm	language	markovian_reasoner	0.090
21	Gemini 2.5 Flash Lite	llm	ascii	markovian_zero_shot	0.083
22	Qwen3-4B	llm	language	markovian_reasoner	0.050
23	Qwen3.5-2B	llm	language	markovian_zero_shot	0.055
24	Qwen3.5-4B	llm	ascii	markovian_zero_shot	0.037
25	Qwen3.5-4B	llm	language	markovian_zero_shot	0.020
26	Qwen3.5-0.8B	llm	ascii	markovian_zero_shot	0.030
27	Qwen3-4B	llm	ascii	markovian_zero_shot	0.025
28	Qwen3-4B	llm	language	markovian_zero_shot	0.015
29	Qwen3.5-0.8B	llm	language	markovian_zero_shot	0.003

Multi Agent ONS

Rank	Agent	Type	Modality	Harness	Score
1	Oracle Agent	other	–	–	0.692
2	Qwen3.5-4B (SFT-250k)	llm	ascii	markovian_zero_shot	0.348
3	Qwen3.5-4B (SFT-250k)	llm	ascii	markovian_reasoner	0.338
4	Qwen3.5-4B (SFT-120k)	llm	ascii	markovian_zero_shot	0.246
5	Qwen3.5-4B (SFT-120k)	llm	ascii	markovian_reasoner	0.238
6	GPT-5 mini	llm	ascii	MarkovianReasoner	0.150
7	PPO Dense (2M)	rl	rgb_array	–	0.432
8	Qwen3.5-4B	llm	ascii	markovian_reasoner	0.134
9	PPO Dense (500k)	rl	rgb_array	–	0.352
10	Gemini 2.5 Flash Lite	llm	ascii	markovian_reasoner	0.098
11	Qwen3.5-4B	llm	language	markovian_reasoner	0.072
12	Qwen3.5-2B	llm	ascii	markovian_reasoner	0.032
13	Qwen3.5-2B	llm	language	markovian_reasoner	0.050
14	Qwen3.5-0.8B	llm	ascii	markovian_reasoner	0.036
15	Qwen3-4B	llm	ascii	markovian_reasoner	0.038
16	Random Agent	other	–	–	0.030
17	PPO Sparse (500k)	rl	rgb_array	–	0.066
18	Gemini 2.5 Flash Lite	llm	language	markovian_zero_shot	0.022
19	Qwen3.5-2B	llm	ascii	markovian_zero_shot	0.022
20	Qwen3.5-0.8B	llm	language	markovian_reasoner	0.014
21	Gemini 2.5 Flash Lite	llm	ascii	markovian_zero_shot	0.020
22	Qwen3-4B	llm	language	markovian_reasoner	0.020
23	Qwen3.5-2B	llm	language	markovian_zero_shot	0.006
24	Qwen3.5-4B	llm	ascii	markovian_zero_shot	0.008
25	Qwen3.5-4B	llm	language	markovian_zero_shot	0.006
26	Qwen3.5-0.8B	llm	ascii	markovian_zero_shot	0.004
27	Qwen3-4B	llm	ascii	markovian_zero_shot	0.008
28	Qwen3-4B	llm	language	markovian_zero_shot	0.006
29	Qwen3.5-0.8B	llm	language	markovian_zero_shot	0.004

Navigation ONS

Rank	Agent	Type	Modality	Harness	Score
1	Oracle Agent	other	–	–	0.975
2	Qwen3.5-4B (SFT-250k)	llm	ascii	markovian_zero_shot	0.568
3	Qwen3.5-4B (SFT-250k)	llm	ascii	markovian_reasoner	0.545
4	Qwen3.5-4B (SFT-120k)	llm	ascii	markovian_zero_shot	0.427
5	Qwen3.5-4B (SFT-120k)	llm	ascii	markovian_reasoner	0.425
6	GPT-5 mini	llm	ascii	MarkovianReasoner	0.456
7	PPO Dense (2M)	rl	rgb_array	–	0.250
8	Qwen3.5-4B	llm	ascii	markovian_reasoner	0.223
9	PPO Dense (500k)	rl	rgb_array	–	0.193
10	Gemini 2.5 Flash Lite	llm	ascii	markovian_reasoner	0.237
11	Qwen3.5-4B	llm	language	markovian_reasoner	0.136
12	Qwen3.5-2B	llm	ascii	markovian_reasoner	0.136
13	Qwen3.5-2B	llm	language	markovian_reasoner	0.128
14	Qwen3.5-0.8B	llm	ascii	markovian_reasoner	0.069
15	Qwen3-4B	llm	ascii	markovian_reasoner	0.072
16	Random Agent	other	–	–	0.040
17	PPO Sparse (500k)	rl	rgb_array	–	0.089
18	Gemini 2.5 Flash Lite	llm	language	markovian_zero_shot	0.037
19	Qwen3.5-2B	llm	ascii	markovian_zero_shot	0.055
20	Qwen3.5-0.8B	llm	language	markovian_reasoner	0.033
21	Gemini 2.5 Flash Lite	llm	ascii	markovian_zero_shot	0.036
22	Qwen3-4B	llm	language	markovian_reasoner	0.040
23	Qwen3.5-2B	llm	language	markovian_zero_shot	0.024
24	Qwen3.5-4B	llm	ascii	markovian_zero_shot	0.013
25	Qwen3.5-4B	llm	language	markovian_zero_shot	0.007
26	Qwen3.5-0.8B	llm	ascii	markovian_zero_shot	0.005
27	Qwen3-4B	llm	ascii	markovian_zero_shot	0.007
28	Qwen3-4B	llm	language	markovian_zero_shot	0.007
29	Qwen3.5-0.8B	llm	language	markovian_zero_shot	0.001

Planning ONS

Rank	Agent	Type	Modality	Harness	Score
1	Oracle Agent	other	–	–	0.928
2	Qwen3.5-4B (SFT-250k)	llm	ascii	markovian_zero_shot	0.557
3	Qwen3.5-4B (SFT-250k)	llm	ascii	markovian_reasoner	0.573
4	Qwen3.5-4B (SFT-120k)	llm	ascii	markovian_zero_shot	0.459
5	Qwen3.5-4B (SFT-120k)	llm	ascii	markovian_reasoner	0.450
6	GPT-5 mini	llm	ascii	MarkovianReasoner	0.334
7	PPO Dense (2M)	rl	rgb_array	–	0.402
8	Qwen3.5-4B	llm	ascii	markovian_reasoner	0.313
9	PPO Dense (500k)	rl	rgb_array	–	0.300
10	Gemini 2.5 Flash Lite	llm	ascii	markovian_reasoner	0.249
11	Qwen3.5-4B	llm	language	markovian_reasoner	0.219
12	Qwen3.5-2B	llm	ascii	markovian_reasoner	0.237
13	Qwen3.5-2B	llm	language	markovian_reasoner	0.244
14	Qwen3.5-0.8B	llm	ascii	markovian_reasoner	0.164
15	Qwen3-4B	llm	ascii	markovian_reasoner	0.106
16	Random Agent	other	–	–	0.150
17	PPO Sparse (500k)	rl	rgb_array	–	0.114
18	Gemini 2.5 Flash Lite	llm	language	markovian_zero_shot	0.098
19	Qwen3.5-2B	llm	ascii	markovian_zero_shot	0.090
20	Qwen3.5-0.8B	llm	language	markovian_reasoner	0.100
21	Gemini 2.5 Flash Lite	llm	ascii	markovian_zero_shot	0.092
22	Qwen3-4B	llm	language	markovian_reasoner	0.071
23	Qwen3.5-2B	llm	language	markovian_zero_shot	0.033
24	Qwen3.5-4B	llm	ascii	markovian_zero_shot	0.061
25	Qwen3.5-4B	llm	language	markovian_zero_shot	0.061
26	Qwen3.5-0.8B	llm	ascii	markovian_zero_shot	0.038
27	Qwen3-4B	llm	ascii	markovian_zero_shot	0.061
28	Qwen3-4B	llm	language	markovian_zero_shot	0.057
29	Qwen3.5-0.8B	llm	language	markovian_zero_shot	0.057

Reasoning ONS

Rank	Agent	Type	Modality	Harness	Score
1	Oracle Agent	other	–	–	0.961
2	Qwen3.5-4B (SFT-250k)	llm	ascii	markovian_zero_shot	0.445
3	Qwen3.5-4B (SFT-250k)	llm	ascii	markovian_reasoner	0.446
4	Qwen3.5-4B (SFT-120k)	llm	ascii	markovian_zero_shot	0.350
5	Qwen3.5-4B (SFT-120k)	llm	ascii	markovian_reasoner	0.341
6	GPT-5 mini	llm	ascii	MarkovianReasoner	0.131
7	PPO Dense (2M)	rl	rgb_array	–	0.191
8	Qwen3.5-4B	llm	ascii	markovian_reasoner	0.124
9	PPO Dense (500k)	rl	rgb_array	–	0.152
10	Gemini 2.5 Flash Lite	llm	ascii	markovian_reasoner	0.090
11	Qwen3.5-4B	llm	language	markovian_reasoner	0.024
12	Qwen3.5-2B	llm	ascii	markovian_reasoner	0.048
13	Qwen3.5-2B	llm	language	markovian_reasoner	0.007
14	Qwen3.5-0.8B	llm	ascii	markovian_reasoner	0.021
15	Qwen3-4B	llm	ascii	markovian_reasoner	0.005
16	Random Agent	other	–	–	0.003
17	PPO Sparse (500k)	rl	rgb_array	–	0.035
18	Gemini 2.5 Flash Lite	llm	language	markovian_zero_shot	0.000
19	Qwen3.5-2B	llm	ascii	markovian_zero_shot	0.018
20	Qwen3.5-0.8B	llm	language	markovian_reasoner	0.003
21	Gemini 2.5 Flash Lite	llm	ascii	markovian_zero_shot	0.000
22	Qwen3-4B	llm	language	markovian_reasoner	0.000
23	Qwen3.5-2B	llm	language	markovian_zero_shot	0.001
24	Qwen3.5-4B	llm	ascii	markovian_zero_shot	0.000
25	Qwen3.5-4B	llm	language	markovian_zero_shot	0.000
26	Qwen3.5-0.8B	llm	ascii	markovian_zero_shot	0.001
27	Qwen3-4B	llm	ascii	markovian_zero_shot	0.000
28	Qwen3-4B	llm	language	markovian_zero_shot	0.000
29	Qwen3.5-0.8B	llm	language	markovian_zero_shot	0.000

DistributionShift-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	100%	48%
Qwen3.5-4B (SFT-250k)	llm	8%	4%	0%	0%
Qwen3.5-4B (SFT-250k)	llm	8%	0%	0%	0%
Qwen3.5-4B (SFT-120k)	llm	0%	0%	0%	0%
Qwen3.5-4B (SFT-120k)	llm	4%	0%	0%	0%
GPT-5 mini	llm	8%	4%	4%	0%
PPO Dense (2M)	rl	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
PPO Dense (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Random Agent	other	0%	4%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	4%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%

Show all agents

Reach all 3 goals across shifting maze phases.

FewShotAdaptation-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	92%	80%	72%	80%
Qwen3.5-4B (SFT-250k)	llm	40%	20%	16%	20%
Qwen3.5-4B (SFT-250k)	llm	24%	20%	12%	28%
Qwen3.5-4B (SFT-120k)	llm	28%	32%	20%	24%
Qwen3.5-4B (SFT-120k)	llm	36%	28%	8%	24%
GPT-5 mini	llm	56%	44%	28%	44%
PPO Dense (2M)	rl	0%	0%	0%	0%
Qwen3.5-4B	llm	36%	44%	28%	24%
PPO Dense (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	28%	44%	32%	24%
Qwen3.5-4B	llm	56%	28%	32%	28%
Qwen3.5-2B	llm	28%	28%	0%	0%
Qwen3.5-2B	llm	8%	32%	24%	0%
Qwen3.5-0.8B	llm	24%	24%	24%	0%
Qwen3-4B	llm	12%	24%	8%	12%
Random Agent	other	28%	20%	12%	12%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	12%	12%	8%	12%
Qwen3.5-2B	llm	24%	12%	16%	0%
Qwen3.5-0.8B	llm	12%	32%	8%	16%
Gemini 2.5 Flash Lite	llm	20%	16%	20%	0%
Qwen3-4B	llm	16%	24%	8%	24%
Qwen3.5-2B	llm	24%	4%	12%	4%
Qwen3.5-4B	llm	0%	4%	4%	0%
Qwen3.5-4B	llm	4%	4%	0%	0%
Qwen3.5-0.8B	llm	16%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	8%	0%	4%
Qwen3.5-0.8B	llm	0%	0%	0%	0%

Show all agents

Watch demo trials to infer the hidden rule, then navigate to the correct candidate object in the test trial.

NoisyObservation-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	80%	84%	68%
Qwen3.5-4B (SFT-250k)	llm	92%	72%	84%	64%
Qwen3.5-4B (SFT-250k)	llm	92%	72%	88%	60%
Qwen3.5-4B (SFT-120k)	llm	96%	80%	92%	56%
Qwen3.5-4B (SFT-120k)	llm	92%	80%	88%	56%
GPT-5 mini	llm	100%	84%	84%	68%
PPO Dense (2M)	rl	100%	56%	24%	16%
Qwen3.5-4B	llm	88%	68%	60%	44%
PPO Dense (500k)	rl	96%	28%	8%	24%
Gemini 2.5 Flash Lite	llm	88%	68%	32%	28%
Qwen3.5-4B	llm	92%	72%	68%	32%
Qwen3.5-2B	llm	60%	36%	8%	0%
Qwen3.5-2B	llm	76%	40%	20%	4%
Qwen3.5-0.8B	llm	48%	36%	8%	8%
Qwen3-4B	llm	56%	32%	4%	12%
Random Agent	other	68%	28%	8%	0%
PPO Sparse (500k)	rl	28%	4%	12%	4%
Gemini 2.5 Flash Lite	llm	60%	28%	8%	8%
Qwen3.5-2B	llm	48%	28%	8%	8%
Qwen3.5-0.8B	llm	44%	32%	4%	8%
Gemini 2.5 Flash Lite	llm	20%	8%	8%	8%
Qwen3-4B	llm	44%	16%	12%	0%
Qwen3.5-2B	llm	0%	24%	4%	8%
Qwen3.5-4B	llm	12%	4%	0%	0%
Qwen3.5-4B	llm	12%	4%	4%	4%
Qwen3.5-0.8B	llm	36%	0%	0%	0%
Qwen3-4B	llm	12%	4%	0%	4%
Qwen3-4B	llm	16%	8%	0%	0%
Qwen3.5-0.8B	llm	20%	20%	0%	0%

Show all agents

Locate and reach the true GOAL amid visual noise.

DelayedGratification-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	100%	100%
Qwen3.5-4B (SFT-250k)	llm	96%	84%	64%	100%
Qwen3.5-4B (SFT-250k)	llm	96%	80%	56%	100%
Qwen3.5-4B (SFT-120k)	llm	76%	60%	32%	56%
Qwen3.5-4B (SFT-120k)	llm	76%	56%	36%	48%
GPT-5 mini	llm	100%	100%	56%	100%
PPO Dense (2M)	rl	100%	88%	20%	0%
Qwen3.5-4B	llm	84%	36%	20%	72%
PPO Dense (500k)	rl	100%	28%	0%	100%
Gemini 2.5 Flash Lite	llm	88%	60%	24%	64%
Qwen3.5-4B	llm	64%	24%	0%	52%
Qwen3.5-2B	llm	60%	52%	20%	36%
Qwen3.5-2B	llm	36%	8%	0%	20%
Qwen3.5-0.8B	llm	16%	16%	0%	16%
Qwen3-4B	llm	56%	28%	8%	32%
Random Agent	other	8%	0%	0%	0%
PPO Sparse (500k)	rl	48%	16%	0%	8%
Gemini 2.5 Flash Lite	llm	20%	0%	0%	24%
Qwen3.5-2B	llm	12%	0%	0%	24%
Qwen3.5-0.8B	llm	20%	4%	0%	4%
Gemini 2.5 Flash Lite	llm	20%	0%	4%	32%
Qwen3-4B	llm	16%	8%	0%	0%
Qwen3.5-2B	llm	16%	4%	0%	12%
Qwen3.5-4B	llm	12%	4%	0%	8%
Qwen3.5-4B	llm	12%	4%	0%	8%
Qwen3.5-0.8B	llm	16%	0%	4%	0%
Qwen3-4B	llm	16%	4%	0%	8%
Qwen3-4B	llm	4%	0%	0%	0%
Qwen3.5-0.8B	llm	4%	0%	0%	0%

Show all agents

Reach the distant true GOAL without collecting any decoy KEY.

FogOfWarExploration-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	92%	76%
Qwen3.5-4B (SFT-250k)	llm	96%	76%	52%	44%
Qwen3.5-4B (SFT-250k)	llm	100%	72%	60%	56%
Qwen3.5-4B (SFT-120k)	llm	72%	72%	36%	24%
Qwen3.5-4B (SFT-120k)	llm	72%	76%	40%	24%
GPT-5 mini	llm	96%	56%	12%	12%
PPO Dense (2M)	rl	92%	36%	12%	28%
Qwen3.5-4B	llm	84%	60%	12%	12%
PPO Dense (500k)	rl	56%	24%	12%	8%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	88%	96%	76%	64%
Qwen3.5-2B	llm	72%	52%	0%	24%
Qwen3.5-2B	llm	48%	44%	28%	16%
Qwen3.5-0.8B	llm	56%	40%	20%	20%
Qwen3-4B	llm	64%	20%	16%	16%
Random Agent	other	52%	56%	12%	16%
PPO Sparse (500k)	rl	40%	20%	0%	0%
Gemini 2.5 Flash Lite	llm	52%	36%	16%	12%
Qwen3.5-2B	llm	36%	24%	4%	4%
Qwen3.5-0.8B	llm	44%	36%	12%	0%
Gemini 2.5 Flash Lite	llm	36%	24%	8%	0%
Qwen3-4B	llm	28%	12%	0%	4%
Qwen3.5-2B	llm	0%	28%	8%	16%
Qwen3.5-4B	llm	16%	8%	12%	0%
Qwen3.5-4B	llm	0%	0%	4%	0%
Qwen3.5-0.8B	llm	20%	4%	0%	4%
Qwen3-4B	llm	4%	0%	4%	0%
Qwen3-4B	llm	12%	8%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%

Show all agents

Find and reach the GOAL despite incomplete map information.

SequenceMemory-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	100%	100%
Qwen3.5-4B (SFT-250k)	llm	24%	12%	0%	4%
Qwen3.5-4B (SFT-250k)	llm	32%	16%	0%	0%
Qwen3.5-4B (SFT-120k)	llm	16%	0%	4%	0%
Qwen3.5-4B (SFT-120k)	llm	28%	12%	0%	0%
GPT-5 mini	llm	0%	0%	0%	0%
PPO Dense (2M)	rl	76%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
PPO Dense (500k)	rl	32%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	8%	0%	0%	0%
Qwen3.5-2B	llm	4%	4%	0%	0%
Qwen3.5-0.8B	llm	4%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Random Agent	other	12%	0%	0%	0%
PPO Sparse (500k)	rl	32%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	12%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%

Show all agents

Memorize shown GEM positions, then visit them in exact order during reproduce phase.

TreasureHunt-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	100%	100%
Qwen3.5-4B (SFT-250k)	llm	8%	0%	0%	0%
Qwen3.5-4B (SFT-250k)	llm	8%	0%	0%	0%
Qwen3.5-4B (SFT-120k)	llm	8%	4%	0%	0%
Qwen3.5-4B (SFT-120k)	llm	0%	0%	0%	0%
GPT-5 mini	llm	20%	4%	0%	0%
PPO Dense (2M)	rl	0%	0%	0%	0%
Qwen3.5-4B	llm	16%	0%	0%	0%
PPO Dense (500k)	rl	4%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	16%	8%	0%	0%
Qwen3.5-4B	llm	8%	0%	0%	0%
Qwen3.5-2B	llm	8%	8%	0%	0%
Qwen3.5-2B	llm	8%	0%	0%	0%
Qwen3.5-0.8B	llm	24%	0%	0%	0%
Qwen3-4B	llm	4%	0%	0%	0%
Random Agent	other	28%	4%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	8%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	8%	4%	0%	0%
Gemini 2.5 Flash Lite	llm	8%	0%	0%	0%
Qwen3-4B	llm	12%	0%	0%	0%
Qwen3.5-2B	llm	0%	4%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	4%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	4%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%

Show all agents

Read scroll clues, triangulate hidden treasure positions, and step on each treasure cell to collect all treasures.

ChaseEvade-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	84%	84%	72%	16%
Qwen3.5-4B (SFT-250k)	llm	76%	48%	16%	16%
Qwen3.5-4B (SFT-250k)	llm	64%	32%	8%	12%
Qwen3.5-4B (SFT-120k)	llm	32%	48%	12%	0%
Qwen3.5-4B (SFT-120k)	llm	36%	40%	12%	0%
GPT-5 mini	llm	4%	0%	0%	0%
PPO Dense (2M)	rl	100%	40%	52%	8%
Qwen3.5-4B	llm	0%	0%	0%	0%
PPO Dense (500k)	rl	96%	32%	60%	12%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Random Agent	other	0%	0%	0%	0%
PPO Sparse (500k)	rl	40%	60%	20%	8%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%

Show all agents

Survive the required steps without enemy collision.

CooperativeTransport-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	84%	64%	52%
Qwen3.5-4B (SFT-250k)	llm	32%	0%	0%	0%
Qwen3.5-4B (SFT-250k)	llm	40%	0%	0%	0%
Qwen3.5-4B (SFT-120k)	llm	20%	0%	0%	0%
Qwen3.5-4B (SFT-120k)	llm	16%	0%	0%	0%
GPT-5 mini	llm	8%	0%	0%	0%
PPO Dense (2M)	rl	0%	0%	0%	0%
Qwen3.5-4B	llm	48%	0%	0%	0%
PPO Dense (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	12%	0%	0%	0%
Qwen3.5-2B	llm	4%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	4%	0%	0%	0%
Random Agent	other	0%	0%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	4%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	4%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%

Show all agents

Push all heavy boxes into holes with NPC cooperation.

EmergentStrategy-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	88%	92%	52%	48%
Qwen3.5-4B (SFT-250k)	llm	60%	32%	16%	0%
Qwen3.5-4B (SFT-250k)	llm	56%	24%	4%	0%
Qwen3.5-4B (SFT-120k)	llm	52%	36%	4%	0%
Qwen3.5-4B (SFT-120k)	llm	36%	40%	0%	8%
GPT-5 mini	llm	92%	8%	16%	0%
PPO Dense (2M)	rl	96%	0%	0%	0%
Qwen3.5-4B	llm	52%	4%	0%	0%
PPO Dense (500k)	rl	4%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	76%	8%	4%	4%
Qwen3.5-4B	llm	20%	8%	0%	0%
Qwen3.5-2B	llm	32%	0%	0%	0%
Qwen3.5-2B	llm	24%	4%	0%	0%
Qwen3.5-0.8B	llm	36%	4%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Random Agent	other	20%	4%	4%	0%
PPO Sparse (500k)	rl	4%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	4%	0%	0%	0%
Qwen3.5-2B	llm	20%	8%	0%	0%
Qwen3.5-0.8B	llm	4%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	12%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%

Show all agents

Exploit NPC behaviors to lure/scare them onto locking pressure plates, permanently opening barriers to reach the GOAL...

Herding-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	80%	76%	44%	36%
Qwen3.5-4B (SFT-250k)	llm	60%	36%	4%	4%
Qwen3.5-4B (SFT-250k)	llm	76%	32%	4%	12%
Qwen3.5-4B (SFT-120k)	llm	20%	12%	0%	4%
Qwen3.5-4B (SFT-120k)	llm	32%	0%	0%	0%
GPT-5 mini	llm	8%	4%	0%	0%
PPO Dense (2M)	rl	100%	52%	32%	0%
Qwen3.5-4B	llm	12%	4%	0%	0%
PPO Dense (500k)	rl	28%	40%	44%	76%
Gemini 2.5 Flash Lite	llm	4%	4%	0%	0%
Qwen3.5-4B	llm	16%	4%	0%	0%
Qwen3.5-2B	llm	0%	4%	0%	0%
Qwen3.5-2B	llm	12%	8%	0%	0%
Qwen3.5-0.8B	llm	0%	4%	0%	0%
Qwen3-4B	llm	4%	8%	0%	0%
Random Agent	other	8%	4%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	4%	4%	8%	0%
Qwen3.5-2B	llm	8%	8%	0%	0%
Qwen3.5-0.8B	llm	8%	4%	0%	0%
Gemini 2.5 Flash Lite	llm	4%	8%	0%	0%
Qwen3-4B	llm	4%	4%	0%	0%
Qwen3.5-2B	llm	8%	4%	0%	0%
Qwen3.5-4B	llm	4%	8%	0%	0%
Qwen3.5-4B	llm	4%	4%	0%	0%
Qwen3.5-0.8B	llm	4%	4%	0%	0%
Qwen3-4B	llm	4%	8%	0%	0%
Qwen3-4B	llm	4%	4%	0%	0%
Qwen3.5-0.8B	llm	4%	4%	0%	0%

Show all agents

Move all SHEEP into the pen zone (TARGET cells).

TagHunt-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	96%	84%	68%	64%
Qwen3.5-4B (SFT-250k)	llm	92%	96%	80%	28%
Qwen3.5-4B (SFT-250k)	llm	100%	100%	84%	28%
Qwen3.5-4B (SFT-120k)	llm	100%	88%	48%	16%
Qwen3.5-4B (SFT-120k)	llm	100%	72%	64%	20%
GPT-5 mini	llm	72%	60%	20%	8%
PPO Dense (2M)	rl	100%	96%	96%	92%
Qwen3.5-4B	llm	88%	48%	8%	4%
PPO Dense (500k)	rl	100%	80%	84%	48%
Gemini 2.5 Flash Lite	llm	92%	4%	0%	0%
Qwen3.5-4B	llm	76%	8%	0%	0%
Qwen3.5-2B	llm	24%	0%	0%	0%
Qwen3.5-2B	llm	52%	0%	0%	0%
Qwen3.5-0.8B	llm	24%	4%	0%	0%
Qwen3-4B	llm	48%	12%	0%	0%
Random Agent	other	20%	0%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	20%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	12%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	16%	0%	0%	0%
Qwen3-4B	llm	28%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	4%	0%	0%	0%
Qwen3.5-4B	llm	4%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	4%	0%	0%	0%
Qwen3-4B	llm	4%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%

Show all agents

Tag all NPCs by stepping onto them.

CuriosityMaze-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	100%	100%
Qwen3.5-4B (SFT-250k)	llm	16%	0%	0%	0%
Qwen3.5-4B (SFT-250k)	llm	12%	0%	0%	0%
Qwen3.5-4B (SFT-120k)	llm	4%	0%	0%	0%
Qwen3.5-4B (SFT-120k)	llm	0%	0%	0%	0%
GPT-5 mini	llm	0%	0%	0%	0%
PPO Dense (2M)	rl	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
PPO Dense (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Random Agent	other	0%	0%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	0%	–
Gemini 3.1 Flash Lite	llm	–	–	0%	–

Show all agents

Visit at least the required percentage of all reachable cells before the step budget runs out.

DynamicObstacles-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	96%	88%	92%	72%
Qwen3.5-4B (SFT-250k)	llm	92%	88%	64%	48%
Qwen3.5-4B (SFT-250k)	llm	92%	92%	60%	40%
Qwen3.5-4B (SFT-120k)	llm	88%	72%	64%	36%
Qwen3.5-4B (SFT-120k)	llm	80%	84%	52%	36%
GPT-5 mini	llm	80%	92%	60%	28%
PPO Dense (2M)	rl	96%	72%	12%	4%
Qwen3.5-4B	llm	64%	60%	36%	0%
PPO Dense (500k)	rl	76%	20%	4%	0%
Gemini 2.5 Flash Lite	llm	64%	68%	32%	8%
Qwen3.5-4B	llm	44%	16%	24%	0%
Qwen3.5-2B	llm	52%	36%	12%	0%
Qwen3.5-2B	llm	36%	32%	8%	0%
Qwen3.5-0.8B	llm	36%	28%	8%	0%
Qwen3-4B	llm	24%	8%	0%	0%
Random Agent	other	8%	0%	0%	0%
PPO Sparse (500k)	rl	60%	12%	12%	0%
Gemini 2.5 Flash Lite	llm	4%	0%	0%	0%
Qwen3.5-2B	llm	16%	4%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	4%	12%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	24%	–
Gemini 3.1 Flash Lite	llm	–	–	44%	–

Show all agents

Reach GOAL without colliding with any NPC.

GoToGoal-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	100%	92%
Qwen3.5-4B (SFT-250k)	llm	100%	92%	76%	68%
Qwen3.5-4B (SFT-250k)	llm	100%	88%	56%	44%
Qwen3.5-4B (SFT-120k)	llm	100%	76%	44%	40%
Qwen3.5-4B (SFT-120k)	llm	100%	76%	28%	28%
GPT-5 mini	llm	100%	88%	56%	44%
PPO Dense (2M)	rl	100%	24%	16%	8%
Qwen3.5-4B	llm	88%	52%	28%	8%
PPO Dense (500k)	rl	100%	40%	12%	0%
Gemini 2.5 Flash Lite	llm	100%	64%	8%	4%
Qwen3.5-4B	llm	68%	20%	0%	4%
Qwen3.5-2B	llm	52%	32%	0%	0%
Qwen3.5-2B	llm	52%	16%	0%	0%
Qwen3.5-0.8B	llm	32%	12%	0%	0%
Qwen3-4B	llm	40%	12%	0%	0%
Random Agent	other	44%	0%	0%	0%
PPO Sparse (500k)	rl	100%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	8%	4%	0%	0%
Qwen3.5-2B	llm	40%	24%	0%	0%
Qwen3.5-0.8B	llm	36%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	28%	16%	8%	4%
Qwen3-4B	llm	40%	4%	0%	4%
Qwen3.5-2B	llm	0%	8%	0%	0%
Qwen3.5-4B	llm	20%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	4%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	20%	–
Gemini 3.1 Flash Lite	llm	–	–	20%	–

Show all agents

Reach the GOAL position.

InstructionFollowing-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	100%	92%
Qwen3.5-4B (SFT-250k)	llm	72%	60%	72%	20%
Qwen3.5-4B (SFT-250k)	llm	76%	76%	72%	24%
Qwen3.5-4B (SFT-120k)	llm	60%	56%	24%	16%
Qwen3.5-4B (SFT-120k)	llm	68%	52%	20%	12%
GPT-5 mini	llm	96%	96%	80%	40%
PPO Dense (2M)	rl	20%	0%	0%	0%
Qwen3.5-4B	llm	64%	68%	4%	0%
PPO Dense (500k)	rl	16%	4%	0%	0%
Gemini 2.5 Flash Lite	llm	52%	56%	0%	0%
Qwen3.5-4B	llm	48%	40%	0%	0%
Qwen3.5-2B	llm	32%	28%	0%	0%
Qwen3.5-2B	llm	16%	16%	0%	0%
Qwen3.5-0.8B	llm	12%	24%	0%	0%
Qwen3-4B	llm	28%	36%	0%	0%
Random Agent	other	12%	20%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	16%	24%	0%	0%
Qwen3.5-2B	llm	4%	8%	0%	0%
Qwen3.5-0.8B	llm	28%	24%	0%	0%
Gemini 2.5 Flash Lite	llm	28%	12%	0%	0%
Qwen3-4B	llm	32%	32%	0%	0%
Qwen3.5-2B	llm	0%	8%	0%	0%
Qwen3.5-4B	llm	12%	4%	0%	0%
Qwen3.5-4B	llm	12%	8%	0%	0%
Qwen3.5-0.8B	llm	4%	8%	0%	0%
Qwen3-4B	llm	12%	8%	0%	0%
Qwen3-4B	llm	12%	4%	0%	0%
Qwen3.5-0.8B	llm	4%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	8%	–
Gemini 3.1 Flash Lite	llm	–	–	0%	–

Show all agents

Reach the unique target object without touching any distractor.

MazeNavigation-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	100%	100%
Qwen3.5-4B (SFT-250k)	llm	100%	28%	24%	4%
Qwen3.5-4B (SFT-250k)	llm	100%	20%	24%	8%
Qwen3.5-4B (SFT-120k)	llm	100%	28%	4%	4%
Qwen3.5-4B (SFT-120k)	llm	100%	20%	4%	4%
GPT-5 mini	llm	80%	16%	4%	0%
PPO Dense (2M)	rl	100%	0%	0%	0%
Qwen3.5-4B	llm	84%	12%	0%	0%
PPO Dense (500k)	rl	100%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	68%	24%	4%	0%
Qwen3.5-4B	llm	72%	4%	0%	0%
Qwen3.5-2B	llm	96%	8%	0%	0%
Qwen3.5-2B	llm	76%	8%	0%	0%
Qwen3.5-0.8B	llm	8%	4%	0%	0%
Qwen3-4B	llm	72%	0%	0%	0%
Random Agent	other	4%	4%	0%	0%
PPO Sparse (500k)	rl	100%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	24%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	4%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	0%	–
Gemini 3.1 Flash Lite	llm	–	–	4%	–

Show all agents

Navigate the maze to reach the GOAL exit.

RecursiveRooms-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	100%	100%
Qwen3.5-4B (SFT-250k)	llm	80%	72%	36%	28%
Qwen3.5-4B (SFT-250k)	llm	84%	68%	40%	28%
Qwen3.5-4B (SFT-120k)	llm	80%	24%	8%	16%
Qwen3.5-4B (SFT-120k)	llm	72%	36%	20%	24%
GPT-5 mini	llm	48%	12%	12%	8%
PPO Dense (2M)	rl	12%	0%	0%	0%
Qwen3.5-4B	llm	80%	32%	0%	0%
PPO Dense (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	40%	4%	8%	8%
Qwen3.5-4B	llm	52%	12%	4%	0%
Qwen3.5-2B	llm	36%	16%	0%	0%
Qwen3.5-2B	llm	28%	12%	0%	12%
Qwen3.5-0.8B	llm	12%	0%	8%	8%
Qwen3-4B	llm	8%	0%	0%	0%
Random Agent	other	12%	0%	0%	4%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	20%	8%	4%	16%
Qwen3.5-0.8B	llm	12%	4%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	4%	0%	0%
Qwen3-4B	llm	8%	4%	4%	0%
Qwen3.5-2B	llm	8%	4%	4%	16%
Qwen3.5-4B	llm	4%	0%	0%	0%
Qwen3.5-4B	llm	4%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	4%	0%	0%	0%
Qwen3-4B	llm	4%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	0%	–
Gemini 3.1 Flash Lite	llm	–	–	12%	–

Show all agents

Navigate through nested rooms to reach GOAL in the deepest room.

ShortestPath-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	96%	96%
Qwen3.5-4B (SFT-250k)	llm	72%	72%	64%	64%
Qwen3.5-4B (SFT-250k)	llm	72%	76%	52%	52%
Qwen3.5-4B (SFT-120k)	llm	68%	64%	24%	36%
Qwen3.5-4B (SFT-120k)	llm	56%	68%	40%	40%
GPT-5 mini	llm	100%	88%	72%	28%
PPO Dense (2M)	rl	100%	72%	0%	0%
Qwen3.5-4B	llm	12%	8%	0%	4%
PPO Dense (500k)	rl	72%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	56%	20%	0%	0%
Qwen3.5-4B	llm	12%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	4%	0%	0%	0%
Random Agent	other	0%	0%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	4%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	4%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	0%	–
Gemini 3.1 Flash Lite	llm	–	–	16%	–

Show all agents

Visit all real GOAL objects within the step budget (optimal path × budget multiplier).

TimingChallenge-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	96%	100%
Qwen3.5-4B (SFT-250k)	llm	96%	68%	60%	80%
Qwen3.5-4B (SFT-250k)	llm	80%	68%	72%	68%
Qwen3.5-4B (SFT-120k)	llm	52%	48%	64%	68%
Qwen3.5-4B (SFT-120k)	llm	44%	60%	72%	64%
GPT-5 mini	llm	36%	24%	36%	36%
PPO Dense (2M)	rl	52%	32%	44%	36%
Qwen3.5-4B	llm	4%	4%	0%	0%
PPO Dense (500k)	rl	56%	40%	48%	28%
Gemini 2.5 Flash Lite	llm	8%	16%	28%	20%
Qwen3.5-4B	llm	0%	8%	8%	0%
Qwen3.5-2B	llm	8%	4%	12%	12%
Qwen3.5-2B	llm	28%	36%	16%	16%
Qwen3.5-0.8B	llm	0%	12%	12%	4%
Qwen3-4B	llm	0%	0%	0%	0%
Random Agent	other	0%	8%	0%	12%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	44%	12%	0%	4%
Qwen3.5-2B	llm	0%	4%	4%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	16%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	4%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	4%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	32%	–
Gemini 3.1 Flash Lite	llm	–	–	32%	–

Show all agents

Cross the patrol zone without collision, then reach GOAL.

BacktrackPuzzle-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	100%	100%
Qwen3.5-4B (SFT-250k)	llm	100%	88%	80%	40%
Qwen3.5-4B (SFT-250k)	llm	100%	96%	76%	56%
Qwen3.5-4B (SFT-120k)	llm	100%	84%	64%	24%
Qwen3.5-4B (SFT-120k)	llm	100%	76%	68%	16%
GPT-5 mini	llm	100%	60%	20%	4%
PPO Dense (2M)	rl	96%	52%	0%	0%
Qwen3.5-4B	llm	88%	32%	20%	24%
PPO Dense (500k)	rl	100%	8%	0%	0%
Gemini 2.5 Flash Lite	llm	16%	8%	8%	8%
Qwen3.5-4B	llm	76%	16%	8%	0%
Qwen3.5-2B	llm	4%	0%	0%	0%
Qwen3.5-2B	llm	4%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Random Agent	other	0%	4%	0%	0%
PPO Sparse (500k)	rl	92%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	4%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	16%	4%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	0%	–
Gemini 3.1 Flash Lite	llm	–	–	60%	–

Show all agents

Activate the correct SWITCH to open the gate, then backtrack to reach GOAL.

KeyDoorPuzzle-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	88%	76%
Qwen3.5-4B (SFT-250k)	llm	96%	20%	0%	0%
Qwen3.5-4B (SFT-250k)	llm	96%	8%	0%	0%
Qwen3.5-4B (SFT-120k)	llm	76%	12%	0%	0%
Qwen3.5-4B (SFT-120k)	llm	64%	0%	0%	0%
GPT-5 mini	llm	100%	44%	4%	0%
PPO Dense (2M)	rl	84%	0%	0%	0%
Qwen3.5-4B	llm	60%	44%	0%	0%
PPO Dense (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	32%	4%	0%	0%
Qwen3.5-4B	llm	36%	28%	0%	0%
Qwen3.5-2B	llm	20%	0%	0%	0%
Qwen3.5-2B	llm	20%	0%	0%	0%
Qwen3.5-0.8B	llm	4%	0%	0%	0%
Qwen3-4B	llm	4%	0%	0%	0%
Random Agent	other	0%	0%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	0%	–
Gemini 3.1 Flash Lite	llm	–	–	0%	–

Show all agents

Reach GOAL after unlocking ALL doors with matching keys.

PackingPuzzle-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	96%	96%	96%	96%
Qwen3.5-4B (SFT-250k)	llm	36%	20%	12%	0%
Qwen3.5-4B (SFT-250k)	llm	32%	20%	24%	4%
Qwen3.5-4B (SFT-120k)	llm	28%	24%	0%	0%
Qwen3.5-4B (SFT-120k)	llm	12%	8%	0%	0%
GPT-5 mini	llm	0%	0%	0%	0%
PPO Dense (2M)	rl	64%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
PPO Dense (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Random Agent	other	0%	0%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	4%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	0%	–
Gemini 3.1 Flash Lite	llm	–	–	0%	–

Show all agents

Push each piece onto its matching-type target slot.

PreciseNavigation-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	56%	32%
Qwen3.5-4B (SFT-250k)	llm	84%	44%	24%	8%
Qwen3.5-4B (SFT-250k)	llm	92%	48%	36%	8%
Qwen3.5-4B (SFT-120k)	llm	92%	32%	20%	0%
Qwen3.5-4B (SFT-120k)	llm	92%	28%	12%	4%
GPT-5 mini	llm	88%	12%	12%	12%
PPO Dense (2M)	rl	64%	0%	8%	4%
Qwen3.5-4B	llm	92%	88%	36%	40%
PPO Dense (500k)	rl	88%	4%	4%	0%
Gemini 2.5 Flash Lite	llm	88%	28%	36%	20%
Qwen3.5-4B	llm	80%	48%	16%	20%
Qwen3.5-2B	llm	76%	28%	20%	16%
Qwen3.5-2B	llm	76%	40%	40%	24%
Qwen3.5-0.8B	llm	40%	12%	20%	4%
Qwen3-4B	llm	48%	4%	4%	8%
Random Agent	other	84%	16%	44%	24%
PPO Sparse (500k)	rl	20%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	36%	4%	8%	0%
Qwen3.5-2B	llm	48%	12%	32%	16%
Qwen3.5-0.8B	llm	52%	16%	24%	8%
Gemini 2.5 Flash Lite	llm	44%	4%	16%	4%
Qwen3-4B	llm	16%	12%	4%	0%
Qwen3.5-2B	llm	0%	12%	4%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	16%	16%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	4%
Claude Haiku 4.5	llm	–	–	32%	–
Gemini 3.1 Flash Lite	llm	–	–	48%	–

Show all agents

Slide across ice to reach the GOAL by planning trajectories through stopping points.

RecipeAssembly-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	100%	100%
Qwen3.5-4B (SFT-250k)	llm	88%	80%	52%	72%
Qwen3.5-4B (SFT-250k)	llm	88%	88%	52%	60%
Qwen3.5-4B (SFT-120k)	llm	56%	48%	16%	20%
Qwen3.5-4B (SFT-120k)	llm	76%	60%	12%	52%
GPT-5 mini	llm	24%	0%	0%	0%
PPO Dense (2M)	rl	88%	0%	0%	0%
Qwen3.5-4B	llm	4%	0%	0%	0%
PPO Dense (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	16%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Random Agent	other	0%	0%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	0%	–
Gemini 3.1 Flash Lite	llm	–	–	0%	–

Show all agents

Collect and deliver all ingredients in recipe order to the crafting station.

ResourceManagement-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	100%	96%
Qwen3.5-4B (SFT-250k)	llm	100%	100%	100%	60%
Qwen3.5-4B (SFT-250k)	llm	100%	100%	100%	68%
Qwen3.5-4B (SFT-120k)	llm	100%	100%	92%	20%
Qwen3.5-4B (SFT-120k)	llm	100%	100%	76%	8%
GPT-5 mini	llm	100%	100%	12%	0%
PPO Dense (2M)	rl	100%	100%	100%	100%
Qwen3.5-4B	llm	100%	100%	0%	0%
PPO Dense (500k)	rl	100%	100%	100%	100%
Gemini 2.5 Flash Lite	llm	100%	100%	0%	0%
Qwen3.5-4B	llm	100%	100%	0%	0%
Qwen3.5-2B	llm	100%	100%	0%	0%
Qwen3.5-2B	llm	100%	100%	0%	0%
Qwen3.5-0.8B	llm	100%	100%	0%	0%
Qwen3-4B	llm	100%	100%	0%	0%
Random Agent	other	100%	100%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	100%	100%	0%	0%
Qwen3.5-2B	llm	100%	100%	0%	0%
Qwen3.5-0.8B	llm	100%	100%	0%	0%
Gemini 2.5 Flash Lite	llm	100%	100%	0%	0%
Qwen3-4B	llm	100%	100%	0%	0%
Qwen3.5-2B	llm	0%	100%	0%	0%
Qwen3.5-4B	llm	100%	100%	0%	0%
Qwen3.5-4B	llm	100%	100%	0%	0%
Qwen3.5-0.8B	llm	100%	0%	0%	0%
Qwen3-4B	llm	100%	100%	0%	0%
Qwen3-4B	llm	100%	100%	0%	0%
Qwen3.5-0.8B	llm	100%	100%	0%	0%
Claude Haiku 4.5	llm	–	–	76%	–
Gemini 3.1 Flash Lite	llm	–	–	48%	–

Show all agents

Keep ALL stations above 0 energy for the entire episode (survive max_steps).

SokobanPush-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	84%	88%	36%
Qwen3.5-4B (SFT-250k)	llm	48%	8%	4%	0%
Qwen3.5-4B (SFT-250k)	llm	48%	12%	0%	0%
Qwen3.5-4B (SFT-120k)	llm	40%	0%	0%	0%
Qwen3.5-4B (SFT-120k)	llm	24%	4%	0%	0%
GPT-5 mini	llm	12%	0%	0%	0%
PPO Dense (2M)	rl	92%	4%	0%	0%
Qwen3.5-4B	llm	20%	4%	0%	0%
PPO Dense (500k)	rl	36%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	12%	0%	0%	0%
Qwen3.5-4B	llm	8%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	8%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	4%	0%	0%	0%
Random Agent	other	0%	0%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	16%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	8%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	4%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	0%	–
Gemini 3.1 Flash Lite	llm	–	–	0%	–

Show all agents

Push all BOX objects onto matching TARGET positions.

TileSorting-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	100%	100%
Qwen3.5-4B (SFT-250k)	llm	100%	84%	44%	12%
Qwen3.5-4B (SFT-250k)	llm	100%	88%	56%	8%
Qwen3.5-4B (SFT-120k)	llm	84%	64%	56%	0%
Qwen3.5-4B (SFT-120k)	llm	100%	80%	48%	0%
GPT-5 mini	llm	88%	0%	12%	0%
PPO Dense (2M)	rl	100%	96%	0%	0%
Qwen3.5-4B	llm	100%	72%	0%	0%
PPO Dense (500k)	rl	92%	84%	56%	0%
Gemini 2.5 Flash Lite	llm	76%	32%	16%	0%
Qwen3.5-4B	llm	96%	4%	0%	0%
Qwen3.5-2B	llm	80%	12%	0%	0%
Qwen3.5-2B	llm	100%	4%	0%	0%
Qwen3.5-0.8B	llm	68%	12%	0%	0%
Qwen3-4B	llm	72%	20%	8%	0%
Random Agent	other	80%	8%	0%	0%
PPO Sparse (500k)	rl	72%	4%	12%	0%
Gemini 2.5 Flash Lite	llm	72%	8%	0%	0%
Qwen3.5-2B	llm	0%	12%	4%	0%
Qwen3.5-0.8B	llm	20%	12%	0%	0%
Gemini 2.5 Flash Lite	llm	64%	0%	0%	0%
Qwen3-4B	llm	20%	0%	0%	0%
Qwen3.5-2B	llm	0%	4%	0%	0%
Qwen3.5-4B	llm	16%	4%	0%	0%
Qwen3.5-4B	llm	16%	4%	0%	0%
Qwen3.5-0.8B	llm	4%	0%	0%	0%
Qwen3-4B	llm	16%	4%	0%	0%
Qwen3-4B	llm	0%	4%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	44%	–
Gemini 3.1 Flash Lite	llm	–	–	24%	–

Show all agents

Arrange tiles to goal configuration (1,2,3...N-1 in row-major order).

ToolUse-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	100%	100%
Qwen3.5-4B (SFT-250k)	llm	100%	100%	100%	100%
Qwen3.5-4B (SFT-250k)	llm	100%	100%	100%	100%
Qwen3.5-4B (SFT-120k)	llm	100%	100%	100%	100%
Qwen3.5-4B (SFT-120k)	llm	100%	100%	100%	100%
GPT-5 mini	llm	100%	100%	100%	100%
PPO Dense (2M)	rl	100%	100%	96%	0%
Qwen3.5-4B	llm	36%	52%	84%	32%
PPO Dense (500k)	rl	56%	52%	100%	0%
Gemini 2.5 Flash Lite	llm	100%	92%	64%	56%
Qwen3.5-4B	llm	52%	56%	0%	28%
Qwen3.5-2B	llm	100%	100%	96%	100%
Qwen3.5-2B	llm	80%	88%	100%	96%
Qwen3.5-0.8B	llm	56%	52%	72%	52%
Qwen3-4B	llm	0%	0%	0%	8%
Random Agent	other	16%	24%	16%	24%
PPO Sparse (500k)	rl	100%	68%	40%	4%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	100%	–
Gemini 3.1 Flash Lite	llm	–	–	96%	–

Show all agents

Collect all SCROLLs to spawn the ORB, pick up the ORB, cross the river, and reach the GOAL for full reward (1.0).

DeceptiveReward-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	100%	100%
Qwen3.5-4B (SFT-250k)	llm	100%	100%	100%	92%
Qwen3.5-4B (SFT-250k)	llm	100%	100%	100%	92%
Qwen3.5-4B (SFT-120k)	llm	100%	100%	72%	72%
Qwen3.5-4B (SFT-120k)	llm	100%	100%	100%	0%
GPT-5 mini	llm	100%	100%	100%	84%
PPO Dense (2M)	rl	100%	100%	100%	100%
Qwen3.5-4B	llm	88%	52%	20%	16%
PPO Dense (500k)	rl	100%	100%	100%	100%
Gemini 2.5 Flash Lite	llm	100%	92%	60%	24%
Qwen3.5-4B	llm	0%	0%	40%	4%
Qwen3.5-2B	llm	68%	56%	24%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	16%	8%	24%	4%
Qwen3-4B	llm	16%	0%	0%	0%
Random Agent	other	4%	0%	4%	0%
PPO Sparse (500k)	rl	100%	0%	12%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	28%	28%	0%	0%
Qwen3.5-0.8B	llm	0%	4%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	4%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	80%	–
Gemini 3.1 Flash Lite	llm	–	–	56%	–

Show all agents

Resist the coin reward gradient.

GraphColoring-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	100%	100%
Qwen3.5-4B (SFT-250k)	llm	76%	16%	0%	4%
Qwen3.5-4B (SFT-250k)	llm	60%	24%	4%	0%
Qwen3.5-4B (SFT-120k)	llm	24%	0%	0%	0%
Qwen3.5-4B (SFT-120k)	llm	20%	8%	0%	0%
GPT-5 mini	llm	0%	0%	0%	0%
PPO Dense (2M)	rl	32%	0%	0%	0%
Qwen3.5-4B	llm	16%	4%	4%	0%
PPO Dense (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	4%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Random Agent	other	0%	0%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	0%	–
Gemini 3.1 Flash Lite	llm	–	–	0%	–

Show all agents

Color all nodes so no two adjacent nodes share the same color.

LightsOut-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	88%	92%	100%
Qwen3.5-4B (SFT-250k)	llm	48%	24%	72%	40%
Qwen3.5-4B (SFT-250k)	llm	40%	32%	56%	56%
Qwen3.5-4B (SFT-120k)	llm	36%	12%	16%	4%
Qwen3.5-4B (SFT-120k)	llm	40%	4%	16%	8%
GPT-5 mini	llm	4%	0%	0%	0%
PPO Dense (2M)	rl	100%	12%	0%	0%
Qwen3.5-4B	llm	36%	4%	0%	0%
PPO Dense (500k)	rl	56%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	16%	0%	0%	0%
Qwen3.5-4B	llm	12%	4%	0%	0%
Qwen3.5-2B	llm	4%	0%	0%	0%
Qwen3.5-2B	llm	8%	0%	0%	0%
Qwen3.5-0.8B	llm	8%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Random Agent	other	0%	0%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	4%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	0%	–
Gemini 3.1 Flash Lite	llm	–	–	0%	–

Show all agents

Turn all lights OFF by toggling switches.

ProgramSynthesis-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	100%	100%
Qwen3.5-4B (SFT-250k)	llm	12%	4%	4%	0%
Qwen3.5-4B (SFT-250k)	llm	8%	16%	4%	0%
Qwen3.5-4B (SFT-120k)	llm	12%	0%	0%	0%
Qwen3.5-4B (SFT-120k)	llm	0%	0%	0%	0%
GPT-5 mini	llm	0%	0%	0%	0%
PPO Dense (2M)	rl	4%	0%	0%	0%
Qwen3.5-4B	llm	8%	0%	0%	0%
PPO Dense (500k)	rl	4%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Random Agent	other	0%	0%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	0%	–
Gemini 3.1 Flash Lite	llm	–	–	0%	–

Show all agents

Push all GEM objects so they form the same relative pattern as the reference SCROLLs.

RuleInduction-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	88%	96%	100%
Qwen3.5-4B (SFT-250k)	llm	12%	36%	36%	40%
Qwen3.5-4B (SFT-250k)	llm	24%	16%	48%	28%
Qwen3.5-4B (SFT-120k)	llm	28%	24%	40%	24%
Qwen3.5-4B (SFT-120k)	llm	40%	48%	44%	28%
GPT-5 mini	llm	0%	0%	0%	0%
PPO Dense (2M)	rl	0%	0%	0%	0%
Qwen3.5-4B	llm	28%	20%	0%	0%
PPO Dense (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	4%	0%	0%	0%
Qwen3.5-4B	llm	0%	4%	4%	4%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	12%	0%	0%
Qwen3.5-0.8B	llm	4%	0%	4%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Random Agent	other	0%	0%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	4%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	0%	–
Gemini 3.1 Flash Lite	llm	–	–	32%	–

Show all agents

Identify real switches via ICE pattern, INTERACT all real ones, pass barrier to GOAL.

SwitchCircuit-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	80%	80%	80%	76%
Qwen3.5-4B (SFT-250k)	llm	64%	40%	0%	0%
Qwen3.5-4B (SFT-250k)	llm	72%	48%	0%	0%
Qwen3.5-4B (SFT-120k)	llm	48%	24%	0%	0%
Qwen3.5-4B (SFT-120k)	llm	52%	24%	0%	0%
GPT-5 mini	llm	4%	0%	0%	0%
PPO Dense (2M)	rl	24%	4%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
PPO Dense (500k)	rl	4%	0%	4%	4%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Random Agent	other	0%	0%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	0%	–
Gemini 3.1 Flash Lite	llm	–	–	0%	–

Show all agents

Plan switch activation order to open all barriers blocking the path to GOAL.

SymbolMatching-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	100%	100%
Qwen3.5-4B (SFT-250k)	llm	64%	20%	8%	12%
Qwen3.5-4B (SFT-250k)	llm	68%	12%	8%	12%
Qwen3.5-4B (SFT-120k)	llm	64%	24%	4%	4%
Qwen3.5-4B (SFT-120k)	llm	60%	8%	0%	0%
GPT-5 mini	llm	0%	0%	0%	0%
PPO Dense (2M)	rl	32%	4%	0%	0%
Qwen3.5-4B	llm	48%	0%	0%	0%
PPO Dense (500k)	rl	16%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	8%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	4%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Random Agent	other	4%	0%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	0%	–
Gemini 3.1 Flash Lite	llm	–	–	0%	–

Show all agents

Pick up each symbol item and deliver it to the matching target of the same type on the right side of the grid.

TaskInterference-v0

Agent	Type	Easy	Medium	Hard	Expert
Oracle Agent	other	100%	100%	100%	100%
Qwen3.5-4B (SFT-250k)	llm	100%	100%	100%	100%
Qwen3.5-4B (SFT-250k)	llm	100%	100%	100%	100%
Qwen3.5-4B (SFT-120k)	llm	100%	100%	100%	88%
Qwen3.5-4B (SFT-120k)	llm	100%	100%	100%	92%
GPT-5 mini	llm	16%	8%	4%	0%
PPO Dense (2M)	rl	0%	0%	0%	0%
Qwen3.5-4B	llm	52%	0%	0%	0%
PPO Dense (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	4%	0%	0%	0%
Qwen3.5-4B	llm	4%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Random Agent	other	0%	0%	0%	0%
PPO Sparse (500k)	rl	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Gemini 2.5 Flash Lite	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-2B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3-4B	llm	0%	0%	0%	0%
Qwen3.5-0.8B	llm	0%	0%	0%	0%
Claude Haiku 4.5	llm	–	–	12%	–
Gemini 3.1 Flash Lite	llm	–	–	0%	–

Show all agents

Raise both GEM and ORB meters to >= threshold simultaneously.