Pre-training-only private-reward inference ablation¶

Scope¶

The appendix ablation removes access to player_1's reward from the focal agent's world-model buffer in two retained repeated matrix games. The focal agent still observes its own reward, as required by ordinary reinforcement learning. player_1 still receives its own reward while producing behaviour; the recording wrapper passes the environment reward stream to that player's learner but stores None for the private reward in wm/D.pkl.

The ablation uses reward_game_variant=private_heterogeneous. This keeps the focal payoff and unsafe event unchanged while assigning a distinct payoff table to player_1:

Here d_b=r^1(1,b)-r^1(0,b) and b is the focal player's action. These are genuinely non-shared rewards, not the symmetric default matrices. Default environment construction remains byte-for-byte compatible at the API level and retains the old payoffs.

Windowed inverse QRE¶

Only joint actions from the world-model pre-training trajectory are passed to fit_windowed_inverse_qre; the function never indexes either reward field. For non-overlapping windows k of size W, let q_k be the empirical probability that the focal player uses action 1 and let y_k count private player action 1. The behavioural model is

y_k ~ Binomial(W, sigmoid(theta_0 + theta_1 q_k)).

With inverse temperature fixed to one,

d_0 = theta_0
d_1 = theta_0 + theta_1.

The implementation minimizes grouped binomial negative log likelihood plus a small L2 penalty using damped Newton steps. It exports the approximate regularized-Hessian standard errors, condition number, action log loss, Brier score, range and standard deviation of q_k, and a SHA-256 digest of the joint-action data. A minimum opponent-mixture variation check fails the job rather than emitting an unidentified estimate.

Reward scale and rationality are confounded, so the fit identifies beta d, not beta and reward units separately. Payoff offsets are also unidentified. The pseudo-reward table therefore uses the minimum-norm mean-zero gauge for each focal action:

r_hat(0,b) = -d_b / 2
r_hat(1,b) =  d_b / 2.

The inferred pseudo rewards are inserted only after the fit and then used as the private targets for the world-model reward head. The resulting learned graph is consumed by opponent-model training in the normal way.

One-step rollout contract¶

The gauge above preserves the private player's one-step action preferences. It is not generally policy-invariant for a multi-step Markov best response: an action-independent offset that depends on the other player's action can alter future returns when current actions change future state and policy. Accordingly, inferred private rewards are accepted only when opponent_rollout_horizon=1. The focal agent's best-response rollout retains its environment-specific horizon because its own reward is observed. The matched known-private-reward control also uses a one-step opponent response, so the comparison isolates reward inference rather than rollout depth.

This is a deliberately small appendix test, not an equilibrium-aware inverse MARL method. Simultaneously learning IPPO behaviour can violate the stationary quantal-response approximation, and a weak or ill-conditioned fit is a valid negative result that must be reported with its diagnostics.

Export isolation and execution¶

model_variant stores every WM, OM, shield, checkpoint, and evaluation artifact below model_variants/<name> so these runs cannot overwrite the primary campaign. For each environment run matched variants such as:

bash notebooks/experiments/scripts/pipeline_env.sh chicken a30 \
  model_variant=private-reward-observed-h1 \
  reward_game_variant=private_heterogeneous \
  reward_inference_mode=observed opponent_rollout_horizon=1 \
  n_steps=100000 algorithms=omsh_credible \
  experiment_tag=reward_inference_appendix_2026_08_v1 num_runs=5

bash notebooks/experiments/scripts/pipeline_env.sh chicken a30 \
  model_variant=private-reward-inverse-qre-h1 \
  reward_game_variant=private_heterogeneous \
  reward_inference_mode=inverse_qre reward_inference_window_size=250 \
  opponent_rollout_horizon=1 n_steps=100000 algorithms=omsh_credible \
  experiment_tag=reward_inference_appendix_2026_08_v1 num_runs=5

Repeat for Bertrand. Evaluate the frozen ippo_shielded_credible_floor_summable_delta_0p05_histories checkpoints with both --model-variant and --reward-game-variant=private_heterogeneous, then generate the appendix JSON, Markdown, and LaTeX table with scripts/summarize_reward_inference_ablation.py. This Credible artifact is the summarizer's default for this already-launched matched diagnostic campaign; it does not reinstate Credible in the primary suite. Pass --artifact-stem explicitly for a different shield condition. The summarizer hashes the action-only prefix in both D.pkl files and refuses to report the comparison unless the observed and inferred conditions used exactly the same pre-training joint actions and both contain the expected five frozen-policy runs. This makes the matched control sensitive only to reward access and its downstream consequences. The paper table reports the fitted advantage RMSE, focal-action mixture range and standard deviation, regularized-Hessian condition number, and action log loss beside the frozen reward--safety result; the machine-readable and Markdown reports additionally retain action Brier.

Artifact contract¶

• wm/D.pkl: focal observed reward plus private inferred pseudo reward in the inverse-QRE condition; no true private reward is recorded.
• wm/reward_inference.json: fit, identification statement, data digest, and one-step rollout contract.
• wm/env_transition_graph.pkl: learned transitions and the reward targets appropriate to the condition.
• om/om_report.json: includes the actual one-step private-opponent rollout configuration.
• reports/<tag>/reward_inference/: matched reward/safety table and reward-fit diagnostics generated after frozen-policy evaluation.

Completed appendix campaign¶

The five-seed campaign reward_inference_appendix_2026_08_v1 completed on 2026-08-14. Its validated reports are under exports/reports/reward_inference_appendix_2026_08_v1/reward_inference/ on the cluster workspace. Both matched conditions use 100,000 pre-training actions, five fresh policy seeds, 1,000 frozen-policy episodes per seed, and evaluation horizon 200.

The inferred and observed-private-reward conditions are exactly equal on frozen reward, unsafe reach, and Wilson upper bound in every paired seed. Mean reward per step is 0.130220 in Bertrand and 1.008573 in Chicken. All 20 condition-seeds have zero unsafe episodes at horizon 200; the seed-level 95% Wilson upper bound is 0.00382676. This exact downstream equality is evidence for the narrow one-step ablation, not recovery of the unidentified payoff offsets or a general inverse-MARL result.