Experience Admission for White-Box Agentic RL: A Position on Dataflow between Exploration and the Base Policy

Sixth and final in a six-part series on what building with AI agents did to one developer. The last piece ended under a blinking cursor in an empty title; this piece is that file: proposals marked as proposals, failure conditions written before the run.

Version: v0.19.3 · 2026-09-24 (aligns abstract with §6.4; separates r routing from update ban; clarifies ℓ timing before θ)
Genre: Technical position paper
Channel note: The author is a newcomer. arXiv (cs.LG endorsement) and the SEA Workshop (venue bar) were both blocked as channels, so this paper is published through neither, and no "submitted / forthcoming" claim is made. The claims and the falsification protocol stand alone, independent of any academic venue.

Abstract

In white-box observable-worker multi-worker RL, a layer is often missing between the optimizer and the collaboration interface: after trajectories are produced, who may materialize, at what granularity, and when θ may update. Black-box APIs are outside the strong-form claim.

This paper calls that layer experience admission control (below, the admission policy). The novelty sits in P1–P4 (a proposed admission policy) and preregistered failure conditions — not in a new score or a proven finding. Hypothesis-level claim: the bottleneck is often how experience is layered, isolated, and let into the base update path at low frequency — not more models seeing each other.

Layered verdict: fix 𝒜, vary only the data plane (A/B/P). No failure condition (1–5) fires, effective-materialization-ratio improvement ≥ 10% vs A, and return not persistently below A (condition 3 unfired) → §6.4 scale-up gate (may scale up; ≠ validated; ≠ must be a first-class module). If it still passes after scale-up → recommend building it as a first-class pipeline module; otherwise withdraw that configuration's strong form (the name may be retired).

Keywords: experience admission; proposed admission policy; white-box multi-worker RL; dataflow; falsification

1. Introduction

Discussions of multi-worker, asynchronous, long-horizon RL often conflate three question layers.

The optimization layer asks how gradients are computed and applied (e.g., group-relative methods [2], single-rollout asynchronous optimization [1]). The collaboration layer asks how multiple workers share context, files, and session state during interaction. The admission layer answers something else: after a trajectory has been generated, who may materialize it, at what granularity, and when θ may update.

The optimizer answers "when/how to update." The collaboration interface answers "how to coordinate." Neither directly defines admission semantics. Rising throughput can still expose noise and high-risk trajectories indiscriminately; shared context does not automatically yield isolation rules. Training stability is related to admission semantics but not identical to it: stability asks whether updates on a given visible set diverge; admission asks which trajectories may enter that visible set, and whether/how they are materialized first. An optimizer can learn to be more stable on a dirty visible set — it still does not answer the policy boundary of the visible set.

Scope (narrow first, then motivate). The strong form of this paper assumes workers can expose one of: hidden representations, local action distributions, or uncertainty signals. This covers self-deployed open weights and observable actor–learner setups. It does not cover mainstream closed API agents — those degrade, at most, into post-hoc text filtering, are never conflated with the strong form, and are outside the claim. LLM agents are one motivating scenario; the validation domain is the white-box observable setting.

In that setting the problem is not LLM-specific: long trajectories, asynchrony, and heterogeneous workers exist in classic deep RL too. IMPALA [5] and Hou et al. [1] live on the optimization/scheduling plane, where "delivered ⇒ learnable" can be the default. What this paper names is the admission policy after delivery. Shielding [9] shrinks action feasibility; CQL [6] constrains value estimates on a given dataset — neither answers layered materialization/update budgets on an online stream.

C1, a minimal counterexample (why "having PER on" is not enough). Suppose the buffer holds an ordinary learnable trajectory τ⁺ and a high-risk, high-TD-error / high-surprise τ‡. When PER [3] reweights by TD-error and τ‡ sits in a high-priority quantile, it can be sampled into the update batch at above-uniform rates; a rollout-time shield that blocks unsafe actions does not remove τ‡ from the training-visible set; deletion-style filtering discards its analysis value. Gap triad: P1 budget semantics absent (the trajectory is already in the pool and learnable by default — no "materialize-off-by-default" constraint) / violates P2·P4 / violates P3. The semantic gap: sampling weights ≠ admission policy.

Note: PER's sampling probability follows a power law in the priority quantile, so a high quantile does not guarantee a draw; but the reweighting mechanism in principle allows high-risk samples to reach the learner above uniform rates — this mechanistic "allowed" is exactly the semantic gap this paper names. Also: if a hazardous trajectory's TD-error is not high, PER will not raise its sampling probability — but that is no rebuttal of the admission layer: the hazard is then merely "unnoticed," not "quarantined and retained by a policy layer." P2's point is not "not learned," but "flagged, retained, analyzable."

So this paper does not replace the optimizer or the collaboration layer; it argues for an explicit data plane between them. Structure: contributions → P1–P4 and their logical interdependence (§3) → neighbors (§3.1) → interface and predicate (§4) → hypotheses (§5) → falsification protocol (§6) → objections (§7) → conclusion.

2. Contributions and non-claims

This is a technical position paper, not a methods paper.

Contribution one: restating the problem. "After trajectories are generated, who may materialize, at what granularity, and when θ may update" is promoted to an independent system layer. If the name can be fully absorbed by existing terms, the name should be retired. Even retired, P1–P4 and the §6 protocol remain independently testable as a system interface.

Contribution two: the proposed admission policy P1–P4. A minimal set of design differences relative to PER/deletion-style filtering, with a logical-interdependence argument (§3.2). The novelty is pressed onto four switchable constraints, not a fingerprint formula. No claim that empirical minimality has been proven by causal ablation.

Contribution three: a falsifiable protocol. A/B (plus a PER arm) and five preregistered failure conditions; condition 5 can fail the strong-form configuration on the current proxy family (§6).

Not claimed: theoretical guarantees; proven causal irreducibility; collaboration products as an RL baseline; optimality of placeholder parameters; strong-form coverage of black-box APIs; LoRA/cloud fine-tuning as main results; the appendix as proven method results; the §6.4 gate as validation success or "must be a first-class module."

3. The proposed admission policy: why it is not "renamed PER/filtering"

Terms (materialization vs deep read). In this paper, materialization means turning a trajectory from streaming state into an inspectable, persisted object; deep read means running costly analysis on materialized trajectories (full unrolling, replay, value estimation). In most implementations both fall under "admission cost," but they are logically separable — one can materialize then deep-read, or deep-read in streaming state and discard. For brevity, budget and metric contexts say "materialization cost" for both; when distinguished, materialization is storage-side and deep read is compute-side. D_read is the set eligible for materialization/deep-read — not the admitted-to-update set.

If the critic asks what remains once the name is removed, the answer is four design constraints. Only when enforced together as a policy is "experience admission control" worth using; any one missing in the long run retires the name. They are a proposed admission policy, not a published scientific finding.

P4 boundary. When hot is empty the sets may satisfy D_adm = D_all, but the strong form still requires P1/P3, and true containment must be restored once a hot-hazard appears. A long-run state of "hot empty + everything learnable by default" = the admission layer is not implemented.

Why P1 and P3 are not merged. They are logically separable: one can materialize without admitting; "admit without materializing" is the dangerous direction. The strong form forbids the latter as a default path: summaries may at most serve routing/promotion proposals; entering D_adm requires independent validation (§4). The names must stay separate, otherwise per-item ablation is impossible (§8).

PER can rank the warm tier; shields can flag hazards — neither automatically yields the whole admission policy. Calling TD-error reweighting "admission" is naming inflation.

3.1 Neighbors: why it is also not "renamed LLM data curation"

Preference filtering [10], RLAIF [11], RAFT-style rejection sampling [12], and alignment-pipeline filters [13] in LLM post-training optimize quality and labeling cost before the batch, usually with a default of: kept ⇒ update θ; dropped ⇒ out of training.

Neighbors are implementation components; not the same object. Multi-phase exploration (e.g., First Return, Then Explore [4]) shows staged policies like "return first, then explore" work in long-horizon settings, but its object is the exploration schedule and state coverage — it defines neither materialization budgets, nor a quarantine-with-retention track, nor an enforced proper containment of D_adm relative to D_all. Orthogonal to this admission policy, not a substitute.

3.2 Logical interdependence (counterfactuals): why the four come together

The following argues logical minimality, not causal ablation. Whether they are empirically irreducible remains §8.

Why all four together (design level): each blocks one collapse mode — "default materialization / delete-or-learn / seen-is-learned / no visible-set hard constraint." With any one missing, the remaining clauses cannot alone support the complete admission-layer claim. P1–P4 as a proposed list should be proposed and tested together; dropping P4 is not a return to pure PER, but it abandons the full visible-set-policy spec. It does not mean an intervention experiment has proven them the world's unique minimal set.

Known limitation (empirical): no causal ablation yet of which constraint drives the behavioral difference vs PER. An aggregate A/B/PER win does not substitute for per-item mechanism validation (§8). Logical interdependence does not fill that gap; it only fends off the "four random engineering intuitions" accusation.

4. Minimal interface and the admission predicate

No full algorithm or formal system. Fix the interface + minimal predicate constraints so the objects are discussable and implementable.

The dataflow:

adm is the admission mapping constrained by P1–P4; the optional review gate is an implementation component only.

• φ: cheap, streaming, local (C3 placeholder, replaceable; not counted as a contribution).
• r(x_t) ∈ {archive, warm, hot}: routes materialization / deep-read priority — not "who matters more," and not by itself the update ban. Update eligibility is decided only by the admission predicate (§4.1).
• ℓ(τ) ∈ {promising, hazard}: assigned on the quarantine / analysis track. Before any write to θ, a still-hot trajectory must already carry an assigned ℓ; otherwise HotHazard is unevaluable and the trajectory stays outside D_adm. After labeling: hot ∧ hazard never enters D_adm; hot ∧ promising may enter only if Validated and InBudget.
• Sets: D_read and D_adm are policy-separated; D_restricted excludes hot-hazard; when hot is nonempty, D_adm ⊊ D_all, and always D_adm ⊆ D_restricted.

4.1 The minimal admission predicate

where:

Validated is promotion validation independent of the routing score; InBudget encodes the materialization quota (P1) and capacity caps on the promotion channel (implementation detail; works with the P3/P4 gates, not owned by P1 alone).

Interaction with 𝒜: at any update step t, the optimizer's batch satisfies B_t ⊆ D_adm. 𝒜 need not internalize admission semantics; it only needs its input set constrained by the admission layer. If B_t ⊄ D_adm, that is a contract breach; if the breaching batch contains a hot-hazard, §6.3 condition 4 fires as well.

4.2 Minimal implementation contract

1. No hot-hazard enters a batch that updates θ.
2. Materialization happens in D_read; writes to θ go only through D_adm (P3).
3. Promotion requires validation independent of the routing score.
4. Archive is not materialized by default; warm materialization is budgeted (P1).
5. An optional gate must not enlarge the allowed sets.

Strong-form assumption: workers expose at least one observable signal; otherwise this is the black-box degradation, outside the claim. "Admit without materializing" is forbidden as a default path.

5. Testable hypotheses

These form the operational version of C2. The main protocol (§6) tests H1/H2/H4 plus P2's minimal execution integrity (condition 4: quarantine barred from updates + a nonempty quarantine must be read/used for peripheral analysis). H3 tests the "measurable peripheral value" magnitude of quarantine-with-retention; it is exploratory and does not enter §6.4.

H1 (primary): coarse routing + selective materialization lowers materialized volume relative to the passive pool while maintaining task performance.
H2 (primary): the three tiers, relative to binary filtering, retain analyzable anomalies without increasing base-policy contamination.
H3 (exploratory): quarantine-with-retention, versus immediate discard, yields measurable positive value on offline monitors/discriminators — condition 4 already forbids "quarantined but never read"; H3 further asks whether reading helps, and is not a §6.4 requirement.
H4 (primary): "low frequency" means limited update-contact frequency with the base policy and visible-set size — not learning rate / SGD step count; constrained updates are more stable than default-open ones.

5.1 H ↔ P

§6 tests the aggregate effect of the primary hypotheses; H3 and per-P ablations belong to §8.

6. Preregistered falsification protocol

6.1 Comparison

Same 𝒜, task, worker class:

6.2 Core metrics (minimal operational definitions)

6.3 Failure conditions

Window K ≥ 3 segments (fixed in the plan). Any one firing rejects the position in that setting.

1. Warm-tier collapse — warm traffic share ≥ 90% (or archive+hot combined < 10%), and effective-materialization-ratio improvement vs A < 5%. Warm traffic share = trajectories routed to warm in the window (or transitions; measurement fixed at preregistration) ÷ total generated in the same window. Measurement: traffic = routing counts; effective materialization ratio = actual cost. They need not agree. Condition 1 is a conjunction: traffic piling into warm and materialization cost not substantially reduced. High traffic with a clear cost drop does not fire.
2. No budget improvement — materialization-ratio improvement < 10% and wall-clock/FLOPs improvement < 5% (default AND; division of labor with condition 1: 1 catches "nominal tiers + no cost reduction," 2 catches "no substantive budget gain vs A overall"). Grey zone: materialization-ratio improvement ∈ [5%, 10%) (or < 5% without firing condition 1) with wall-clock ≥ 5% — conditions 1–2 both go unfired, but §6.4 is unmet (which requires ≥ 10%) — no scale-up claim, and no threshold changes post hoc. Any alternative joint criterion (weighted OR, "wall-clock compensates ratio," etc.) must be fixed explicitly at preregistration; otherwise condition 2 runs as AND.
3. In ≥ ⌈2K/3⌉ of K segments, return is significantly below A (preregistered α/MID).
4. Hard (P2 integrity): (a) any hot-hazard enters D_adm and updates θ; or (b) within the evaluation window the quarantine region (hot, or a preregistered quarantine set) is nonempty but its trajectories are never read/deep-read and never used for any preregistered peripheral analysis (monitoring, discriminators, audit logs, etc.) — "quarantine-to-forget": P2 execution incomplete, hard failure. If hot is empty, (b) does not apply.
5. Hard (proxy–gain test) — tests whether the current preregistered proxy family still carries learnable information (see §7.2), in three clauses; any clause firing fails condition 5 (all three must be run; do not report only the passing ones):
   • Correlation: one-sided test of the calibrated Spearman ρ (routing score vs materialization gain), alternative ρ > 0; preregistered α (e.g., 0.05), never changed post hoc. Nonsignificant → fires.
   • Random baseline: i.i.d. uniform archive/non-archive labels at the same archive marginal frequency; if the router's F1/accuracy on "is-archive" is within δ of that baseline (δ preregistered) → indistinguishable from random → fires.
   • Proxy-family boundary: "changing family" = an architectural change of the score function (e.g., entropy → prediction error, or fixed features → learnable projection). Tuning thresholds/normalization/windows within the same function = same configuration; the test must be rerun; renaming one's way around a rejection is not allowed. After failure, at most weak post-hoc filtering remains, never conflated with the strong form; a new family requires fresh preregistration and inherits nothing.

6.4 Scale-up gate (≠ validation success)

Relative to A: effective-materialization-ratio improvement ≥ 10%; condition 3 unfired; conditions 4–5 unfired (including 4(b): a nonempty quarantine must be read/used for peripheral analysis).

Reading: this only means no failure condition fired, licensing larger experiments. It does not mean validation success, is not a "weak success," and does not by itself imply "must become a first-class module." With small samples or high variance it must not be read as evidence of effect. H3's "measurable positive value" still awaits the scale-up stage or §8; the gate does not assert it alone.
Design recommendation (normative, not a protocol conclusion): if the scaled experiments still pass, then recommend making the admission policy a first-class module of the training pipeline, rather than the default passive pool/PER.

(Optional observation: materialization-ratio improvement ≥ 10% with return not below A's preregistered MID — still not promoted to a primary claim.)

7. Objections

7.1 "This is just renamed replay/filtering"

Pressed to the P1–P4 of §3 and the logical interdependence of §3.2. If existing terms can fully absorb it, retire the name. The interface remains testable per §2/§6. The empirical causal decomposition is still missing (§8).

7.2 "A cheap signal may not correlate with learnable value"

This is the deepest statistical fragility of the strong form. ICM/RND [7,8] show uncertainty signals have prior correlation with learning state in short-horizon settings; whether some preregistered proxy family remains correlated in long-horizon agentic settings is open — condition 5 exists precisely to test it. If the correlation cannot be established, what fails is the strong-form configuration on that proxy family — not a threshold tuned for another chance. Only an architectural proxy change (see §6.3) may be preregistered as a new family and rerun; tuning within the same function is not a new family. At most, weak post-hoc filtering survives.

7.3 "No theory, no hard enough experiments"

The main text defends: a proposed policy + logical interdependence + a falsifiable protocol. The appendix is not a third layer. If ablations/LLM main results are demanded → concede §8, no number inflation.

7.4 "White-box is too narrow for the breadth of the motivation"

The narrowing is accepted: title and abstract are limited to white-box observability. Cloud black-box APIs are real mainstream practice — and another paper's weak-form problem; this one does not stretch to cover them.

7.5 Applicability boundary (informal)

This protocol targets long-horizon RL settings with high materialization/deep-read costs and heterogeneous workers. If trajectories are short (say mean length L < 100), single-worker, with negligible materialization cost, the admission layer's extra system complexity may exceed its benefit; a passive pool or PER remains a reasonable default. Even when an experiment passes §6, a designer may still decline to ship for engineering-cost reasons — a pass only says "the hypothesis was not falsified in that setting"; it mandates no wholesale replacement. The claim of this paper: in suitable settings, treat the admission layer as an explicit design dimension, not a universal default.

8. Future work

(1) Causal ablation of P1–P4 (empirical minimality after logical interdependence). (2) White-box open-weight multi-seed A/B/PER. (3) Learnable routing aligned with condition 5. (4) H3: on top of condition 4(b)'s guarantee that "a nonempty quarantine must be used," add independent peripheral value metrics and controls, then decide whether to fold it into the scale-up gate. (5) Whether the black-box weak form retains part of the policy semantics — a separate paper, not mixed into the strong form.

Conclusion

In white-box observable multi-worker asynchronous RL there is a frequently ignored dataflow-organization problem between the optimizer and the collaboration interface. This paper names it experience admission control and proposes P1–P4 as a proposed admission policy (with logical interdependence and a falsification protocol). Whether it holds empirically is an open question.

Testable verdict (layered): fix 𝒜, vary only the data plane. Meeting the §6.4 scale-up gate → larger experiments allowed, not validation success and not, by itself, "must become a first-class module." If it still passes after scale-up, then, as a design recommendation, make the admission policy a first-class module of the training pipeline; otherwise withdraw that configuration's strong-form claim.

What is sought is not "proven correct," but a publicly testable, falsifiable question: on observable workers, should experience enter the base update path in a layered, isolated, low-frequency way?

References

1. Hou, Z., Li, Y., Tang, J., Dong, Y. Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning. arXiv:2607.07508, 2026.
2. Shao, Z., Wang, P., Zhu, Q., et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (GRPO). arXiv:2402.03300, 2024.
3. Schaul, T., Quan, J., Antonoglou, I., Silver, D. Prioritized Experience Replay. ICLR, 2016.
4. Ecoffet, A., et al. First Return, Then Explore. Nature, 2021.
5. Espeholt, L., et al. IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures. ICML, 2018.
6. Kumar, A., Zhou, A., Tucker, G., Levine, S. Conservative Q-Learning for Offline Reinforcement Learning. NeurIPS, 2020.
7. Pathak, D., Agrawal, P., Efros, A., Darrell, T. Curiosity-driven Exploration by Self-supervised Prediction. ICML, 2017.
8. Burda, Y., Edwards, H., Storkey, A., Klimov, A. Exploration by Random Network Distillation. ICLR, 2019.
9. Alshiekh, M., et al. Safe Reinforcement Learning via Shielding. AAAI, 2018.
10. Hu, Y., et al. Towards Comprehensive Preference Data Collection for Reward Modeling. arXiv:2406.16486, 2024.
11. Lee, H., et al. RLAIF vs. RLHF. arXiv:2309.00267, 2023.
12. Dong, H., et al. RAFT: Reward rAnked FineTuning. TMLR, 2023.
13. Wang, Z., et al. Reinforcement Learning for LLM Post-Training: A Survey (v1 originally titled A Comprehensive Survey of LLM Alignment Techniques). arXiv:2407.16216, 2024.

Previous in this series: The Mirror Cannot Reflect Thought · From "show me your code" to "show me your idea" · Judging Fatigue: From Verifying AI to Verifying Myself · The Boundary of the Harness · A Reviewer Nailed Me in Six Places