How can a batch publisher avoid starving ready work when new items are created during the same run? In automated publishing workflows, a common failure mode occurs when a pipeline generates a new draft and immediately opens it for selection alongside existing work. If the publication engine sorts all available drafts by file path or creation timestamp without regard for its age, a newly generated draft can preempt older, fully validated work. When that new draft turns out to be invalid or a duplicate, the entire run might exit having published nothing, leaving valid older drafts waiting indefinitely. For a related implementation, see Managing Concurrent Git Commits During Automated.
The Core Mechanics of Draft Starvation
Single-item publishers usually operate under a strict execution budget, often constrained to processing a single publication item per run to control resource usage and maintain deterministic output. The vulnerability arises during the ingestion and generation phase. A typical workflow executes these steps:
- Scan the workspace for ready drafts.
- Generate a new candidate item based on recent updates.
- Sort all candidate paths using a standard alphabetical or chronological comparator.
- Select the top item to validate and publish.
If the generation step creates a file whose name or timestamp sorts ahead of an older, waiting item, the engine selects the new item. If the new item fails validation, the run terminates. The older item never gets evaluated because the sorting logic consistently favors the newer path.
Resolving Priority with Workspace Snapshots
To ensure queue fairness without introducing a complex secondary queuing service, you can capture a snapshot of the ready queue before candidate generation begins. By separating pre-existing ready work from newly generated items, the publisher can enforce a strict priority hierarchy: process older ready work first, and only evaluate newly generated items if the pre-existing queue is empty.
Here is a conceptual configuration illustrating how a task runner can separate the pre-existing snapshot from newly generated candidates:
{
"queue_policy": "snapshot_first",
"max_items_per_run": 1,
"fallback_to_generated": true,
"duplicate_check": "pre_and_post"
}
When implementing this policy in your execution script, filter the candidate list by checking against the pre-run snapshot. If items exist in the initial snapshot, restrict selection to that subset.
Implementation Example
Consider a Python-based queue manager that collects pending paths, runs a generation hook, and selects the final target. By taking a snapshot before generation, we protect older items from being pushed down the sort order.
from pathlib import Path
def select_next_item(work_dir: Path, generated_path: Path | None) -> Path | None:
# Step 1: Capture pre-existing ready items before generation
initial_ready = sorted([p for p in work_dir.glob("*.draft") if not p.name.startswith("_")])
# Step 2: If older ready work exists, prioritize it strictly
if initial_ready:
return initial_ready[0]
# Step 3: Fall back to newly generated work if the queue was empty
if generated_path and generated_path.exists():
return generated_path
return None
This pattern preserves the single-item execution budget while guaranteeing that older items cannot be indefinitely bypassed by newly introduced files.
Failure Modes and Edge Cases
Even with a snapshot-based priority rule, several operational failure modes require attention during design:
- Stale Snapshots: If the workspace undergoes concurrent modifications from external processes after the snapshot is taken, the initial list may reference paths that were deleted or modified.
- Blocked Oldest Items: If the oldest item fails validation due to persistent schema errors, it will continue to block the queue unless the runner implements a failure threshold or dead-letter mechanism.
- Duplicate Detection Timing: Running duplicate checks only at the final publication stage is inefficient. Effective pipelines perform duplicate detection before expensive generation steps and again immediately prior to publication. For a related implementation, see Schema First Gates Ai Publishing Pipelines.
Queue fairness is distinct from quality approval. Ensuring an item is processed first does not exempt it from validation gates. By snapshotting the queue state prior to generation, publishers maintain predictable progress and eliminate silent starvation.
Conclusion
Single-item publishers rely on predictable queue ordering to make steady progress through backlogs. By capturing the state of ready work before new items are generated and prioritizing pre-existing paths, you can eliminate draft starvation without adding architectural complexity. Inspect your sorting logic today to ensure older work receives the priority it deserves.
