Support on GitHub: github.com/phero20/concurrent-resource-scheduler (Give it a star if you find it useful!)
View Docs: pkg.go.dev/github.com/phero20/concurrent-resource-scheduler
What happens when thousands of concurrent requests compete for a small pool of reusable resources?
You can put a mutex around a slice and hope for the best.
Or you can design the scheduler around concurrency from the beginning.
I chose the second option.
I built Concurrent Resource Scheduler (CRS), a domain-agnostic Go library for selecting, prioritizing, routing, and maintaining reusable resources under heavy concurrent load.
The core idea is simple:
MANY CONCURRENT REQUESTS
│
▼
┌───────────────────┐
│ Resource Scheduler│
└─────────┬─────────┘
│
┌──────────────┼──────────────┐
│ │ │
▼ ▼ ▼
Priority Acquire State
Heap Strategy Management
│ │ │
└──────────────┼──────────────┘
│
▼
BEST AVAILABLE
RESOURCE
But making that work correctly under concurrency is where things get interesting.
CRS is designed for use cases such as:
- LLM/API gateways
- API key pools
- proxy rotation
- database replicas
- GPU workers
- backend pools
- worker resources
- connection pools
- rate-limited providers
- reusable compute resources
The scheduler itself does not know what a resource means.
It only knows:
"I have resources. I need to safely maintain them, prioritize them, and return an appropriate one to a concurrent caller."
Table of Contents
- The Problem
- The Naive Approach
- Why a Global Mutex Becomes a Problem
- The Core Idea Behind CRS
- Architecture at a Glance
- Sharded Priority Heaps
- Why Sharding Helps
- The O(1) Lookup Map
- Priority and Acquire Are Different Problems
- Acquire Strategies
- Round Robin
- Weighted Acquire
- Adaptive Acquire
- Affinity Routing
- Shared vs Exclusive Acquisition
- Resource Lifecycle
- Atomic State Transitions
- The Inactive Store
- Batch Operations
- Updates Without Destroying Heap Ordering
- Cooldowns
- Asynchronous Events
- Observability
- Prometheus Integration
- Concurrency Model
- Complexity
- Testing the Library
- Race Detector Validation
- Real-World Load Testing
- 10,000 Concurrent Workers
- Burst Testing
- Failure Testing
- Cooldown Stress Testing
- What the Load Tests Actually Tell Us
- A Minimal Example
- LLM Gateway Example
- Why CRS Is Domain-Agnostic
- Project Structure
- Design Principles
- Lessons Learned
- When You Should NOT Use CRS
- Future Directions
- Final Thoughts
The Problem
Let's start with a realistic scenario.
Imagine an LLM gateway with 100 API keys.
Each key may have different:
- rate-limit availability
- priority
- health
- cooldown state
- provider
- capacity
- temporary availability
Thousands of requests arrive concurrently.
A simplified system looks like:
┌──────────────┐
Request 1 ────────► │
Request 2 ────────► │
Request 3 ────────► Gateway │
Request 4 ────────► │
Request 5 ────────► │
... │ │
Request N ────────► │
└──────┬───────┘
│
▼
┌───────────────┐
│ Resource Pool │
└───────┬───────┘
│
┌──────────────┼──────────────┐
▼ ▼ ▼
API Key 1 API Key 2 API Key N
The scheduler now has to answer:
- Which resource should this request use?
- Which resource has the best priority?
- Is the resource currently active?
- Can multiple requests use it simultaneously?
- Should this resource temporarily leave the pool?
- Which shard should we search?
- Should requests stick to the same shard?
- What happens when the resource is released?
- How do we update its priority?
- How do we observe all of this without slowing down the hot path?
That is the problem CRS tries to solve.
The Naive Approach
The easiest implementation looks something like:
type Scheduler struct {
mu sync.Mutex
resources []*Resource
}
Then:
func (s *Scheduler) Acquire() *Resource {
s.mu.Lock()
defer s.mu.Unlock()
// Scan resources.
// Find the best one.
// Return it.
return best
}
At first glance, this looks perfectly reasonable.
For 10 resources and 2 goroutines, it probably is.
But imagine:
Resources: 10,000
Concurrent requests: 5,000
Now every operation fights over one lock.
GLOBAL MUTEX
│
┌─────────────┼─────────────┐
│ │ │
▼ ▼ ▼
Worker 1 Worker 2 Worker 3
│ │ │
└─────────────┼─────────────┘
│
WAITING
The scheduler becomes serialized around the lock.
Why a Global Mutex Becomes a Problem
There are several problems.
1. Lock contention
Only one goroutine can manipulate the pool at a time.
2. Linear scanning
If resources are stored in an array, finding the best resource can become:
O(N)
per acquisition.
3. Priority maintenance
If resources have priorities that change, the scheduler has to continuously maintain ordering.
4. State transitions
Resources can move between:
ACTIVE
INACTIVE
REMOVED
and those transitions must be synchronized.
5. Observability
Metrics and event callbacks should not block the scheduler.
The challenge is therefore not just:
"How do I build a priority queue?"
It is:
"How do I build a concurrent priority resource manager where priority, acquire, lifecycle, and observability coexist?"
The Core Idea Behind CRS
The central architectural decision was:
Don't put one global lock around the entire priority structure.
Instead, CRS partitions resources into independently locked shards.
Conceptually:
CRS
│
┌──────────┼──────────┐
│ │ │
▼ ▼ ▼
Shard 1 Shard 2 Shard N
│ │ │
Heap Heap Heap
│ │ │
Mutex Mutex Mutex
Each shard owns its own heap and its own lock.
This is the heart of CRS.
Architecture at a Glance
APPLICATION
│
Add / Acquire / Release / Update
│
▼
┌────────────────────┐
│ CRS Scheduler │
└─────────┬──────────┘
│
┌───────────────┼────────────────┐
│ │ │
▼ ▼ ▼
Acquire Lookup Inactive
Strategy Map Store
│ │ │
▼ ▼ │
Candidate Shard O(1) Node │
│ │
▼ │
┌─────────────────────────────────┐ │
│ ACTIVE HEAP SHARDS │ │
│ │ │
│ Heap 1 Heap 2 Heap N │ │
│ +Mutex +Mutex +Mutex │ │
└─────────────────────────────────┘ │
│ │
└──────────────┬─────────────────┘
│
▼
EVENT DISPATCHER
│
┌─────────┴──────────┐
▼ ▼
Telemetry Cooldown
│ │
▼ ▼
Prometheus Resource State
This separation is intentional.
Sharded Priority Heaps
Each shard maintains a priority heap.
For example:
Shard 1
[Priority 10]
/ \
[Priority 20] [Priority 30]
/ \
[40] [50]
Another shard:
Shard 2
[Priority 5]
/ \
[Priority 15] [Priority 25]
Every shard has its own synchronization boundary.
Shard 1 Shard 2 Shard 3
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ Mutex │ │ Mutex │ │ Mutex │
├─────────────┤ ├─────────────┤ ├─────────────┤
│ Priority │ │ Priority │ │ Priority │
│ Heap │ │ Heap │ │ Heap │
└─────────────┘ └─────────────┘ └─────────────┘
There is no global heap mutex.
Why Sharding Helps
Suppose we have 32 shards.
Instead of:
1 global lock
we have:
32 independently locked heaps
Different goroutines can operate on different shards simultaneously.
Goroutine A ─────► Shard 1 ─────► lock
Goroutine B ─────► Shard 7 ─────► lock
Goroutine C ─────► Shard 19 ────► lock
Goroutine D ─────► Shard 27 ────► lock
The locks are independent.
This doesn't magically eliminate contention.
If every request targets the same shard, that shard can still become contended.
That's why CRS also separates:
acquire strategy
from
priority ordering
This distinction is extremely important.
The O(1) Lookup Map
A heap is excellent at answering:
"What is the best resource?"
But a heap is not ideal for answering:
"Where is resource X?"
Searching a heap can require scanning.
CRS therefore maintains an additional lookup structure.
Conceptually:
ID
│
▼
┌──────────────────────┐
│ Lookup Map │
│ │
│ "backend-01" ───────► Node
│ "backend-02" ───────► Node
│ "backend-03" ───────► Node
└──────────────────────┘
The lookup map is protected independently with a read/write mutex.
This gives the scheduler an O(1)-style membership/location lookup by application-defined key.
That is particularly useful for:
Get
Update
Remove
Release
Exclude
Include
without scanning every heap.
Priority and Acquire Are Different Problems
This is one of the most important design ideas in CRS.
A resource can have:
Priority = 10
but that doesn't necessarily tell us:
Which shard should we inspect first?
These are separate decisions.
CRS therefore separates:
REQUEST
│
▼
ACQUIRE STRATEGY
│
▼
SHARD SELECTION
│
▼
PRIORITY HEAP
│
▼
BEST RESOURCE
This allows different routing strategies to be plugged into the scheduler without changing the underlying heap implementation.
Acquire Strategies
CRS provides several acquire approaches:
- Round Robin
- Weighted
- Adaptive
- Consistent Hashing for affinity routing
Each solves a different problem.
Round Robin
Round Robin is the simplest.
Request 1 → Shard 1
Request 2 → Shard 2
Request 3 → Shard 3
Request 4 → Shard 4
Request 5 → Shard 1
...
It is simple and predictable.
Use it when:
- shards are roughly equivalent
- you want even distribution
- resource capacity is similar
Weighted Acquire
Not every shard is necessarily equal.
Imagine:
GPU 1 → 24 GB VRAM
GPU 2 → 24 GB VRAM
GPU 3 → 80 GB VRAM
GPU 4 → 80 GB VRAM
You may want larger resources to receive more work.
Weighted acquire lets you express relative capacity.
Conceptually:
Shard 1: weight 1
Shard 2: weight 1
Shard 3: weight 4
Shard 4: weight 4
Traffic can then be distributed proportionally.
This is useful for:
- heterogeneous GPUs
- backend instances with different capacity
- API providers with different quotas
- worker pools with different performance characteristics
Adaptive Acquire
Round Robin doesn't know anything about current load.
Adaptive acquire attempts to account for shard activity.
Conceptually:
REQUEST
│
▼
┌──────────────────┐
│ Inspect shard │
│ load information │
└────────┬─────────┘
│
┌────────┼────────┐
▼ ▼ ▼
Shard A Shard B Shard C
busy low busy
│
▼
choose B
The scheduler uses lightweight shard-level state to favor less-contended shards without introducing another global lock.
This is useful when the resource pool is dynamic and simple round-robin distribution isn't enough.
Affinity Routing
Sometimes you don't want random distribution.
You want:
user-123 → same shard
user-456 → same shard
CRS supports affinity routing through consistent hashing.
Conceptually:
HASH RING
┌───────────────────┐
│ │
S1 │ S2 │
│ │
│ │
S4 │ S3 │
│ │
└───────────────────┘
▲
│
hash("user-123")
The same affinity identifier deterministically maps to the same shard.
This is useful for:
- sticky sessions
- tenant affinity
- cache locality
- connection locality
- stateful workers
Shared vs Exclusive Acquisition
CRS supports two major acquisition semantics.
Shared
The resource remains active.
ACTIVE
│
│ Acquire
▼
ACTIVE
Multiple callers can acquire the same resource.
This is useful when resources represent things like:
- API keys
- read replicas
- stateless endpoints
- shared provider capacity
Exclusive
The resource temporarily leaves the active pool.
ACTIVE
│
│ Acquire
▼
INACTIVE
│
│ Release
▼
ACTIVE
This is useful when a resource represents something that cannot safely be used by multiple concurrent operations.
Examples:
- GPU worker
- exclusive connection
- physical device
- single-use worker
- exclusive job executor
Resource Lifecycle
A CRS resource essentially moves through states.
┌───────────┐
│ ADD │
└─────┬─────┘
▼
┌─────────────┐
│ ACTIVE │
└──────┬──────┘
│
┌─────────┼─────────┐
│ │ │
Acquire Exclude Remove
│ │ │
▼ ▼ ▼
INACTIVE INACTIVE DELETED
│
Release
│
▼
ACTIVE
The important invariant is:
A resource should exist in exactly one state/location at a time.
Atomic State Transitions
Imagine:
Goroutine A: Acquire(resource)
Goroutine B: Remove(resource)
Goroutine C: Update(resource)
all happening at almost the same time.
Without careful synchronization, you can get:
Resource exists in heap
AND
Resource exists in inactive store
or:
Lookup map says ACTIVE
but heap doesn't contain it
Those are catastrophic consistency bugs.
CRS therefore treats state transitions as carefully synchronized operations involving:
- lookup state
- heap state
- inactive state
- shard locks
The Inactive Store
The inactive store is particularly important for Exclusive acquisition.
Suppose:
backend-01
is acquired exclusively.
It is removed from the active heap and stored as inactive.
ACTIVE HEAP
backend-01
backend-02
backend-03
After acquisition:
ACTIVE HEAP
backend-02
backend-03
INACTIVE STORE
backend-01
When released:
INACTIVE STORE
│
│ Release
▼
ACTIVE HEAP
The lookup map allows CRS to locate the resource without scanning every heap.
Batch Operations
Adding resources one at a time is easy.
But imagine importing 10,000 resources.
You don't want partial state like:
Batch:
1 ✓
2 ✓
3 ✓
4 ✓
5 ✗
6 ?
7 ?
...
CRS provides BatchAdd with atomic insertion behavior.
Conceptually:
BatchAdd
│
▼
┌─────────────────┐
│ Validate batch │
└────────┬────────┘
│
▼
┌─────────────────┐
│ Prepare changes │
└────────┬────────┘
│
▼
┌─────────────────┐
│ Insert shards │
└────────┬────────┘
│
▼
COMPLETE
The goal is to avoid exposing a partially inserted batch.
Updates Without Destroying Heap Ordering
Suppose:
A = priority 10
B = priority 20
C = priority 30
Now:
C → priority 5
C should move toward the top.
Simply changing the value isn't enough.
The heap must be repaired.
Conceptually:
Before:
A(10)
/ \
B(20) C(30)
Update:
C(30) → C(5)
After:
C(5)
/ \
A(10) B(20)
This preserves the priority-queue invariant.
Cooldowns
Real resources sometimes need a cooldown period.
For example:
API key hits rate limit
│
▼
cooldown 5s
│
▼
available again
CRS includes a cooldown extension.
The architecture is event-driven:
Acquire
│
▼
Release
│
▼
Event Dispatcher
│
▼
Cooldown Manager
│
▼
Exclude resource
│
▼
wait
│
▼
Include resource
The cooldown extension is asynchronous. That means there can be a small eventual-consistency window between a release event and the cooldown observer processing it.
That is intentional.
Asynchronous Events
Observability and extensions should not unnecessarily slow the scheduler's hot path.
CRS therefore uses an event dispatcher.
Scheduler
│
│ emit event
▼
┌───────────────┐
│ Buffered │
│ Event Stream │
└───────┬───────┘
│
background worker
│
┌──────────┼──────────┐
▼ ▼ ▼
Observer 1 Observer 2 Observer 3
│ │ │
▼ ▼ ▼
Metrics Cooldown Prometheus
The scheduler doesn't need to execute arbitrary observer logic while holding heap locks.
Observability
A production scheduler should answer:
How many acquisitions happened?
How many releases?
Which resources are being used?
How many failures?
What is the current resource count?
CRS includes telemetry support using atomic counters and asynchronous events.
The goal is:
Scheduler hot path
│
▼
lightweight event
│
▼
asynchronous telemetry
rather than doing expensive monitoring work while holding scheduler locks.
Prometheus Integration
CRS provides a Prometheus exporter extension.
Conceptually:
CRS
│
▼
Telemetry
│
▼
┌─────────────────┐
│ Atomic Counters │
└────────┬────────┘
│
▼
Prometheus Collector
│
▼
/metrics endpoint
This makes it possible to expose scheduler activity to an existing monitoring stack.
Concurrency Model
The concurrency model is based on multiple independent synchronization boundaries.
Scheduler
│
┌────────────┼────────────┐
│ │ │
▼ ▼ ▼
Shard 1 Shard 2 Shard N
Mutex Mutex Mutex
│ │ │
▼ ▼ ▼
Heap Heap Heap
Separately:
Lookup Map
│
sync.RWMutex
And:
Telemetry
│
atomic counters / event channel
This is much more granular than one mutex around everything.
Complexity
The scheduler is designed around heap and lookup properties.
| Operation | Complexity | Synchronization |
|---|---|---|
| Add | O(log N) | Single shard lock |
| BatchAdd | O(log N) per insertion | Shard locks |
| Acquire | Acquire + heap operation | Shard lock |
| AcquireByAffinity | Hash lookup + heap operation | Shard lock |
| Release | O(log N) | Single shard lock |
| Update | O(log N) active / O(1) inactive | Shard lock |
| Remove | O(log N) active / O(1) inactive | Shard lock |
| Get | O(1)-style lookup | Lookup synchronization |
| Stats | O(Shards) | Short shard reads |
Where:
N = resources in a shard
Actual performance depends on workload, shard count, resource distribution, acquire policy, and contention.
Testing the Library
A concurrency library cannot be validated with only:
go test ./...
Unit tests answer:
"Does this operation behave correctly?"
Load tests answer:
"What happens when thousands of goroutines continuously hammer it?"
The CRS test strategy includes:
TESTING
│
┌─────────────┼─────────────┐
│ │ │
▼ ▼ ▼
Unit Tests Race Tests Load Tests
│ │ │
▼ ▼ ▼
correctness data races behavior
Tests cover:
- scheduler construction
- configuration validation
- Add
- BatchAdd
- Get
- Len
- Stats
- Acquire
- Release
- Update
- Include
- Exclude
- Remove
- Shutdown
- acquire strategies
- event observers
- cooldown
- telemetry
- Prometheus collection
- concurrent stress behavior
Race Detector Validation
One of the most important commands for a concurrent Go library is:
go test -count=1 -race ./...
The race detector doesn't prove that a library is bug-free.
But it can catch a class of extremely dangerous concurrency problems that ordinary tests may miss.
The final release was validated with the race detector without reported data races.
Real-World Load Testing
For CRS, I created a dedicated load-test harness.
The harness models:
- concurrent workers
- backend processing delay
- random request durations
- backend failures
- request cancellation
- different scenarios
- resource utilization
- acquisition latency
- backend latency
- total latency
- throughput
- cooldown behavior
- burst traffic
The important distinction is that these are load-test results, not universal benchmarks.
10,000 Concurrent Workers
One of the larger tests used:
Concurrency: 10,000 workers
Resources: 4 backends
Duration: 60 seconds
Scenario: normal
Acquire: adaptive
Policy: shared
Race detector: enabled
The test produced approximately:
Total requests: 2,248,610
Successful: 2,226,165
Failed: 22,445
Throughput: ~37,137.28 req/s
Peak simultaneous: 10,000
Resource utilization was extremely well-balanced:
backend-01 25.01%
backend-02 24.96%
backend-03 25.03%
backend-04 25.00%
The scheduler acquisition latency was remarkably low, and should be clearly distinguished from the simulated backend latency:
Acquire Latency:
mean: 7.59µs
max: 51.893ms
Backend Latency:
p50: 260.334ms
p95: 477.661ms
p99: 496.863ms
Total Request Latency:
p50: 262.430ms
p95: 479.425ms
p99: 498.559ms
Accounting check verified complete consistency:
success + backend failures = 2,248,610
backend requests = 2,248,610
total request attempts = 2,248,610
accounting: OK
Note: The 22,445 failures correspond exactly to the configured 1% simulated backend failure rate, not scheduler acquire failures (which remained at 0). Acquire timeouts and release failures were also 0.
The test environment was:
OS: Windows
Architecture: amd64
CPU cores: 12
GOMAXPROCS: 12
Go: 1.25.5
The backend latency was simulated and should not be confused with CRS scheduler latency. These are workload-specific test results and should NOT be presented as universal benchmark claims.
Burst Testing
Real systems often receive bursts.
Traffic
│
│ ███████████
│ ███████████
│ ███████████
│
│ ███
│ ███
│ ███
└────────────────────────► time
The burst scenario generated approximately:
Requests: 468,791
Successful: 464,109
Failed: 4,682
Throughput: ~7,746 req/s
Peak simultaneous: 5,000
The success rate was approximately:
99%
Backend latency remained around:
p50 ≈ 260 ms
p95 ≈ 476 ms
p99 ≈ 495 ms
These backend numbers came from the simulated workload.
Failure Testing
A scheduler should also behave correctly when resources fail.
I tested a failure scenario with:
Concurrency: 1000
Resources: 4
Duration: 60 seconds
Failure rate: 10%
The workload produced approximately:
Requests: 231,443
Successful: 208,293
Failed: 23,150
Success rate: 90%
The scheduler reported:
Acquire failures: 0
Release failures: 0
Backend failures: 23,150
That distinction matters.
The scheduler successfully acquired resources while simulated backend operations failed at the expected rate.
Cooldown Stress Testing
Cooldown was one of the more interesting tests.
With exclusive acquisition, resources temporarily leave the active pool.
4 resources
│
▼
┌─────────────┐
│ ACTIVE │
└──────┬──────┘
│
Acquire
│
▼
┌─────────────┐
│ INACTIVE │
└──────┬──────┘
│
cooldown
│
▼
┌─────────────┐
│ ACTIVE │
└─────────────┘
At 1,000 concurrent workers, the scheduler generated a very large number of acquisition attempts while only a limited number of resources were available.
This demonstrated an important property:
High concurrency doesn't mean unlimited successful backend concurrency.
If only four resources exist and the policy is exclusive, four resources are still four resources.
What the Load Tests Actually Tell Us
The load tests gave several useful observations.
1. Sharding worked under heavy concurrency
The scheduler continued operating with thousands of concurrent workers without race-detector failures.
2. Acquire remained balanced
The adaptive workload distributed acquisitions across four backends at approximately 25% each.
3. Resource state remained consistent
The accounting checks showed that acquired resources were not silently lost.
4. Failures remained distinguishable
The harness separated:
Acquire failure
Backend failure
Release failure
Timeout
A scheduler failure and a backend failure are very different operational problems.
5. Cooldown changes the workload completely
With exclusive resources, the bottleneck becomes resource availability rather than CPU.
A Minimal Example
The basic usage pattern is:
package main
import (
"fmt"
"log"
"github.com/phero20/concurrent-resource-scheduler/config"
"github.com/phero20/concurrent-resource-scheduler/scheduler"
)
type Worker struct {
ID string
Priority int
}
func main() {
compare := func(a, b *Worker) int {
if a.Priority < b.Priority {
return -1
}
if a.Priority > b.Priority {
return 1
}
return 0
}
keyFunc := func(w *Worker) string {
return w.ID
}
cfg := config.Config[*Worker, string]{
HeapCount: 1,
Comparator: compare,
KeyFunc: keyFunc,
}
sched, err := scheduler.New(cfg)
if err != nil {
log.Fatal(err)
}
defer sched.Shutdown()
sched.Add(&Worker{
ID: "worker-1",
Priority: 50,
})
sched.Add(&Worker{
ID: "worker-2",
Priority: 10,
})
resource, err := sched.Acquire()
if err != nil {
log.Fatal(err)
}
fmt.Println("Acquired:", resource.ID)
}
The application controls the resource type, key, and comparison logic.
The scheduler handles the concurrent resource management.
LLM Gateway Example
Imagine:
type APIKey struct {
ID string
Provider string
Remaining int
}
Your application could define priority based on remaining quota.
Conceptually:
API Key
│
├── Provider
├── Remaining quota
└── Health
The comparator could make a key with more remaining capacity more desirable.
The scheduler doesn't need to know what those fields mean.
The application owns that logic.
CRS maintains the ordering and concurrent lifecycle.
This is the core idea behind being domain-agnostic.
Why CRS Is Domain-Agnostic
The scheduler doesn't contain logic like:
if resource.IsGPU() {
...
}
or:
if provider == "openai" {
...
}
Instead, the application provides:
Resource type
+
Key function
+
Comparator
+
Acquire strategy
This makes the same scheduler applicable to many domains.
Example: GPU Workers
Job
│
▼
┌─────────────┐
│ CRS │
└──────┬──────┘
│
┌─────────┼─────────┐
▼ ▼ ▼
GPU 1 GPU 2 GPU 3
24 GB 24 GB 80 GB
Weighted acquire could favor larger GPUs.
Exclusive acquisition could prevent two jobs from taking the same GPU.
Example: Database Replicas
Query
│
▼
CRS
│
┌──────────┼──────────┐
▼ ▼ ▼
DB-1 DB-2 DB-3
20% 70% 35%
The application could define priority around current load.
The scheduler remains unaware that the resources happen to be databases.
Example: Proxy Pool
HTTP Request
│
▼
CRS
│
┌──────────────┼──────────────┐
▼ ▼ ▼
Proxy A Proxy B Proxy C
healthy cooldown healthy
Cooldown can temporarily remove a failing proxy.
Affinity can keep a particular tenant or session mapped consistently.
Project Structure
concurrent-resource-scheduler/
│
├── config/
├── scheduler/
├── acquire/
├── internal/
│ ├── heap/
│ ├── lookup/
│ └── node/
├── extensions/
│ ├── cooldown/
│ ├── metrics/
│ └── prometheus/
├── events/
├── errors/
├── stats/
├── examples/
│ ├── basic/
│ ├── batch/
│ ├── affinity/
│ ├── cooldown/
│ ├── exclusive/
│ ├── shared/
│ └── prometheus/
├── docs/
│ ├── API.md
│ ├── ARCHITECTURE.md
│ ├── OVERVIEW.md
├── README.md
├── CHANGELOG.md
├── CONTRIBUTING.md
├── LICENSE
└── go.mod
The main architectural boundary is:
Public API
│
▼
Scheduler
│
├── Acquire
├── Heap
├── Lookup
├── Events
└── Extensions
Design Principles
Several principles shaped CRS.
1. No global heap lock
The heap state is partitioned.
2. Separate routing from priority
Acquire chooses where to look.
The heap chooses what is best.
3. Keep the scheduler domain-agnostic
The application owns business logic.
4. Keep observability outside the hot path
Telemetry should not become the scheduler bottleneck.
5. Make state transitions explicit
Resources are either active, inactive, or removed.
6. Optimize for concurrent workloads
Concurrency is not an afterthought.
It is part of the architecture.
Lessons Learned
Building a concurrent library taught me something important:
Concurrency problems are usually not caused by one complicated function.
They are caused by interactions between simple functions.
For example:
Acquire
+
Release
+
Update
+
Remove
+
Cooldown
+
Events
+
Concurrent callers
Each function can be correct individually while the combination is broken.
That's why concurrency testing needs to go beyond unit tests.
Another Lesson: Observability Is Part of the Design
When you have:
10,000 requests
4 resources
multiple shards
multiple goroutines
failures
cooldowns
you need to know what actually happened.
That's why CRS exposes events and telemetry around scheduler activity.
Without observability, debugging concurrent systems becomes guesswork.
Another Lesson: Load Tests Need Accounting
One of the most useful parts of the load harness was accounting.
For example:
total requests
│
├── acquire failures
│
└── reached backend
│
├── successful
└── backend failure
If:
success + backend_failure
doesn't match:
backend_requests
something is wrong.
Accounting checks are powerful for detecting concurrency bugs.
When You Should NOT Use CRS
CRS isn't intended to replace every queue or pool implementation.
You probably don't need it if you simply have:
10 objects
+
one goroutine
A simple slice may be better.
You also probably don't need CRS if your problem is purely:
"I need a FIFO queue."
Use a simpler primitive.
CRS becomes interesting when you need combinations of:
concurrency
+
priority
+
acquire
+
resource lifecycle
+
affinity
+
cooldowns
+
observability
Future Directions
Potential future areas include:
- deeper benchmarking
- additional acquire strategies
- richer scheduling policies
- more advanced resource health models
- better operational tooling
- additional observability integrations
- workload-specific tuning
- broader real-world validation
The goal is not to optimize for feature count.
The goal is to keep the core scheduler:
small
predictable
composable
concurrent
domain-agnostic
Installation
Install CRS with:
go get github.com/phero20/concurrent-resource-scheduler
Then explore the examples and documentation in the repository.
Final Thoughts
The interesting part of building a scheduler isn't writing:
Acquire()
The interesting part is everything around it.
You need to think about:
RESOURCE SCHEDULING
┌─────────┐
│ Priority│
└────┬────┘
│
┌─────────────────────┼─────────────────────┐
│ │ │
▼ ▼ ▼
Concurrency Acquire Lifecycle
│ │ │
▼ ▼ ▼
Sharded heaps Adaptive/Weighted Active/Inactive
│ Round Robin Release
│ │ │
└─────────────────────┼─────────────────────┘
│
▼
Observability
│
┌──────────┴──────────┐
▼ ▼
Telemetry Prometheus
CRS was built around that complete picture.
It is not just a priority queue.
It is a concurrent resource-management layer that combines:
- sharded priority heaps
- concurrent-safe lookup
- configurable acquire
- priority ordering
- shared and exclusive acquisition
- affinity routing
- resource lifecycle management
- cooldown extensions
- asynchronous events
- telemetry
- Prometheus integration
And the project has been tested beyond basic unit tests, including:
Race detector
+
Concurrent stress tests
+
1,000 workers
+
2,000 workers
+
5,000 workers
+
10,000 workers
+
Burst workloads
+
Failure workloads
+
Cooldown workloads
The most important result isn't a single throughput number.
It's that the architecture gives me a foundation where concurrency, scheduling policy, resource state, and observability can evolve independently.
That's what I wanted to build.
If You're Building Something Similar
I'd love to hear how you approach resource scheduling.
Especially if you're working on:
- LLM gateways
- GPU schedulers
- proxy pools
- database routing
- distributed workers
- API key rotation
- connection pools
- high-concurrency Go services
What would you change about this architecture?
What workloads should I test next?
What concurrency problems have you encountered in production?
I'd genuinely like to hear from people who have operated systems like these in the real world.
Project
Support on GitHub: github.com/phero20/concurrent-resource-scheduler (Give it a star if you find it useful!)
View Docs: pkg.go.dev/github.com/phero20/concurrent-resource-scheduler
Concurrent Resource Scheduler (CRS)
A high-performance, domain-agnostic Go resource scheduler built around sharded priority heaps, concurrent-safe lookup, pluggable acquire strategies, affinity routing, lifecycle management, cooldowns, and observability.
If you find the project useful, consider giving it a star.
Found a bug? Open an issue.
Have an idea? Start a discussion.
The best validation for a concurrency library isn't another local test.
It's seeing it survive workloads you didn't design yourself.
