When background jobs share one .NET process, they can delay one another even when the application contains no obvious lock. ThreadPool capacity, connection pools, storage locks, and global admission limits are all shared boundaries. This guide separates those layers, shows a correct SemaphoreSlim pattern, and supplies a small .NET 10 verification project for capacity, cancellation, and cleanup behavior. The objective is not maximum parallelism. It is a scheduler whose waiting points and failure conditions are observable and testable.

Why jobs wait even when there is no application lock

The absence of an application lock only means the entry point is not explicitly serialized. Jobs still wait whenever they share finite execution, connection, storage, or I/O capacity, so diagnosis must follow the complete resource path.

A scheduler often contains several distinct intervals that a single duration hides. A job may wait to be claimed, wait for an application permit, execute its handler, wait for a connection, run a storage operation, and then persist its result. Calling the entire interval “execution time” leads to the wrong remedy. Increasing global parallelism can shorten one queue while lengthening connection or storage queues, producing more contention without increasing useful completions.

Record at least four monotonic durations: queue wait, admission wait, core execution, and result commit. Add a job category and a normalized outcome to each structured event. These measurements distinguish work that has not started from work that entered the handler but is waiting below it. They also prevent a common interpretation error: asynchronous I/O may consume little CPU and no dedicated thread while it still holds a connection, an admission permit, or a place in the job lifecycle.

Resource boundaries differ in scope. The ThreadPool is process-wide. A connection pool is commonly grouped by connection settings. A storage engine can serialize access at a row, page, file, or other internal boundary. A batch coordinator may introduce another capacity limit whose lifetime differs from the individual job. Draw these boundaries before choosing a limiter. Otherwise, a single global gate merely conceals which resource requires protection.

The source note behind this article emphasizes a durable reasoning method rather than an environment-specific conclusion: ask whether apparently separate entry points eventually use the same resource. Two jobs can instantiate different handler objects and still call the same lower-level operation. Architecture review therefore needs a call-path check as well as a lock search. Public guidance should retain this general model while excluding system-specific names, values, and operational details.

Place bounded concurrency around the scarce resource

Bounded concurrency belongs as close as practical to the scarce shared resource, not automatically around the entire job. Separate gates by workload or resource when one large category could otherwise consume every global permit.

A single process-wide gate is simple, but simplicity does not imply isolation. If one category submits a large batch, it can occupy all permits and delay short, unrelated work. A stronger model combines a conservative global ceiling with resource-specific ceilings. Database-heavy jobs can use one gate, while CPU-oriented transformations use another. Both remain subject to process safety, yet a burst in one category cannot automatically exhaust the complete allocation for every category.

Acquire timing matters. If a job takes a permit and then performs parsing, validation, or other work that does not touch the scarce resource, it extends the critical region unnecessarily. Prepare safe local input first, call WaitAsync immediately before resource access, and release immediately after that access ends. Do not shrink the region so far that it breaks a required consistency boundary, however. A process-local semaphore cannot replace a storage transaction or protect callers in another process.

Capacity values must come from measurements in a test project or sandbox. Begin conservatively, observe admission latency, execution latency, connection acquisition, error categories, CPU, and ThreadPool signals, and change one limit at a time. Processor count alone is not a sound limit for I/O-bound work. An unrelated benchmark is not evidence for a different dependency graph. This article makes no throughput claim; its verifiable claim is only that a correctly used gate bounds entries to its configured permit count.

Fairness also requires explicit observation. SemaphoreSlim is a capacity primitive, not a durable scheduler and not a business-priority engine. Track queue delay by category and percentile rather than relying only on the overall average. If one category waits much longer, separate pools, weighted scheduling, or a bounded channel may be appropriate. Apply those changes only after the measurements demonstrate starvation, because additional queues add operating and shutdown complexity.

Use SemaphoreSlim with asynchronous acquisition and exact release

The safe pattern awaits WaitAsync, enters try only after acquisition succeeds, and releases exactly once in finally. If cancellation happens before acquisition, the caller owns no permit and must not call Release.

SemaphoreSlim provides lightweight in-process admission control. It does not coordinate across processes, preserve work across restarts, or supply a durable queue. The following .NET 10 pattern limits a safe group of delayed operations to three concurrent entries. The counter exists only to make the test observable; a normal handler does not need a shared active count unless metrics require one.

var gate = new SemaphoreSlim(3, 3);
var active = 0;

async Task RunAsync(int id, CancellationToken cancellationToken)
{
    await gate.WaitAsync(cancellationToken);
    try
    {
        Interlocked.Increment(ref active);
        await Task.Delay(30, cancellationToken);
        Console.WriteLine($"completed:{id}");
    }
    finally
    {
        Interlocked.Decrement(ref active);
        gate.Release();
    }
}

Putting unconditional release around the wait itself is a subtle defect. WaitAsync can throw because cancellation was requested before a permit became available. In that branch, releasing would create a permit that was never acquired and may eventually exceed the configured maximum. Conversely, omitting finally leaks a permit whenever core work fails. Later jobs then wait indefinitely or until their own cancellation policy fires, making a deterministic cleanup bug look like intermittent latency.

Task.WhenAll waits for a collection, but it does not impose a concurrency limit. Creating a very large number of started tasks and making all of them wait on one semaphore still allocates task state and queue entries. For a bounded input set, that tradeoff may be acceptable. For continuous or very large input, prefer a bounded channel or a fixed consumer count so backpressure exists before thousands of waiting tasks are created.

Keep ownership easy to audit. A helper that acquires permits in one method and releases them in a distant callback invites double release and leak paths. Put acquisition and release in one lexical scope when possible. If abstraction is necessary, use an asynchronous lease whose disposal releases once, then test cancellation during acquisition and exceptions during use. The contract should be visible to reviewers without reconstructing a distributed state machine.

Treat cancellation, timeout, and failure as different outcomes

Cancellation means the caller no longer requests continuation, timeout is a policy-defined time boundary, and failure means execution did not complete correctly. Record and handle these outcomes separately instead of reducing them to one false value.

Admission and execution can use the same cancellation signal, or they can have separate policy windows. Either design can be valid, but telemetry must name the phase. An admission timeout points toward capacity, unfairness, or excess input. An execution timeout points toward a slow dependency or unexpectedly large work item. Combining them removes the evidence needed to tune the correct boundary.

Do not infer that retry is safe merely because a timeout occurred. A caller that stops waiting cannot assume a downstream operation stopped before making a change. Before enabling retry, define idempotency, state transitions, and what constitutes an already-completed unit. Test duplicate inputs only in a disposable environment. The sample in this article cancels a local delay and changes no external state, so it is safe to reproduce; that property must not be generalized to real writes.

Failure paths must release the admission permit, preserve the original exception stack, identify the failed phase, and let an owning layer decide the outcome. Avoid catching an exception and returning apparent success just to let a batch complete. When collecting many jobs, retain a status per item and summarize completed, canceled, and failed items separately. Task.WhenAll can surface aggregate completion, but it should not erase item-level evidence.

Shutdown deserves its own contract. Stop accepting new input, signal waiting and active work, wait for a bounded drain period, and report what remains. Do not dispose a shared semaphore while active code can still release it. A bounded queue makes this sequence easier to reason about because acceptance and consumption are explicit stages. Tests should cover shutdown while jobs are waiting, while one job throws, and after all jobs complete normally.

Verification and reproduction on a fixed .NET 10 project

Verification must cover the capacity ceiling, cancellation while waiting, and permit restoration after failure. These checks ran in an isolated console project with .NET SDK 10.0.400, and every observable condition passed.

  1. Create a .NET 10 console project, start twelve delayed operations behind a SemaphoreSlim with three permits, and record peak active entries with atomic operations.
  2. Acquire all three permits, wait for a fourth with a fifty-millisecond CancellationTokenSource, and require OperationCanceledException as the passing branch.
  3. Acquire one permit, throw a deliberate local test exception, release in finally, and require CurrentCount to return to three with a zero process exit code.
dotnet --version
dotnet new console --framework net10.0 -n SchedulerVerification
dotnet run --project SchedulerVerification --configuration Release

The scheduler run returned these real observations: all twelve items completed, peak active entries equaled three, and three permits were available afterward. The cancellation branch printed PASS. After the deliberate exception, the available permit count again equaled three. The process exited successfully. These results validate capacity and cleanup semantics for the sample; they are not a benchmark and do not establish a recommended limit for any database or service.

The test deliberately avoids an assertion on completion order and total elapsed milliseconds. Thread scheduling order is not part of the stated contract, and timing thresholds make a small correctness test unnecessarily sensitive to machine load. The stable invariants are more useful: peak entry never exceeds capacity, waiting acquisition can be canceled, and every successful acquisition is matched by one release even when work throws.

A failing observation is equally explicit. Peak activity above three fails capacity control. Reaching the fourth permit instead of the cancellation branch fails cancellation behavior. A final count other than three fails ownership cleanup. A nonzero process exit fails the combined test. These pass and fail conditions make the sample suitable for a local regression check without requiring external accounts, persistent data, or irreversible operations.

The first official reference documents the asynchronous semaphore overloads and their cancellation behavior. The second documents .NET ThreadPool starvation diagnosis. The third documents cooperative cancellation for parallel operations. All three HTTPS pages returned HTTP 200 during this run. The two language editions use the same references and source SHA-256 so their technical evidence remains aligned.

Diagnose ThreadPool starvation separately from downstream waiting

Begin diagnosis with phase timings, then use official .NET diagnostics to inspect the ThreadPool. Change code only when evidence points to synchronous blocking; do not treat every latency increase as ThreadPool starvation.

Microsoft describes a common starvation pattern in which the ThreadPool thread count rises while CPU usage remains well below available capacity and latency increases. This is a diagnostic signal, not a verdict by itself. Observe counters together with workload input and dependency timing. Connection exhaustion or storage waiting can produce similar application-level symptoms while requiring a different correction.

Use a fixed sequence. First, compare queue wait with handler execution. Second, inspect resource-acquisition timing. Third, correlate CPU and ThreadPool counters. Fourth, capture a trace only when the previous signals justify deeper stack analysis. If one workload category is affected, inspect whether it performs synchronous I/O or holds a permit around unrelated work. If all categories slow together, inspect process-wide pools and global gates.

Storage waiting must be confirmed with the appropriate storage-side diagnostics in a sandbox. Application telemetry can establish when connection acquisition and command execution begin and end, but it cannot prove the internal reason for a lock wait or an inefficient plan. Keep test values disposable and generic. This article therefore reports only the in-process test that actually ran and does not invent an external dependency result.

Synchronous-over-asynchronous patterns deserve special attention. Blocking on task results occupies a worker while the continuation may also need ThreadPool service, amplifying delays under load. Prefer async calls through the complete call path. If a synchronous library boundary cannot be removed immediately, isolate it, measure it, and constrain its concurrency rather than wrapping it in unbounded task creation.

Choose a bounded queue when admission alone is insufficient

A semaphore limits entries to a region, but a bounded queue also limits pending work and creates backpressure. Choose the queue model for continuous input, explicit consumers, or controlled shutdown semantics.

With a small finite list, starting tasks and gating the scarce section is understandable and easy to test. With a stream, unbounded task creation moves the overload problem into memory and scheduling metadata. A bounded channel allows the writer to wait or reject according to a documented policy when capacity is full. Fixed consumers make maximum active handlers obvious and give shutdown a clear completion point.

Backpressure policy is part of correctness. Waiting preserves items but increases producer latency. Rejecting requires an observable response and a safe caller strategy. Dropping an item is valid only when the domain explicitly allows it and metrics make the drop visible. Do not select a policy merely because it improves a throughput graph. Document who owns an item at each stage and how cancellation changes ownership.

Priority can complicate fairness. A single first-in queue is predictable but may delay urgent work behind a large batch. Multiple bounded queues can reserve capacity, although they require a scheduling rule and starvation metrics. Start with the simplest model that satisfies the known workload, measure category wait time, and introduce weighting only when evidence demonstrates a need.

Durability is a separate decision. An in-memory queue is appropriate for disposable or reconstructable work in one process. If accepted work must survive a restart, use a durable mechanism designed for that requirement rather than expecting SemaphoreSlim or an in-memory channel to provide persistence. The capacity-control reasoning still applies to consumers, but acceptance, acknowledgment, and retry become explicit parts of the system contract.

Design checklist and conclusion

A dependable bounded-concurrency design states which resource is protected, where waiting occurs, when cancellation takes effect, how permits are restored, how errors are observed, and whether workload categories are isolated.

  • Map scheduler, ThreadPool, connection-pool, and storage boundaries before changing limits.
  • Measure queue wait, admission wait, core execution, and result commit separately.
  • Place gates around scarce access, not around unrelated preparation work.
  • Enter try/finally only after WaitAsync succeeds, and release once per acquisition.
  • Pass cancellation through admission and work while distinguishing timeout and failure.
  • Use bounded queues or fixed consumers for continuous input and real backpressure.
  • Track wait by workload category so an overall average cannot hide starvation.
  • Test cancellation, exceptions, and shutdown, not only the all-success path.
  • Treat every capacity value as a measured configuration, not a universal constant.

The central conclusion is simple: no lock does not mean no waiting, and greater parallelism does not guarantee greater useful throughput. Make resource boundaries and waiting phases observable, protect the actual bottleneck with bounded concurrency, and verify cleanup with stable invariants. That produces a scheduler contract that reviewers can understand and tests can enforce, rather than a concurrency number maintained by intuition.