GitLab IIS Deployment: Build Before Switching the Site Directory

The most dangerous GitLab IIS deployment sequence is not a missing start command. It is stopping the service and deleting the current site directory before running frontend and .NET build steps that can still fail. One interrupted build can remove the last working release, while later pipelines fail on the residual “already stopped” state and hide the first causal error.

This guide defines a safer GitLab IIS deployment boundary. A release is built and verified in a separate staging directory before cutover begins. IIS lifecycle operations are state-aware and idempotent. Directory promotion retains a rollback copy, and failure handling preserves observable state rather than silently promoting an incomplete release; it does not guarantee automatic recovery from every file-system fault. The executable lab touches only a local sandbox directory and an in-memory runtime state; it does not call IIS or modify a shared site.

The source note supplies the failure context, not evidence of a completed production fix. Manifest, health, and cleanup gates are extended design recommendations. The existing GitLab Runner PowerShell fail-fast guide already covers building a candidate first and propagating errors. This article adds IIS state idempotence, interruption states between two directory moves, and bounded recovery decisions.

Why must an IIS deployment build before deleting or replacing the current release?

Deleting first converts an ordinary build failure into a service-loss event. Build the complete candidate in an unexposed staging directory, verify its release contract, and only then enter the short, controlled cutover phase that can affect the current site.

A typical web deployment includes package restoration, frontend compilation, tests, .NET publishing, configuration transformation, and static-asset generation. Any one of those operations can stop because of a tool mismatch, an application error, unavailable disk space, or a transient process failure. If the current directory is removed before these uncertain operations finish, availability depends on work that has not passed a release gate.

A backup alone does not make that sequence safe. The backup must be identified, readable, and connected to an executable recovery path. A message placed after a fail-fast command may never run. A fixed backup name can also collide with a previous interrupted deployment. The release process should know which directory is current, which is a candidate, and which is the last rollback copy at every transition.

Building first also improves incident analysis. When several pipelines fail in sequence, the newest error may be a secondary symptom left by an earlier run: the application pool is already stopped, the current directory is missing, or a backup name is occupied. Investigators should move backward to the first transition from success to failure and distinguish the original failure from residual-state failures. Fixing only the newest stop error can leave the destructive ordering unchanged.

The governing invariant is simple: before cutover, the current release remains complete and runnable. A failed restore, frontend build, test, or publish can delete its own staging directory, but it cannot stop the service or alter the current directory. This invariant is easier to review and test than a long script that relies on every command succeeding after deletion.

Which gates belong in a build-to-staging workflow?

A build-to-staging workflow should verify command exit status, required outputs, release identity, and path boundaries. The candidate becomes eligible for cutover only after every gate passes and the job records a traceable release result.

The first gate is process success. Restoration, compilation, tests, and publish commands must preserve their original nonzero exit codes. GitLab Runner generates a script and executes it through the configured shell, so YAML line structure, PowerShell parsing, and error behavior are part of the deployment program. A fragment that works in an interactive terminal is not sufficient evidence that the generated runner script behaves the same way.

The second gate is output completeness. A zero exit code from the final publish command is useful but not a complete release contract. Check the application entry point, principal assemblies, static asset directory, and a release marker. If frontend and backend outputs are produced separately, verify that both belong to the same pipeline. Avoid brittle checks based on the exact number of incidental files, because harmless tool changes can add or remove temporary output.

The third gate is traceability. A staging directory can include the pipeline identifier, while a small release manifest records a commit identifier, build time, environment class, and output summary without environment-specific values. GitLab job artifacts can carry files between jobs and preserve selected evidence. However, artifact collection conditions and retention must be configured explicitly. Do not assume every failed job automatically uploads every useful file.

The fourth gate is the file-system boundary. If the release design uses directory renaming to shorten cutover, current, staging, and backup should be on the same intended volume and under an approved parent directory. Do not extrapolate cross-volume file behavior to directory moves. This design uses same-volume directory moves only; cross-volume deployment needs a separately designed copy, verification, and cutover process. This article deliberately calls the operation directory promotion rather than claiming database-style atomicity. Its guarantees must be tested on the actual file system used by the deployment.

The fifth gate is independent deployability. A later deployment job should consume the verified candidate rather than rebuilding after the current site is stopped. Rebuilding during cutover reintroduces the original risk. If jobs run on different runners, transfer the candidate through a controlled artifact or release store and verify the manifest after download, before any IIS state changes.

Why must AppCmd stop and start operations be idempotent?

IIS lifecycle operations should query the current state before deciding whether to stop or start. “Already stopped” and “already started” are acceptable expected states, while unknown state, missing target, or failed state discovery should stop the deployment.

Microsoft documents AppCmd operations for listing, starting, and stopping IIS sites and application pools. A deployment can use those operations as a small state machine: stop only from Started, continue from Stopped, and fail closed from any unrecognized result. This makes reruns, manual pre-stops, and recovery from a partially completed pipeline predictable rather than fatal for the wrong reason.

Idempotence is not permission to ignore every error. Broad error suppression can hide a misspelled application-pool name, an unavailable IIS management tool, an access problem, or selection of the wrong site. Only a positively identified “already in the desired state” result should be treated as success. All other results need a visible failure category and a nonzero job outcome.

Start behavior requires the same discipline. After directory promotion, start the target only when the release contract says it should be running, then query the resulting state. If startup fails, preserve current, backup, and diagnostic evidence. Do not immediately delete the rollback directory. If a service was intentionally stopped before the deployment, the expected post-deployment state should be an explicit input rather than an assumption hidden inside the script.

PowerShell try, catch, and finally blocks can centralize recovery and reporting, but GitLab Windows shell behavior still needs validation. Complex release logic is usually safer in a version-controlled .ps1 file invoked by a short CI job. That arrangement reduces quoting differences and lets the same file run in a local sandbox, a test runner, and the controlled deployment job.

A useful lifecycle result includes both action and reason: StoppedNow, AlreadyStopped, StartedNow, AlreadyStarted, StateUnknown, or TargetMissing. Reason codes make retries and incident reviews auditable without printing full environment configuration. They also prevent the pipeline from treating every nonzero tool result as the same failure.

How should directory promotion and rollback be structured?

A bounded cutover renames current to backup, promotes the already verified staging directory to current, and restores backup when promotion fails before a valid current directory exists. Keep the backup until startup and health verification succeed.

Immediately before cutover, recheck the staging release marker and required files. This catches an incomplete artifact transfer or accidental cleanup between build and deploy jobs. Then move the old current directory to a uniquely controlled backup name and move staging into the current location. PowerShell Move-Item supplies the directory move operation, but the release script remains responsible for missing sources, existing destinations, volume boundaries, and recovery decisions.

If the second move fails while current is absent and backup still exists, a limited recovery path can move backup back to current, restore the intended runtime state, and fail the job. If current contains a partial or unexpected directory, automatic overwrite is unsafe. Preserve the directories and stop for a documented recovery procedure. Automation should handle tested interruption points, not pretend that every file-system state has a safe universal repair.

Do not remove backup before health verification. A health gate can check that the process starts, a local health endpoint responds, required static assets are available, and the exposed release marker matches the expected candidate. If the release includes data-structure changes, directory rollback is not a complete system rollback. Compatibility and recovery for those changes require a separate design; restoring website files cannot reverse every external side effect.

Cleanup is a separate phase. A failed unexposed staging directory can be removed after evidence is retained. Successful backups can be aged out by count or time. Every cleanup operation should validate the resolved parent directory and an expected naming pattern before recursive deletion. An empty variable or malformed path must not broaden the deletion scope.

Finally, make the current-release invariant observable. The acceptance target after a deployment attempt is one readable current directory. If a file-system fault prevents that target, fail closed and retain evidence for manual recovery; it is not an unconditional guarantee. A successful cutover should additionally have a verified release marker and the expected runtime state. A failed pre-cutover build should still expose the old marker. These are concrete assertions that can be automated without requiring a full browser test.

Verification and reproduction: Windows PowerShell 5.1 directory-swap sandbox

This sandbox uses Windows PowerShell 5.1 and creates a fresh deployment_swap_lab_ directory beside the script for each scenario, without deleting existing data. It tests failed build isolation, successful promotion with backup, and an already-stopped runtime state. The revised fixture was executed in the scheduler environment.

Save the following content as deployment_swap_lab.ps1 in an empty test directory, and retain its unique scenario directories for inspection.
Run powershell.exe -NoProfile -NonInteractive -ExecutionPolicy Bypass -File .\deployment_swap_lab.ps1; the script creates fresh sandbox child directories and performs no recursive deletion.
For the failed-build case, require the current marker to remain old-release, the runtime state to remain Started, and the result to be BuildFailedBeforeCutover.
For successful promotion, require current to contain new-release, backup to contain old-release, and the runtime state to finish Started even when it began Stopped.
Pass only when all ten named PASS lines appear, the final line is ALL PASS, and the process exits with code zero. Any missing marker, exception, or nonzero exit fails the lab.

Set-StrictMode -Version Latest
$ErrorActionPreference = 'Stop'

$LabRoot = $null

function Reset-Lab {
    $script:LabRoot = Join-Path $PSScriptRoot ('deployment_swap_lab_' + [guid]::NewGuid().ToString('N'))
    New-Item -ItemType Directory -Path $script:LabRoot | Out-Null
}

function New-ReleaseDirectory([string]$Path, [string]$Marker) {
    New-Item -ItemType Directory -Path $Path | Out-Null
    Set-Content -LiteralPath (Join-Path $Path 'release.txt') -Value $Marker -NoNewline
}

function Stop-IfRunning([hashtable]$RuntimeState) {
    if ($RuntimeState.AppPool -eq 'Started') {
        $RuntimeState.AppPool = 'Stopped'
        return 'StoppedNow'
    }
    if ($RuntimeState.AppPool -eq 'Stopped') {
        return 'AlreadyStopped'
    }
    throw "Unexpected application-pool state: $($RuntimeState.AppPool)"
}

function Invoke-StagedCutover(
    [string]$CurrentPath,
    [string]$StagePath,
    [string]$BackupPath,
    [hashtable]$RuntimeState,
    [bool]$BuildSucceeded
) {
    if (-not $BuildSucceeded) {
        return 'BuildFailedBeforeCutover'
    }
    if (-not (Test-Path -LiteralPath (Join-Path $StagePath 'release.txt'))) {
        throw 'Staged release is incomplete.'
    }
    if (-not (Test-Path -LiteralPath (Join-Path $CurrentPath 'release.txt'))) {
        throw 'Current release is missing or incomplete.'
    }
    if (Test-Path -LiteralPath $BackupPath) {
        throw 'Backup destination already exists; preserve it and stop.'
    }

    $previousState = $RuntimeState.AppPool
    [void](Stop-IfRunning $RuntimeState)
    $oldMoved = $false
    try {
        Move-Item -LiteralPath $CurrentPath -Destination $BackupPath
        $oldMoved = $true
        Move-Item -LiteralPath $StagePath -Destination $CurrentPath
        $RuntimeState.AppPool = 'Started'
        return 'CutoverComplete'
    }
    catch {
        if ($oldMoved -and (-not (Test-Path -LiteralPath $CurrentPath)) -and (Test-Path -LiteralPath $BackupPath)) {
            Move-Item -LiteralPath $BackupPath -Destination $CurrentPath
            $RuntimeState.AppPool = $previousState
        }
        throw
    }
}

function Assert-Equal([string]$Name, $Actual, $Expected) {
    if ($Actual -ne $Expected) {
        throw "$Name failed. Expected [$Expected], actual [$Actual]."
    }
    Write-Output "PASS: $Name"
}

Reset-Lab
$current = Join-Path $LabRoot 'site-current'
$stage = Join-Path $LabRoot 'site-stage'
$backup = Join-Path $LabRoot 'site-backup'
$state = @{ AppPool = 'Started' }
New-ReleaseDirectory $current 'old-release'
New-ReleaseDirectory $stage 'incomplete-release'
$result = Invoke-StagedCutover $current $stage $backup $state $false
Assert-Equal 'failed build leaves current release intact' (Get-Content (Join-Path $current 'release.txt') -Raw) 'old-release'
Assert-Equal 'failed build does not stop the application pool' $state.AppPool 'Started'
Assert-Equal 'failed build exits before cutover' $result 'BuildFailedBeforeCutover'

Reset-Lab
$current = Join-Path $LabRoot 'site-current'
$stage = Join-Path $LabRoot 'site-stage'
$backup = Join-Path $LabRoot 'site-backup'
$state = @{ AppPool = 'Started' }
New-ReleaseDirectory $current 'old-release'
New-ReleaseDirectory $stage 'new-release'
$result = Invoke-StagedCutover $current $stage $backup $state $true
Assert-Equal 'successful cutover promotes staged release' (Get-Content (Join-Path $current 'release.txt') -Raw) 'new-release'
Assert-Equal 'successful cutover keeps rollback copy' (Get-Content (Join-Path $backup 'release.txt') -Raw) 'old-release'
Assert-Equal 'successful cutover restarts the application pool' $state.AppPool 'Started'
Assert-Equal 'successful cutover reports completion' $result 'CutoverComplete'

Reset-Lab
$current = Join-Path $LabRoot 'site-current'
$stage = Join-Path $LabRoot 'site-stage'
$backup = Join-Path $LabRoot 'site-backup'
$state = @{ AppPool = 'Stopped' }
New-ReleaseDirectory $current 'old-release'
New-ReleaseDirectory $stage 'new-release'
$result = Invoke-StagedCutover $current $stage $backup $state $true
Assert-Equal 'already-stopped state is idempotent' $result 'CutoverComplete'
Assert-Equal 'already-stopped cutover promotes staged release' (Get-Content (Join-Path $current 'release.txt') -Raw) 'new-release'
Assert-Equal 'already-stopped cutover finishes started' $state.AppPool 'Started'

Write-Output 'ALL PASS'

Test boundary: The ten assertions do not inject promotion, restoration, or startup failures, so they do not validate rollback. The fixture rejects an existing backup without deleting it. When safe restoration cannot be established, it leaves the simulated runtime stopped for manual review. This is a single-process state demonstration; real deployments also need serialization to prevent concurrent cutovers.

Expected output (also observed when rerunning the revised fixture):

PASS: failed build leaves current release intact
PASS: failed build does not stop the application pool
PASS: failed build exits before cutover
PASS: successful cutover promotes staged release
PASS: successful cutover keeps rollback copy
PASS: successful cutover restarts the application pool
PASS: successful cutover reports completion
PASS: already-stopped state is idempotent
PASS: already-stopped cutover promotes staged release
PASS: already-stopped cutover finishes started
ALL PASS

The revised fixture was actually run using powershell.exe -File. All ten assertions printed PASS, the final marker was ALL PASS, and the process exited with code zero. The result supports three narrow claims: a failed build does not alter current, a successful promotion retains a rollback copy, and an already-stopped state does not fail the state-aware cutover.

The lab does not call AppCmd and does not prove real IIS permissions, locked-file behavior, cross-volume moves, application startup time, request draining, or workload performance. Those concerns require an IIS integration sandbox and application-specific evidence. No availability percentage, deployment duration, or performance gain is inferred from this directory-state test.

Readers can extend the reproduction in a disposable local or isolated test environment by creating a synthetic IIS site and application pool, parsing appcmd list state, repeating stop and start, and forcing a startup failure before verifying rollback. These are reader-reproducible steps; not executed in the scheduler environment. Do not use a shared site as the first destructive test.

What should a release checklist and audit record contain?

A reliable GitLab IIS deployment records separate results for build, stage verification, pre-cutover state, directory promotion, startup, health verification, and cleanup. Failure at one gate must never relabel an incomplete candidate as current.

Before release, confirm that the candidate exists in an independent staging directory; every build command preserves failure status; required files and release identity pass inspection; current, stage, and backup resolve under the approved parent; lifecycle commands accept only known states; backup remains until health checks pass; and cleanup targets only expired directories that match the release naming policy.

The audit record should state which release was current before the attempt, which candidate was evaluated, whether cutover started, which directory moves completed, what IIS state was observed, and whether health verification passed. It should not include environment-specific values that are unnecessary for diagnosis. Structured reason codes are more useful than a final generic “deployment failed” message.

For consecutive failures, review the earliest failing pipeline first. Later failures may merely report that the service is already stopped or that a directory from the original failure remains. Separating causal failure from residual state prevents a team from patching symptoms while retaining a delete-first deployment design.

The operational KPI is not simply a green pipeline rate. Track pre-cutover failure count, cutover failure count, automatic recovery result, health-check failure count, stale staging cleanup, and time spent in the service-affecting phase. These measures show whether failures remain isolated before cutover and whether recovery paths behave as designed. They should be interpreted as engineering evidence, not as a guarantee of uninterrupted service.

The final rule is to create a deployable candidate before changing service state. Stopping IIS and moving current are high-impact actions, so they belong after build and verification. Staging, state-aware lifecycle control, rollback-preserving directory promotion, and an executable sandbox turn an unsafe delete-first script into a release process with explicit, reviewable boundaries.