SQL HADR Triage

Always On availability groups · DR-exercise failure triage & T-SQL library

Runs entirely in this browser. No network calls, nothing leaves this page — works on restricted networks and opened straight from a local file or USB. Queries are read-only diagnostics unless marked otherwise; anything destructive is flagged.

Triage Path

Mass audit — everything that doesn't fail over

One read-only script, run on the new primary after failover (or on the DR replica before the exercise). Each section returns a labeled result set — empty sections are good news. Compare CONFIG / LINKED_SERVER / CREDENTIAL sections against the same run on prod.

/* =====================================================
   HADR DR AUDIT — run on the NEW PRIMARY
   Read-only. Each section returns a labeled result set.
   ===================================================== */
SET NOCOUNT ON;

/* 1 · Replica + database sync state */
SELECT 'AG_STATE' AS chk, ag.name AS ag,
       ar.replica_server_name AS replica, ars.role_desc AS role_now,
       DB_NAME(drs.database_id) AS db,
       drs.synchronization_state_desc AS sync_state,
       drs.suspend_reason_desc AS suspend_reason,
       drs.log_send_queue_size AS send_kb,
       drs.redo_queue_size AS redo_kb
FROM sys.dm_hadr_database_replica_states drs
JOIN sys.availability_replicas ar ON drs.replica_id = ar.replica_id
JOIN sys.availability_groups ag ON drs.group_id = ag.group_id
JOIN sys.dm_hadr_availability_replica_states ars
     ON drs.replica_id = ars.replica_id;

/* 2 · Orphaned users in every writable user db */
DECLARE @sql nvarchar(max) = N'';
SELECT @sql += N'USE ' + QUOTENAME(name) + N';
SELECT ''ORPHANED_USER'' AS chk, DB_NAME() AS db,
       dp.name AS db_user, dp.type_desc
FROM sys.database_principals dp
LEFT JOIN sys.server_principals sp ON dp.sid = sp.sid
WHERE sp.sid IS NULL AND dp.type IN (''S'',''U'',''G'')
  AND dp.name NOT IN
      (''guest'',''dbo'',''sys'',''INFORMATION_SCHEMA'');
'
FROM sys.databases
WHERE database_id > 4 AND state_desc = 'ONLINE'
  AND DATABASEPROPERTYEX(name,'Updateability') = 'READ_WRITE';
EXEC sys.sp_executesql @sql;

/* 3 · Log truncation blockers */
SELECT 'LOG_REUSE' AS chk, name AS db,
       log_reuse_wait_desc, recovery_model_desc
FROM sys.databases
WHERE database_id > 4
  AND log_reuse_wait_desc <> 'NOTHING';

/* 4 · Backup gaps (FULL-recovery dbs: log 1h+, full 7d+) */
SELECT 'BACKUP_GAP' AS chk, d.name AS db,
  MAX(CASE WHEN b.type='D' THEN b.backup_finish_date END) AS last_full,
  MAX(CASE WHEN b.type='L' THEN b.backup_finish_date END) AS last_log
FROM sys.databases d
LEFT JOIN msdb.dbo.backupset b ON b.database_name = d.name
WHERE d.database_id > 4 AND d.recovery_model_desc = 'FULL'
GROUP BY d.name
HAVING MAX(CASE WHEN b.type='L' THEN b.backup_finish_date END)
         < DATEADD(HOUR,-1,GETDATE())
    OR MAX(CASE WHEN b.type='D' THEN b.backup_finish_date END)
         < DATEADD(DAY,-7,GETDATE())
    OR MAX(b.backup_finish_date) IS NULL;

/* 5 · Services + disabled Agent jobs */
SELECT 'SERVICE' AS chk, servicename, status_desc, startup_type_desc
FROM sys.dm_server_services;
SELECT 'JOB_DISABLED' AS chk, name AS job
FROM msdb.dbo.sysjobs WHERE enabled = 0;

/* 6 · Backup preference — do backups run HERE now? */
SELECT 'BACKUP_PREF' AS chk, ag.name AS ag,
       ag.automated_backup_preference_desc,
       adc.database_name,
       sys.fn_hadr_backup_is_preferred_replica(adc.database_name)
         AS backs_up_here
FROM sys.availability_groups ag
JOIN sys.availability_databases_cluster adc
     ON ag.group_id = adc.group_id;

/* 7 · TDE state + certificates present on this server */
SELECT 'TDE_DB' AS chk, DB_NAME(database_id) AS db,
  CASE encryption_state WHEN 3 THEN 'ENCRYPTED'
       WHEN 2 THEN 'IN PROGRESS' WHEN 1 THEN 'UNENCRYPTED'
       ELSE CONVERT(varchar(10),encryption_state) END AS tde_state
FROM sys.dm_database_encryption_keys;
SELECT 'CERT' AS chk, name, expiry_date
FROM master.sys.certificates WHERE name NOT LIKE '##%';

/* 8 · Server-level objects apps depend on */
SELECT 'LINKED_SERVER' AS chk, name, data_source
FROM sys.servers WHERE is_linked = 1;
SELECT 'CREDENTIAL' AS chk, name, credential_identity
FROM sys.credentials;
SELECT 'AGENT_PROXY' AS chk, name FROM msdb.dbo.sysproxies;

/* 9 · Config fingerprint — diff by eye vs prod */
SELECT 'CONFIG' AS chk, name, value_in_use
FROM sys.configurations
WHERE name IN ('max server memory (MB)',
  'max degree of parallelism','cost threshold for parallelism',
  'optimize for ad hoc workloads','backup compression default',
  'remote admin connections');

Login transfer — generate CREATE LOGIN scripts

Run on the prod primary. Output = ready-to-run statements for the DR replica. SQL logins carry their original SID + password hash, so db users bind automatically — no orphans, no password resets. Review output before running (skip accounts that shouldn't exist in DR).

/* SQL logins — same SID + hashed password */
SELECT 'CREATE LOGIN ' + QUOTENAME(p.name)
 + ' WITH PASSWORD = '
 + CONVERT(varchar(512), l.password_hash, 1) + ' HASHED'
 + ', SID = ' + CONVERT(varchar(100), p.sid, 1)
 + ', DEFAULT_DATABASE = ' + QUOTENAME(p.default_database_name)
 + ', CHECK_POLICY = '
 + CASE WHEN l.is_policy_checked = 1 THEN 'ON' ELSE 'OFF' END
 + ', CHECK_EXPIRATION = '
 + CASE WHEN l.is_expiration_checked = 1 THEN 'ON' ELSE 'OFF' END
 + ';' AS stmt
FROM sys.server_principals p
JOIN sys.sql_logins l ON p.principal_id = l.principal_id
WHERE p.type = 'S' AND p.name NOT LIKE '##%' AND p.name <> 'sa'

UNION ALL
/* Windows logins / groups */
SELECT 'CREATE LOGIN ' + QUOTENAME(name)
 + ' FROM WINDOWS WITH DEFAULT_DATABASE = '
 + QUOTENAME(default_database_name) + ';'
FROM sys.server_principals
WHERE type IN ('U','G') AND name NOT LIKE 'NT %';

/* Server role memberships */
SELECT 'ALTER SERVER ROLE ' + QUOTENAME(r.name)
 + ' ADD MEMBER ' + QUOTENAME(m.name) + ';' AS stmt
FROM sys.server_role_members rm
JOIN sys.server_principals r
     ON rm.role_principal_id = r.principal_id
JOIN sys.server_principals m
     ON rm.member_principal_id = m.principal_id
WHERE m.name NOT LIKE '##%' AND m.name <> 'sa';

/* Explicit server-level grants — REVIEW before running */
SELECT 'GRANT ' + sp.permission_name
 + ' TO ' + QUOTENAME(p.name) + ';' AS stmt
FROM sys.server_permissions sp
JOIN sys.server_principals p
     ON sp.grantee_principal_id = p.principal_id
WHERE sp.state = 'G' AND sp.class = 100
  AND p.name NOT LIKE '##%'
  AND p.name NOT IN ('sa','public');

Rebuild linked servers on DR

Run on prod: first the inventory (what exists + how logins map), then the generator. Remote passwords can never be scripted out — fill those from your vault.

/* inventory — servers + login mappings */
SELECT s.name, s.data_source, s.provider, s.product,
       ll.uses_self_credential AS self_map,
       ll.remote_name AS remote_login,
       sp.name AS for_local_login
FROM sys.servers s
LEFT JOIN sys.linked_logins ll ON s.server_id = ll.server_id
LEFT JOIN sys.server_principals sp
       ON ll.local_principal_id = sp.principal_id
WHERE s.is_linked = 1;

/* generator — recreate on DR */
SELECT 'EXEC sp_addlinkedserver @server=N''' + name
 + ''', @srvproduct=N''' + ISNULL(product,'')
 + ''', @provider=N''' + provider
 + ''', @datasrc=N''' + ISNULL(data_source,'') + ''';' AS stmt
FROM sys.servers WHERE is_linked = 1;

/* then per server — map the login (password from vault): */
-- EXEC sp_addlinkedsrvlogin @rmtsrvname = N'LINKEDNAME',
--   @useself = 'FALSE', @locallogin = NULL,
--   @rmtuser = N'remote_user', @rmtpassword = '<vault>';

/* verify from DR: */
-- EXEC sp_testlinkedserver N'LINKEDNAME';

DR gotcha: even recreated perfectly, a linked server pointing at a prod-side hostname is dead if that site is down — check whether it should point at the target's own DR partner instead.

Rebuild credentials & Agent proxies on DR

/* generator — secrets can't be extracted, pull from vault */
SELECT 'CREATE CREDENTIAL ' + QUOTENAME(name)
 + ' WITH IDENTITY = N''' + credential_identity
 + ''', SECRET = N''<vault>'';' AS stmt
FROM sys.credentials;

/* proxies that ride on those credentials */
SELECT 'EXEC msdb.dbo.sp_add_proxy @proxy_name=N''' + p.name
 + ''', @credential_name=N''' + c.name
 + ''', @enabled=1;' AS stmt
FROM msdb.dbo.sysproxies p
JOIN sys.credentials c ON p.credential_id = c.credential_id;

/* which subsystems each proxy is granted (redo on DR) */
SELECT p.name AS proxy, s.subsystem
FROM msdb.dbo.sysproxysubsystem ps
JOIN msdb.dbo.sysproxies p ON ps.proxy_id = p.proxy_id
JOIN msdb.dbo.syssubsystems s
     ON ps.subsystem_id = s.subsystem_id;

Server-level permissions: the grants generator is in the login-transfer section above. Easy button when tooling is allowed: Copy-DbaLogin -Source PROD -Destination DR (dbatools) moves logins, SIDs, passwords, and roles in one shot. Agent jobs: Copy-DbaAgentJob.

Fleet sweep — PowerShell mass-runner

Runs every section above against a whole server list via Invoke-Sqlcmd — one CSV per section plus summary.csv, findings table at the end, connect failures flagged per server. Read-only. Copy the script below and save it as Invoke-HadrAudit.ps1.

# usage
.\Invoke-HadrAudit.ps1 -Servers 'DRSQL01','DRSQL02\INST1'
.\Invoke-HadrAudit.ps1 -ServerList .\servers.txt -OutDir C:\Temp\hadr-audit
.\Invoke-HadrAudit.ps1 -Servers DRSQL01 -SqlCredential (Get-Credential)

# servers.txt = one instance per line, # comments allowed

<#
.SYNOPSIS
  HADR DR fleet audit — runs the read-only "things that don't fail over" checks
  against a list of SQL Server instances and writes per-section CSVs plus a
  findings summary. Companion to the SQL HADR Triage page (Detect tab).

.DESCRIPTION
  Read-only. Requires Invoke-Sqlcmd (SqlServer module, or legacy SQLPS).
  Windows auth by default; pass -SqlCredential for SQL auth.
  Works on Windows PowerShell 5.1 and PowerShell 7+.

.EXAMPLE
  .\Invoke-HadrAudit.ps1 -Servers 'DRSQL01','DRSQL02\INST1'

.EXAMPLE
  .\Invoke-HadrAudit.ps1 -ServerList .\servers.txt -OutDir C:\Temp\hadr-audit

.EXAMPLE
  .\Invoke-HadrAudit.ps1 -Servers DRSQL01 -SqlCredential (Get-Credential)
#>
[CmdletBinding()]
param(
  [string[]]$Servers,
  [string]$ServerList,               # text file, one instance per line, # = comment
  [string]$OutDir = ".\hadr-audit",
  [pscredential]$SqlCredential,
  [int]$QueryTimeout = 60
)

$ErrorActionPreference = 'Stop'

# ---------- resolve server list ----------
if ($ServerList) {
  if (-not (Test-Path $ServerList)) { throw "Server list not found: $ServerList" }
  $Servers = @($Servers) + @(Get-Content $ServerList | Where-Object { $_ -and $_ -notmatch '^\s*#' })
}
$Servers = @($Servers | Where-Object { $_ } | ForEach-Object { $_.Trim() } | Select-Object -Unique)
if ($Servers.Count -eq 0) { throw "No servers given. Use -Servers or -ServerList." }

# ---------- ensure Invoke-Sqlcmd ----------
if (-not (Get-Command Invoke-Sqlcmd -ErrorAction SilentlyContinue)) {
  $loaded = $false
  foreach ($m in 'SqlServer','SQLPS') {
    try { Import-Module $m -DisableNameChecking -ErrorAction Stop; $loaded = $true; break } catch { }
  }
  if (-not $loaded) {
    throw "Invoke-Sqlcmd not available. Run: Install-Module SqlServer -Scope CurrentUser"
  }
}
$sqlcmdParams = (Get-Command Invoke-Sqlcmd).Parameters

# ---------- audit sections (all read-only) ----------
$Sections = [ordered]@{

AG_STATE = @'
SELECT ag.name AS ag, ar.replica_server_name AS replica,
       ars.role_desc AS role_now, DB_NAME(drs.database_id) AS db,
       drs.synchronization_state_desc AS sync_state,
       drs.suspend_reason_desc AS suspend_reason,
       drs.log_send_queue_size AS send_kb, drs.redo_queue_size AS redo_kb
FROM sys.dm_hadr_database_replica_states drs
JOIN sys.availability_replicas ar ON drs.replica_id = ar.replica_id
JOIN sys.availability_groups ag ON drs.group_id = ag.group_id
JOIN sys.dm_hadr_availability_replica_states ars ON drs.replica_id = ars.replica_id;
'@

SEEDING_MODE = @'
SELECT ag.name AS ag, ar.replica_server_name AS replica,
       ar.seeding_mode_desc, ar.availability_mode_desc, ar.failover_mode_desc
FROM sys.availability_replicas ar
JOIN sys.availability_groups ag ON ar.group_id = ag.group_id;
'@

ORPHANED_USER = @'
SET NOCOUNT ON;
DECLARE @sql nvarchar(max) = N'';
SELECT @sql += N'USE ' + QUOTENAME(name) + N';
SELECT DB_NAME() AS db, dp.name AS db_user, dp.type_desc
FROM sys.database_principals dp
LEFT JOIN sys.server_principals sp ON dp.sid = sp.sid
WHERE sp.sid IS NULL AND dp.type IN (''S'',''U'',''G'')
  AND dp.name NOT IN (''guest'',''dbo'',''sys'',''INFORMATION_SCHEMA'');
'
FROM sys.databases
WHERE database_id > 4 AND state_desc = 'ONLINE'
  AND DATABASEPROPERTYEX(name,'Updateability') = 'READ_WRITE';
EXEC sys.sp_executesql @sql;
'@

LOG_REUSE = @'
SELECT name AS db, log_reuse_wait_desc, recovery_model_desc
FROM sys.databases
WHERE database_id > 4 AND log_reuse_wait_desc <> 'NOTHING';
'@

BACKUP_GAP = @'
SELECT d.name AS db,
  MAX(CASE WHEN b.type='D' THEN b.backup_finish_date END) AS last_full,
  MAX(CASE WHEN b.type='L' THEN b.backup_finish_date END) AS last_log
FROM sys.databases d
LEFT JOIN msdb.dbo.backupset b ON b.database_name = d.name
WHERE d.database_id > 4 AND d.recovery_model_desc = 'FULL'
GROUP BY d.name
HAVING MAX(CASE WHEN b.type='L' THEN b.backup_finish_date END) < DATEADD(HOUR,-1,GETDATE())
    OR MAX(CASE WHEN b.type='D' THEN b.backup_finish_date END) < DATEADD(DAY,-7,GETDATE())
    OR MAX(b.backup_finish_date) IS NULL;
'@

SERVICE = @'
SELECT servicename, status_desc, startup_type_desc
FROM sys.dm_server_services;
'@

JOB_DISABLED = @'
SELECT name AS job FROM msdb.dbo.sysjobs WHERE enabled = 0;
'@

BACKUP_PREF = @'
SELECT ag.name AS ag, ag.automated_backup_preference_desc,
       adc.database_name,
       sys.fn_hadr_backup_is_preferred_replica(adc.database_name) AS backs_up_here
FROM sys.availability_groups ag
JOIN sys.availability_databases_cluster adc ON ag.group_id = adc.group_id;
'@

TDE_DB = @'
SELECT DB_NAME(database_id) AS db,
  CASE encryption_state WHEN 3 THEN 'ENCRYPTED' WHEN 2 THEN 'IN PROGRESS'
       WHEN 1 THEN 'UNENCRYPTED' ELSE CONVERT(varchar(10),encryption_state) END AS tde_state
FROM sys.dm_database_encryption_keys;
'@

CERT = @'
SELECT name, expiry_date FROM master.sys.certificates WHERE name NOT LIKE '##%';
'@

LINKED_SERVER = @'
SELECT name, data_source FROM sys.servers WHERE is_linked = 1;
'@

CREDENTIAL = @'
SELECT name, credential_identity FROM sys.credentials;
'@

AGENT_PROXY = @'
SELECT name FROM msdb.dbo.sysproxies;
'@

CONFIG = @'
SELECT name, CONVERT(bigint, value_in_use) AS value_in_use
FROM sys.configurations
WHERE name IN ('max server memory (MB)','max degree of parallelism',
  'cost threshold for parallelism','optimize for ad hoc workloads',
  'backup compression default','remote admin connections');
'@
}

# ---------- which rows count as ISSUES (vs informational) ----------
$IssueFilters = @{
  AG_STATE      = { ($_.suspend_reason -isnot [System.DBNull] -and $_.suspend_reason) -or
                    ($_.sync_state -notin @('SYNCHRONIZED','SYNCHRONIZING')) }
  ORPHANED_USER = { $true }
  LOG_REUSE     = { $true }
  BACKUP_GAP    = { $true }
  SERVICE       = { $_.status_desc -ne 'Running' -and $_.startup_type_desc -ne 'Disabled' }
  TDE_DB        = { $_.tde_state -eq 'IN PROGRESS' }
}

$noise = 'RowError','RowState','Table','ItemArray','HasErrors'   # DataRow baggage

function Invoke-Audit {
  param([string]$Server,[string]$Query)
  $p = @{ ServerInstance = $Server; Query = $Query; QueryTimeout = $QueryTimeout; ErrorAction = 'Stop' }
  if ($SqlCredential) {
    if ($sqlcmdParams.ContainsKey('Credential')) { $p.Credential = $SqlCredential }
    else {
      $p.Username = $SqlCredential.UserName
      $p.Password = $SqlCredential.GetNetworkCredential().Password
    }
  }
  if ($sqlcmdParams.ContainsKey('TrustServerCertificate')) { $p.TrustServerCertificate = $true }
  if ($sqlcmdParams.ContainsKey('Encrypt')) { $p.Encrypt = 'Optional' }
  Invoke-Sqlcmd @p
}

# ---------- run ----------
$stamp  = Get-Date -Format 'yyyyMMdd-HHmmss'
$runDir = Join-Path $OutDir $stamp
New-Item -ItemType Directory -Path $runDir -Force | Out-Null
$summary = New-Object System.Collections.Generic.List[object]

foreach ($srv in $Servers) {
  Write-Host "`n== $srv ==" -ForegroundColor Cyan
  try { Invoke-Audit -Server $srv -Query 'SELECT @@SERVERNAME AS s;' | Out-Null }
  catch {
    Write-Warning "  CONNECT FAIL: $($_.Exception.Message)"
    $summary.Add([pscustomobject]@{ Server=$srv; Section='CONNECT'; Rows=0; Issues=1; Note=$_.Exception.Message })
    continue
  }

  foreach ($name in $Sections.Keys) {
    try {
      $rows = @(Invoke-Audit -Server $srv -Query $Sections[$name])
      $issues = 0
      if ($IssueFilters.ContainsKey($name) -and $rows.Count -gt 0) {
        $issues = @($rows | Where-Object $IssueFilters[$name]).Count
      }
      if ($rows.Count -gt 0) {
        $rows |
          Select-Object -Property @{n='Server';e={$srv}},* -ExcludeProperty $noise |
          Export-Csv -Path (Join-Path $runDir "$name.csv") -NoTypeInformation -Append
      }
      $summary.Add([pscustomobject]@{ Server=$srv; Section=$name; Rows=$rows.Count; Issues=$issues; Note='' })
      $color = 'DarkGray'
      if ($issues -gt 0) { $color = 'Yellow' }
      Write-Host ("  {0,-14} rows: {1,4}  issues: {2}" -f $name, $rows.Count, $issues) -ForegroundColor $color
    }
    catch {
      # Section can fail legitimately (e.g. no AG on this instance) — record and move on
      $summary.Add([pscustomobject]@{ Server=$srv; Section=$name; Rows=0; Issues=0; Note=$_.Exception.Message })
      Write-Host ("  {0,-14} skipped: {1}" -f $name, $_.Exception.Message.Split("`n")[0]) -ForegroundColor DarkYellow
    }
  }
}

# ---------- summary ----------
$summary | Export-Csv -Path (Join-Path $runDir 'summary.csv') -NoTypeInformation
$flagged = @($summary | Where-Object { $_.Issues -gt 0 })

Write-Host "`n================ FINDINGS ================" -ForegroundColor Cyan
if ($flagged.Count -eq 0) {
  Write-Host "No issues flagged across $($Servers.Count) server(s). CSVs in $runDir" -ForegroundColor Green
} else {
  $flagged | Format-Table Server, Section, Rows, Issues, Note -AutoSize
  Write-Host "Details per section in: $runDir" -ForegroundColor Yellow
}

Step 0 — is automatic seeding even configured?

SELECT ag.name AS ag, ar.replica_server_name,
       ar.seeding_mode_desc, ar.availability_mode_desc,
       ar.failover_mode_desc
FROM sys.availability_replicas ar
JOIN sys.availability_groups ag ON ar.group_id = ag.group_id
ORDER BY ag.name, ar.replica_server_name;

MANUAL = no auto-seeding; databases must arrive by backup/restore (below). AUTOMATIC = SQL streams the db itself — monitor with the seeding DMVs in the Queries tab.

Enable automatic seeding

-- on the PRIMARY: switch the DR replica to auto-seed
ALTER AVAILABILITY GROUP [YourAG]
  MODIFY REPLICA ON 'DRSERVER'
  WITH (SEEDING_MODE = AUTOMATIC);

-- on the SECONDARY: let the AG create databases
ALTER AVAILABILITY GROUP [YourAG] GRANT CREATE ANY DATABASE;

Needs matching data/log file paths on both replicas (or SQL 2017+ for some flexibility), and the db added to the AG on the primary. Seeding a big db over the DR link is slow — check the ETA DMV before waiting on it.

Manual seeding — backup/restore runbook

The reliable path when auto-seeding is off, blocked, or too slow. Everything restores WITH NORECOVERY until the join.

/* TDE database? Move the certificate FIRST or the
   restore fails. On PROD: */
BACKUP CERTIFICATE YourTDECert
  TO FILE = '\\share\YourTDECert.cer'
  WITH PRIVATE KEY (FILE = '\\share\YourTDECert.pvk',
       ENCRYPTION BY PASSWORD = '<strong-password>');

/* On DR (create a master key first if none): */
CREATE MASTER KEY ENCRYPTION BY PASSWORD = '<strong-password>';
CREATE CERTIFICATE YourTDECert
  FROM FILE = '\\share\YourTDECert.cer'
  WITH PRIVATE KEY (FILE = '\\share\YourTDECert.pvk',
       DECRYPTION BY PASSWORD = '<strong-password>');

/* STEP 1 — on PRIMARY: copy-only full backup
   (COPY_ONLY leaves the diff chain alone) */
BACKUP DATABASE [YourDb]
  TO DISK = '\\share\YourDb_seed.bak'
  WITH COPY_ONLY, COMPRESSION, CHECKSUM, STATS = 5;

/* STEP 2 — on DR: check logical names / paths */
RESTORE FILELISTONLY FROM DISK = '\\share\YourDb_seed.bak';

/* STEP 3 — on DR: restore, MOVE if paths differ */
RESTORE DATABASE [YourDb]
  FROM DISK = '\\share\YourDb_seed.bak'
  WITH NORECOVERY, CHECKSUM, STATS = 5,
  MOVE 'YourDb_Data' TO 'D:\Data\YourDb.mdf',
  MOVE 'YourDb_Log'  TO 'L:\Log\YourDb_log.ldf';

/* STEP 4 — on PRIMARY: fresh log backup */
BACKUP LOG [YourDb]
  TO DISK = '\\share\YourDb_seed.trn'
  WITH COMPRESSION, CHECKSUM;

/* STEP 5 — on DR: restore the log, still NORECOVERY */
RESTORE LOG [YourDb]
  FROM DISK = '\\share\YourDb_seed.trn'
  WITH NORECOVERY;

/* STEP 6 — join the AG.
   If this replica was never joined (run on DR once): */
ALTER AVAILABILITY GROUP [YourAG] JOIN;
/* then per database (on DR): */
ALTER DATABASE [YourDb]
  SET HADR AVAILABILITY GROUP = [YourAG];

/* STEP 7 — verify: AG health query (Queries tab)
   → state should go SYNCHRONIZING, queues draining */

Join error	Move
"log in this backup set begins at LSN..."	A log backup is missing from the middle — restore them in order, or take/restore one more fresh log backup
Restored copy too old to join	One more log backup on primary → restore NORECOVERY → join again
Unknown log backups splitting the chain	Old prod backup jobs still firing — find them (msdb backupset query, Queries tab) and stop the extra job
Db stuck "Restoring..."	That's correct pre-join — do NOT restore WITH RECOVERY; join the AG instead

AG health — one-glance overview

SELECT ag.name AS ag_name, ar.replica_server_name,
       ars.role_desc, DB_NAME(drs.database_id) AS db,
       drs.synchronization_state_desc,
       drs.synchronization_health_desc,
       drs.suspend_reason_desc,
       drs.log_send_queue_size, drs.redo_queue_size
FROM sys.dm_hadr_database_replica_states drs
JOIN sys.availability_replicas ar
     ON drs.replica_id = ar.replica_id
JOIN sys.availability_groups ag
     ON drs.group_id = ag.group_id
JOIN sys.dm_hadr_availability_replica_states ars
     ON drs.replica_id = ars.replica_id
ORDER BY ag.name, ar.replica_server_name, db;

Why can't the log truncate?

SELECT name, log_reuse_wait_desc,
       recovery_model_desc, state_desc
FROM sys.databases
ORDER BY name;

Orphaned users on new primary

-- run in each user database
SELECT dp.name AS db_user, dp.type_desc
FROM sys.database_principals dp
LEFT JOIN sys.server_principals sp ON dp.sid = sp.sid
WHERE sp.sid IS NULL
  AND dp.type IN ('S','U','G')
  AND dp.name NOT IN ('guest','dbo','sys','INFORMATION_SCHEMA');

-- fix (per user, once matching login exists):
-- ALTER USER [username] WITH LOGIN = [loginname];

Resume suspended data movement

-- on the replica where the db shows SUSPENDED
ALTER DATABASE [YourDb] SET HADR RESUME;

Automatic seeding progress

SELECT ag.name, ds.database_name,
       ds.current_state, ds.failure_state_desc,
       ds.start_time, ds.completion_time
FROM sys.dm_hadr_automatic_seeding ds
JOIN sys.availability_groups ag ON ds.ag_id = ag.group_id
ORDER BY ds.start_time DESC;

SELECT database_name, transfer_rate_bytes_per_second,
       transferred_size_bytes, database_size_bytes,
       start_time_utc, estimate_time_complete_utc
FROM sys.dm_hadr_physical_seeding_stats;

Listener config as SQL sees it

SELECT l.dns_name, l.port, ip.ip_address,
       ip.state_desc, ip.network_subnet_ip
FROM sys.availability_group_listeners l
JOIN sys.availability_group_listener_ip_addresses ip
     ON l.listener_id = ip.listener_id;

Cluster / DNS checks (PowerShell, on a node)

# resolve what clients actually get
nslookup your-listener-name

# cluster resources behind the listener
Get-ClusterResource |
  Where-Object ResourceType -in 'Network Name','IP Address' |
  Format-Table Name, State, OwnerGroup

# listener DNS behavior (run per Network Name resource)
Get-ClusterResource "AG_ListenerName" |
  Get-ClusterParameter RegisterAllProvidersIP, HostRecordTTL

# flush stale DNS on an app/client box
ipconfig /flushdns

Last backups per database

SELECT d.name,
  MAX(CASE WHEN b.type='D' THEN b.backup_finish_date END) AS last_full,
  MAX(CASE WHEN b.type='I' THEN b.backup_finish_date END) AS last_diff,
  MAX(CASE WHEN b.type='L' THEN b.backup_finish_date END) AS last_log
FROM sys.databases d
LEFT JOIN msdb.dbo.backupset b ON b.database_name = d.name
GROUP BY d.name ORDER BY d.name;

-- is THIS replica the preferred backup replica?
SELECT sys.fn_hadr_backup_is_preferred_replica('YourDb');

Network path / IP connectivity checks (general)

Run from the CLIENT/app box (PowerShell/cmd), working outside-in. These prove whether it's DNS, IP, port, or auth before you touch SQL.

# 1 · name resolution — what IP does the client actually get?
nslookup your-sql-name
# multi-subnet AG may return several IPs; note which is the live node

# 2 · IP + port reachability (THE key test)
Test-NetConnection your-sql-name -Port 1433
# TcpTestSucceeded : True  = network path open (login issue if it still fails)
#                    False = firewall or nothing listening on that port

# 3 · named instance? the SQL Browser uses UDP 1434
Test-NetConnection your-sql-host -Port 1434 -InformationLevel Detailed

# 4 · path / where it dies
tracert your-sql-name
# 5 · flush a stale client DNS cache after a failover
ipconfig /flushdns

-- run ON the SQL host: what port/IP is SQL actually listening on?
--   (or read the errorlog line "Server is listening on")
SELECT local_net_address, local_tcp_port, client_net_address
FROM sys.dm_exec_connections WHERE session_id = @@SPID;
EXEC xp_readerrorlog 0, 1, N'Server is listening on';

# run ON the SQL host: is the port open in the Windows firewall?
Get-NetFirewallRule -Enabled True |
  Where-Object DisplayName -match 'SQL' |
  Format-Table DisplayName, Direction, Action
# quick listening-ports view
netstat -ano | findstr :1433

Backup health & agent checks (general)

Run these when a backup alert fires (agent retransmission/crash-loop) or any time you need to prove you're protected. Comments say where to run each.

-- run on the SQL instance: anything with NO backup in the last 24h
SELECT d.name, d.recovery_model_desc,
       MAX(b.backup_finish_date) AS last_backup,
       DATEDIFF(HOUR, MAX(b.backup_finish_date), GETDATE()) AS hours_ago
FROM sys.databases d
LEFT JOIN msdb.dbo.backupset b ON b.database_name = d.name
WHERE d.database_id > 4
GROUP BY d.name, d.recovery_model_desc
HAVING MAX(b.backup_finish_date) < DATEADD(HOUR,-24,GETDATE())
    OR MAX(b.backup_finish_date) IS NULL
ORDER BY hours_ago DESC;

-- run on the SQL instance: backup jobs that FAILED recently (msdb)
SELECT j.name AS job, h.run_date, h.run_time, h.message
FROM msdb.dbo.sysjobhistory h
JOIN msdb.dbo.sysjobs j ON h.job_id = j.job_id
WHERE h.run_status = 0            -- 0 = failed
  AND (j.name LIKE '%backup%' OR j.name LIKE '%cohesity%' OR h.message LIKE '%backup%')
ORDER BY h.run_date DESC, h.run_time DESC;

-- run on the SQL instance: recent backup THROUGHPUT (slow = link/agent problem)
SELECT TOP 20 database_name, type,
       backup_finish_date,
       CAST(backup_size/1048576.0 AS decimal(10,1)) AS mb,
       DATEDIFF(SECOND, backup_start_date, backup_finish_date) AS secs,
       CAST(backup_size/1048576.0 /
            NULLIF(DATEDIFF(SECOND, backup_start_date, backup_finish_date),0)
            AS decimal(10,1)) AS mb_per_sec
FROM msdb.dbo.backupset
ORDER BY backup_finish_date DESC;

:: run on the HOST (cmd/PowerShell), NOT in SQL:
:: VSS writers Cohesity/agent-based backups depend on - any FAILED = the cause
vssadmin list writers
:: look for the SQL Server VSS Writer + others; State should be [1] Stable, no error

⚠ Forced failover (data-loss risk)

-- ONLY with incident-command approval, primary unrecoverable.
-- On the target secondary:
ALTER AVAILABILITY GROUP [YourAG]
  FORCE_FAILOVER_ALLOW_DATA_LOSS;

-- afterwards, every other replica's dbs are SUSPENDED.
-- On each, after validating:
ALTER DATABASE [YourDb] SET HADR RESUME;

Error code lookup

Type an error number (18456, 1418, 9002…) or a word from the message. Exact/contains matching here — the Ops Console has the full fuzzy search plus the wait-type board.

What does NOT fail over with the AG

The AG moves databases only. Everything server-level lives outside it and must already exist on the DR replica. The Detect tab's mass audit checks every row of this table in one run:

Object	Gotcha after failover
Logins (SQL + Windows)	SQL logins must be created with the same SID or db users orphan
Agent jobs & operators	Backup/maintenance jobs may not exist — or run on the wrong replica
Linked servers	Missing or pointing at prod-side names
Credentials / proxies	Jobs fail with auth errors
Server-level permissions & roles	App works but admin/monitoring accounts lose rights
TDE certificates / master key	Encrypted DB won't come online without the cert restored
Instance config (max memory, trace flags, tempdb)	Performance surprises on the DR side

Listener & DNS quick facts

Setting	Why it bites in DR
`RegisterAllProvidersIP = 1`	All subnet IPs registered; legacy clients without `MultiSubnetFailover=True` may try the dead IP and hang
`RegisterAllProvidersIP = 0`	Only the online IP registers — but clients wait on `HostRecordTTL` (default 1200s!) for DNS to catch up
`MultiSubnetFailover=True`	Connection string tries all IPs in parallel — the right answer for multi-subnet AGs
AD computer object (CNO/VCO)	Listener creation/online fails if the cluster object can't update DNS / create the VCO in AD

log_reuse_wait_desc decoder

Value	Meaning · move
`LOG_BACKUP`	Take a log backup. Check backup jobs actually run on the current primary / preferred replica
`AVAILABILITY_REPLICA`	A secondary is suspended/behind — log can't truncate past what it hasn't received. Fix or resume the secondary; if it's gone for good, remove it from the AG
`ACTIVE_TRANSACTION`	Long-running/open txn — find it (`DBCC OPENTRAN`) before the disk fills
`DATABASE_MIRRORING`/`REPLICATION`	Other HA features holding the log — investigate that feature

Sync vs async — failover rules

Synchronous-commit replica in SYNCHRONIZED state → planned failover, no data loss. Asynchronous (typical for DR sites) → planned failover is blocked; you either switch it to sync and wait for SYNCHRONIZED, or take a forced failover and accept possible data loss. After any forced failover, all other replicas suspend and need review + HADR RESUME.

Database state decoder — when to act

You see	Normal when…	Act when / what to do
Restoring…	seeding or mid restore chain; required before AG join	Never RESTORE WITH RECOVERY if it's joining an AG — run the JOIN. Unexpected = someone SET HADR OFF
Recovering	right after startup/failover	Stuck for long = redo backlog or errorlog issue — check DB STARTUP progress
Recovery Pending	never	Errorlog first: missing file (17204), TDE cert (33111), permissions (5120). Fix cause, SET ONLINE
Suspect	never	Recovery failed — restore from backup; EMERGENCY only for salvage with approval
Offline / Single_user	deliberate maintenance	Find who/why before flipping back to ONLINE / MULTI_USER
Synchronized	sync replica, healthy	Nothing — safe no-loss failover target
Synchronizing	always, on async replicas	On a SYNC replica: wait for Synchronized before failover; never arriving = check queues/network
Not Synchronizing	never	Movement stopped — get suspend_reason, fix, HADR RESUME (suspended secondary blocks log truncation!)
Reverting	after failover (undo phase)	Wait — interrupting makes it longer
Initializing / Seeding	auto-seeding in flight	Stalled at 0 = path/permission problem — Seed tab

Searchable versions with queries: type any state into the Ops Console search, or tap its DB states chip.

Version & edition — check first, it changes the answer

Availability-group features differ sharply by SQL Server version AND edition (Enterprise vs Standard). Run this before trusting any AG step or DMV below.

SELECT SERVERPROPERTY('ProductVersion')  AS version,      -- e.g. 15.0.x = 2019, 16.0.x = 2022
       SERVERPROPERTY('ProductLevel')    AS patch_level,  -- RTM / SPn / CUn
       SERVERPROPERTY('Edition')         AS edition,       -- Enterprise / Standard / Developer
       SERVERPROPERTY('EngineEdition')   AS engine_edition,-- 3 = Enterprise-features, 2 = Standard
       SERVERPROPERTY('IsHadrEnabled')   AS hadr_enabled;  -- 1 = Always On turned on

Feature	Needs	Watch out
Always On AGs (full)	Enterprise, 2012+	Not on Standard at all
Basic Availability Groups	Standard, 2016+	2 replicas, 1 db, NO readable secondary, NO backups on secondary
Readable secondaries / read-only routing	Enterprise, 2012+	Enterprise only; improved 2016+
Automatic seeding	2016+	2012/2014 = manual backup/restore seeding ONLY (the seeding DMVs return nothing)
3 automatic-failover replicas	2016+	2012/2014 cap at 2 sync-auto-failover
Distributed AGs	Enterprise, 2016+	—
Read-scale AG (no WSFC)	2017+	Clusterless; no automatic failover
Contained AGs (own master/msdb)	2022	Newest; logins/jobs travel WITH the AG

Queries here that use sys.dm_hadr_automatic_seeding / physical_seeding_stats assume 2016+. On 2012/2014, use the manual backup/restore runbook (Seed tab) — auto-seeding simply doesn't exist. Standard Edition: expect Basic AG limits, not the full toolset.

Post-failover smoke test

#	Check
1	All DBs on new primary: SYNCHRONIZED/SYNCHRONIZING, none SUSPENDED or NOT SYNCHRONIZING
2	Listener resolves to the new primary's IP from an app server, not just the DB node
3	App service accounts can log in — no orphaned users
4	Log backups running on the new primary; `log_reuse_wait_desc` clean
5	Agent jobs enabled/appropriate for this replica's role
6	Old primary (if reachable) rejoined as secondary and resumed

Triage log

Timestamped record of what you saw and did — feeds the DR-exercise report. Stored only in this browser.

Personal decision-support template for SQL Server Always On DR exercises. Not official policy of any institution — defer to your site's approved runbook and DBA/incident-command guidance where they differ. Anything marked with ⚠ can lose data; get approval first.

Triage Path

Edit decision tree (JSON)