SAN storage failover in a critical surveillance system should be tested by deliberately simulating real fault conditions, such as a controller failure, path loss or power loss, while live cameras are recording and the VMS is in normal use. A pass means recording continues without gaps, alarms reach operators, and the system returns to full redundancy afterwards.
A failover that has never been tested under load is an assumption, not a safeguard. The sections below cover which failures to simulate, how to prove that no video is lost, how the VMS behaves, what failback and rebuild should look like, and how often to repeat the exercise across your IP CCTV camera system.
Which failure scenarios should a SAN failover test simulate?
A SAN failover test should simulate every realistic single point of failure in the storage path: a storage controller, a fabric or network link, a switch, a power supply, a drive, and the recording server connection. Each scenario should be run separately while the system is under full recording load.
Testing only one fault, usually a pulled drive, gives a false sense of security. A practical test plan covers:
- Controller failure: power down or disable one storage controller and confirm the surviving controller takes over all volumes.
- Path or link loss: disconnect a single Fibre Channel, iSCSI or SAS path and confirm multipath failover works without dropping the volume.
- Switch failure: power off one switch in the storage fabric to confirm the redundant fabric carries the full load.
- Power supply or feed loss: remove one power supply or one power feed to prove the redundant supplies and separate circuits do their job.
- Drive failure: pull a drive to trigger a hot spare rebuild, ideally during peak recording.
- Recording server failure: where failover recording servers are configured, stop the primary service or server and confirm the standby assumes recording.
Test in a defined order, starting with the least disruptive faults and moving towards the most severe. Agree a rollback plan in advance, and where possible run the first full test in a maintenance window or in a duplicate environment before touching production.
How can you confirm no video is lost during failover?
You confirm no video is lost by comparing recorded footage against a known reference before, during and after the fault. Check the VMS timeline for gaps on every test camera, verify frame continuity in playback, and review storage and VMS logs for write errors, dropped streams or reconnection events.
A reliable verification method combines several checks:
- Select test cameras across different recording servers, resolutions and frame rates, including your most demanding streams.
- Mark the fault time precisely, using synchronised clocks across storage, servers and the VMS so log entries line up.
- Review the timeline for each camera across the failover window. Any gap, however short, is a failure of the “no video lost” requirement.
- Play back footage across the event frame by frame, looking for missing or corrupt frames rather than relying on the timeline alone.
- Check logs in the storage system, operating system and VMS for I/O errors, timeouts and stream retries.
Be realistic about what “no loss” means. Some architectures cause a very brief pause while paths switch or volumes remount, and cameras with onboard buffering or edge storage may backfill the gap afterwards. Define your acceptance criteria before testing, for example zero gaps on the primary streams, and measure against that rather than judging by eye. Sizing the throughput headroom correctly beforehand matters too, and a tool such as the server selection guidance helps establish that the surviving path can carry the full load.
How does failover affect VMS recording and live viewing?
Failover can affect VMS recording and live viewing differently. Live viewing relies on the camera stream through the recording server, so it may continue uninterrupted, while recording depends on storage being writable. Poorly configured systems can show brief recording pauses, playback errors or recording servers reporting storage offline.
During the test, observe both sides of the experience at the same time. For recording, confirm the VMS detects the storage change without manual intervention, resumes writing to the same archive locations, and does not mark the database as corrupt. For live viewing, confirm operators keep a stable picture and that alarms, motion events and analytics metadata continue to be generated and stored.
Pay particular attention to these behaviours in your CCTV software:
- Whether the recording service restarts or stays running when the storage path changes.
- How long the VMS takes to recognise that storage is available again.
- Whether playback of footage recorded before, during and after the event is smooth and searchable.
- Whether client workstations and video walls reconnect automatically.
- Whether alerts are raised to administrators so the fault is not silent.
Behaviour varies between VMS platforms and versions, so always test against the exact platform, drivers and settings you will run in production rather than relying on generic assumptions.
What should happen during failback and rebuild?
During failback, services should return to the original controller or path in a controlled way with recording uninterrupted, and any rebuild should complete in the background without pushing recording below its required write performance. Failback and rebuild are often where problems first appear, so they deserve their own test.
Check the following once the failed component is restored:
- Controlled failback: confirm whether failback is automatic or manual, and that returning volumes to their preferred controller does not cause a second interruption.
- Rebuild performance: a RAID rebuild consumes disk and controller resources. Measure whether sustained write throughput stays above what your cameras need while the rebuild runs.
- Redundancy status: verify the array and paths report fully healthy before you declare the test complete. Running degraded on a single controller or path leaves you exposed to a second fault.
- Alerting: confirm notifications fire for the fault, the rebuild start and the return to normal.
- Hot spare: after a drive rebuild, replace the failed drive and restore the spare so the next failure is also covered.
Rebuild windows deserve extra caution on large arrays, because longer rebuilds extend the period of reduced protection. Hardware RAID with battery-backed cache and hot spares, as used in Secure Logiq’s enterprise HD surveillance servers, is designed to keep recording during a drive failure, but the principle applies to any design: prove it under your own load rather than assuming it.
How often should SAN failover be tested?
SAN failover should be tested at commissioning, after any significant change, and then on a regular schedule, commonly at least annually for critical systems and more often where policy or risk demands it. Retest after firmware, VMS, network or capacity changes, because these can silently alter failover behaviour.
A sensible testing cadence includes:
- Before go-live: a full acceptance test covering every scenario above.
- After changes: storage firmware, multipath drivers, VMS upgrades, switch configuration and added cameras or retention all warrant a retest of the affected paths.
- Routine reviews: scheduled tests, typically annually or per your own risk and compliance requirements, with less disruptive checks such as path and alert verification done more frequently.
- After real incidents: any genuine failure is a prompt to review whether the system behaved as designed.
Document each test with the scenario, expected result, actual result and any remediation. This record is valuable for audits, tender requirements and client assurance, and it shows whether performance is drifting over time.
Planning failover testing is easiest when it is designed into the architecture from the start. To discuss a resilient storage design for your next project, explore Secure Logiq’s system design service or contact the team with your requirements.
Frequently Asked Questions
How do I get started planning a SAN failover test if I have never done one before?
Begin by documenting your storage architecture, including controllers, paths, switches, power feeds and recording servers, then list each single point of failure as a test scenario. Define acceptance criteria (for example, zero gaps on primary streams), agree a rollback plan, and schedule the first run in a maintenance window or a duplicate environment. Make sure you have a current backup of VMS configuration and have notified stakeholders before you simulate any fault.
Can I run a failover test on a live production system without risking footage?
It is possible but carries risk, so reduce it by starting with the least disruptive faults, such as a single path disconnect, before moving to controller or power tests. Schedule tests for low-activity periods, confirm the system is fully healthy beforehand, and have engineers on hand to reverse the fault immediately. If the risk is unacceptable, replicate the setup in a staging environment and use it for the most severe scenarios.
What should I do if a failover test reveals gaps in recording?
Record the exact fault time and use the synchronised logs from the storage, operating system and VMS to identify where the delay occurred, such as multipath timeouts, volume remounting or the recording service restarting. Common remedies include tuning multipath and timeout settings, updating drivers or firmware, adjusting VMS storage-detection settings, and enabling camera edge storage to backfill short gaps. Retest after every change to confirm the fix actually works.
Do cameras with onboard (edge) storage make SAN failover testing unnecessary?
No. Edge storage is a useful safety net because many cameras can backfill short gaps once the recording server reconnects, but it is not a substitute for a resilient central storage design. Edge capacity is limited, backfill can add network load, and not all cameras or VMS platforms support it equally. Test with edge recording both enabled and disabled so you know the true behaviour of the primary storage path.
How can I check that a RAID rebuild won't starve my cameras of write performance?
Trigger a drive failure while the system is recording at peak load, then monitor sustained write throughput, disk latency and VMS storage warnings throughout the rebuild. Compare the measured throughput with your total camera bitrate, including headroom for motion peaks. If recording suffers, consider lowering the rebuild priority in the controller settings, adding drives or spindles, or using a design with greater performance headroom.
What mistakes do teams most often make when testing storage failover?
The most common mistakes are testing only a drive pull, running tests without full recording load, failing to synchronise clocks so logs cannot be correlated, and judging results by eye rather than against defined criteria. Others include forgetting to test failback and rebuild, not checking that alerts actually reach administrators, and never retesting after firmware or VMS upgrades. A written test plan with expected and actual results helps avoid all of these.
What documentation should I keep after a failover test?
Keep a record for each scenario covering the date, the firmware and VMS versions in use, the fault simulated, the expected result, the actual result, any gaps found and the remediation applied. Attach supporting evidence such as timeline screenshots, log extracts and throughput graphs. This evidence is valuable for audits, tender and compliance requirements, and for spotting performance drift between annual tests.
Related Articles
- What is the difference between RAID protection and storage replication for CCTV archives?
- What role do SAN switches play in a large CCTV system?
- What is Logiqal Video Volume and why was it designed for video surveillance?
- How does server throughput affect HD video recording quality?
- What is the difference between M series and H series surveillance servers?