When a SAN storage controller fails, CCTV recording usually continues only if a second controller takes over the storage paths. In a single-controller design, or when failover is misconfigured, recording servers lose their volumes and cameras stop writing to disk. Footage already stored is normally intact on the drives, but it may be inaccessible until the controller is restored.
The outcome depends on how the SAN, network paths and video management software are designed together, not on the controller alone. The sections below explain why recording stops, what happens to existing footage, how failover works, what to do first, and how to design an IP CCTV camera system that tolerates controller loss.
Why do cameras stop recording after a controller failure?
Cameras stop recording because the recording servers lose the storage volumes they write to. The controller manages every read and write between servers and drives. If it fails with no working peer, the volumes disappear from the servers, and the CCTV software has nowhere to write incoming video streams.
The cameras themselves usually keep running. The failure sits between the recording server and the disks. Most IP cameras and CCTV software platforms respond in one of three ways:
- Recording server stalls or errors: the VMS reports archive unavailable or disk errors and stops writing.
- Cameras buffer to onboard storage: where cameras have SD cards and edge recording is enabled, footage may be held locally and back-filled later. Buffer duration is limited and varies by camera and bitrate.
- Streams are dropped: without edge recording, video during the outage is simply not captured.
Even when a second controller exists, recording can still pause briefly while paths switch over. A short gap during failover is common. A prolonged outage usually points to a design or configuration fault rather than the failure itself.
Is existing footage lost when a controller fails?
No, existing footage is not normally lost when a controller fails. Video is written to the physical drives, not held in the controller. A failed controller typically makes the data temporarily unreachable rather than destroying it, provided the drives, RAID metadata and cache have not also been damaged.
The main risk is unwritten data. Controllers use cache to speed up writes, and if that cache is not protected by battery or flash backup, recently acknowledged video may be lost or the file system may be left inconsistent. Controller faults that corrupt array metadata are rarer but more serious.
Two important points apply here. First, RAID protects against drive failure, not controller failure, and it is not a backup. Second, a controller fault occurring at the same time as a degraded or rebuilding array raises the risk of data loss. Treat any controller failure as a reason to protect the remaining data before attempting repairs.
How do dual-controller SANs handle failover for CCTV?
A dual-controller SAN handles failover by keeping two controllers connected to the same drives. If one fails, the surviving controller takes ownership of its volumes and continues serving them. In an active/passive design the standby takes over, while active/active designs spread volumes across both and move them on failure.
For this to protect CCTV recording, several elements must be in place:
- Redundant paths: each recording server needs separate connections to each controller, using multipath I/O so the operating system can reroute without losing the volume.
- Mirrored write cache: cache contents should be mirrored between controllers or protected so acknowledged writes survive a failure.
- Redundant switching and power: a single switch, host adapter or power feed can negate the benefit of two controllers.
- VMS tolerance: the CCTV software must cope with a brief pause in storage availability and resume writing cleanly.
Dual controllers also introduce trade-offs. In some active/active designs, a failure forces one controller to carry the whole workload, which can reduce throughput. With continuous high-bitrate video, the surviving controller must be able to sustain the full sequential write load, so sizing for degraded operation matters.
What should be done immediately after a controller fails?
Preserve the data first and restore recording second. Confirm the failure scope, avoid forcing rebuilds or reinitialising volumes, and check whether a surviving controller or standby path is serving storage. Then restore recording using the fastest safe route while the failed component is replaced.
- Confirm what has failed: check SAN management alerts to establish whether it is the controller, a path, a switch or a host adapter.
- Do not reinitialise or recreate volumes: this can overwrite recoverable footage. Avoid forced array rebuilds until the cause is understood.
- Check failover status: verify the second controller has taken ownership and servers still see their volumes.
- Check recording status in the VMS: identify gaps, which cameras are affected and whether any are buffering to edge storage.
- Protect critical footage: export evidence-sensitive clips from available archives before further work begins.
- Raise a support request: engage the SAN vendor or support provider with logs, alerts and the array state.
- Consider temporary recording: if storage will be down for a prolonged period, redirect cameras to a spare recording server or local disks.
Document the time of failure and the recovery steps. This helps with incident reporting and with explaining recording gaps to stakeholders.
How can recorded footage be recovered after failure?
Recovery usually means replacing the failed controller and re-presenting the intact volumes to the recording servers. If the drives and array metadata are undamaged, footage typically reappears once the new controller imports the configuration. Where the array or file system is damaged, specialist data recovery may be required.
The realistic recovery routes, from most to least straightforward, are:
- Controller replacement: a like-for-like controller reads the array configuration from the drives and restores access.
- Failover to the peer controller: in dual-controller systems, access may already be restored.
- File system or VMS archive repair: some video archives need integrity checks or re-indexing to make footage playable.
- Edge recording back-fill: cameras with onboard storage can often be queried to retrieve footage for the outage window, depending on VMS support.
- Specialist recovery: for corrupted arrays or damaged drives, external recovery services may be needed, with no guarantee of full success.
Recovery speed and success depend on warranty terms, spares availability and whether the failure was isolated or part of a wider fault. Verify these before a deployment goes live rather than during an outage.
How can CCTV storage be designed to survive controller failure?
CCTV storage survives controller failure by removing single points of failure at every layer: controllers, paths, switches, power, and recording servers. The goal is continuous recording through the failure, with a tested recovery process for what redundancy cannot cover.
Key design considerations include:
- Dual controllers with protected cache: choose an architecture where a controller loss does not interrupt volumes or risk unwritten data.
- Multipath networking: separate switches and cabling for each path, with servers configured and tested for path failover.
- Sizing for degraded mode: confirm the system still sustains full camera bitrates, including analytics metadata, when running on one controller.
- Recording server resilience: failover or redundant recording servers so that a host failure does not compound a storage failure.
- Edge storage as a safety net: enable onboard camera recording where the VMS supports back-filling gaps.
- Failure testing before handover: pull a controller or path during commissioning and confirm recording continues.
A SAN is not automatically the right choice for every deployment. For smaller systems, resilient purpose-built HD surveillance servers with hardware RAID, redundant power and hot spares can deliver strong availability with less complexity. Larger multi-site or high-retention estates may justify shared storage, as seen in the Qatar University SAN storage case study. Matching architecture to the actual risk profile, bandwidth and retention requirements is what prevents both under-engineering and unnecessary cost. For wider planning guidance, see the server selection guide.
If you are specifying resilient storage for a demanding IP CCTV camera system, Secure Logiq’s system design service can help you evaluate failover requirements and design an architecture matched to your recording and retention needs.
Frequently Asked Questions
How do I test whether my SAN failover will actually keep CCTV recording?
Run a controlled failure test during commissioning or a planned maintenance window. Fail over a controller or disconnect one storage path while cameras are recording, then check the VMS for gaps, errors and how long it took to resume writing. Repeat the test for each path, switch and power feed, and document the results so you have a baseline for future changes.
How long can a camera's onboard SD card cover a storage outage?
It depends on the card capacity, the camera’s bitrate and whether the card is also used for other recording modes. A 128 GB card at a moderate bitrate may hold a day or more, but high-resolution streams with analytics can reduce that to a few hours. Calculate the figure for your own camera settings and confirm that your VMS supports automatic back-filling, as not all platforms do.
What should I check first if recording does not resume after the controller recovers?
Start by confirming that the recording servers can see the volumes again, as a rescan of disks or a multipath refresh is sometimes needed. Then check that the file system is mounted with the expected drive letter or path, and that the VMS archive location is online. If the VMS still reports errors, restart the recording service and look for archive integrity or re-indexing options before changing any storage configuration.
Is a dual-controller SAN always better than a single high-quality recording server?
Not necessarily. Dual-controller SANs add cost and complexity, and they only help if paths, switches, power and the VMS are also redundant. For smaller or mid-sized systems, a well-specified server with hardware RAID, hot spares and redundant power supplies often gives better value and simpler maintenance. SANs make more sense for large, multi-server or long-retention deployments.
What are the most common mistakes that cause failover to fail in CCTV storage?
The most common mistakes are having only one switch or host adapter, not configuring multipath I/O on the recording servers, and using unprotected write cache. Another frequent issue is sizing storage for normal operation only, so the surviving controller cannot sustain full camera bitrates. Failover is also often never tested, so these faults are only discovered during a real outage.
What should I verify about warranty and support before buying a SAN for CCTV?
Check the hardware replacement time, whether on-site support is included and if spare controllers can be held locally. Ask whether the replacement controller must match the firmware version and whether it can import the existing array configuration without reinitialising. These details determine how long your system runs without full redundancy after a failure.
How do I get started improving the resilience of an existing CCTV storage system?
Begin with a single-points-of-failure audit covering controllers, paths, switches, power and recording servers. Then review VMS logs for past storage errors, confirm cache protection and check whether edge recording is enabled on your cameras. Prioritise fixes by risk and cost, such as adding a second path or enabling edge storage, before considering a full storage replacement.
Related Articles
- What happens when CCTV storage write speeds cannot keep up with incoming video streams?
- How should surveillance storage be designed for disaster recovery?
- What happens when a surveillance server drive fails?
- What RAID level is best for HD surveillance servers?
- How does a surveillance server differ from a generic IT server?