Lead Forensics

How should SAN storage performance be tested before a CCTV system goes live?

SAN storage performance should be tested before a CCTV system goes live by simulating the full production workload: every camera stream at its real bitrate, run as a sustained mixed read and write load for at least 24 hours, then repeated while the array is degraded and rebuilding. A short synthetic benchmark on an idle array proves very little. Surveillance recording is continuous sequential writing with unpredictable bursts of playback, export and analytics, so test conditions must reflect that. The sections below cover the metrics, tools, test duration, failure testing, common false passes and the results that justify go-live for any IP CCTV camera system built on surveillance storage.

Which performance metrics matter most for CCTV storage?

The metrics that matter most for CCTV storage are sustained sequential write throughput, write latency under load, dropped or missing frames, and performance while the array is degraded. Peak IOPS and headline read speeds matter far less, because recording is a constant, write-dominated workload.

Measure each of these against your design figures rather than against vendor datasheets:

  • Sustained write throughput: the aggregate camera bitrate, plus headroom for overhead, must be written continuously without the rate tapering off once caches fill.
  • Latency, including worst case: average latency hides stalls. Record the 99th percentile and maximum latency, since a single multi-second pause can cause a recording server to drop frames.
  • Concurrent read load: playback, investigation and export requests compete with recording, so measure write performance while reads are active.
  • Frame loss and recording gaps: the VMS ultimately judges storage, so check its logs and recorded timelines, not only the storage tool’s output.
  • Path utilisation: monitor controller CPU, cache behaviour, and network or Fibre Channel link saturation during the test.

Document the design target before testing begins. Without a stated required throughput and retention period, test results have nothing to be compared against.

What tools can simulate CCTV workloads on a SAN?

CCTV workloads on a SAN can be simulated with general storage benchmarking tools such as fio, Iometer or Vdbench, configured for large sequential writes across many parallel streams, or with surveillance-specific simulators that emulate camera streams directly. The closer the simulation is to real camera behaviour, the more reliable the result.

Generic tools are useful for establishing raw capability of the LUNs, but they need careful configuration. Use block sizes and queue depths typical of video recording, run one write stream per simulated camera rather than a single large stream, and add a separate read workload to represent playback. Writing a single file in one stream flatters almost every array.

Surveillance-specific simulation is more representative. Secure Logiq’s Logiqal Benchmark utility, for example, lets designers virtually deploy a multi-camera IP surveillance network, specifying camera resolutions, compression formats, frame rates and analytics workloads, to validate server and storage throughput limits before physical deployment. Whichever tool you choose, many VMS vendors also provide camera simulators that push real recorded streams into the recording servers, which exercises the complete chain rather than the storage alone.

Use more than one method. Raw benchmark results show what the storage can do; VMS-level simulation shows what the whole recording system actually does.

How long should a sustained write test run?

A sustained write test should run for at least 24 hours, and ideally 48 to 72 hours, at full design load. Short tests of a few minutes only measure cache performance. Surveillance storage must prove it can hold its write rate long after caches, tiers and thin-provisioned space have been exhausted.

Several behaviours only appear over time:

  • Cache exhaustion: write performance often falls sharply once the controller or SSD cache fills.
  • Thermal effects: drives and controllers under continuous load can behave differently after hours of heat build-up.
  • Background tasks: scheduled scrubs, snapshots, replication and consistency checks can compete with recording at predictable times.
  • Retention rollover: once storage fills and the VMS starts overwriting old footage, the workload changes to simultaneous writing and deleting. If feasible, test in this state too.

Run the test across a working day boundary so that peak-hour playback and export activity overlap with recording. If the test period cannot cover every scenario, prioritise the 24-hour full-load run, then the failure scenarios described next.

How do you test SAN performance during failures and rebuilds?

You test SAN performance during failures by deliberately inducing faults while the full recording workload runs, then measuring throughput, latency and frame loss throughout. The key scenarios are a failed drive and the resulting rebuild, a failed controller or path, and loss of a power supply.

Run each scenario under full load, because a SAN that performs well when healthy can fall below the required write rate once it is degraded. Work through these tests in order:

  1. Single drive failure: remove or fail a drive, confirm recording continues, and note the immediate impact on write performance.
  2. Rebuild under load: allow the rebuild to start, then record throughput and latency until completion. Note how long the rebuild takes, since that is the period of reduced protection.
  3. Path and controller failover: disconnect a Fibre Channel or iSCSI path, or fail a controller, and measure how long recording is interrupted, if at all.
  4. Power redundancy: remove one power supply to confirm the array and recording servers continue uninterrupted.
  5. Recovery: restore failed components and confirm that performance returns to baseline and the VMS resumes cleanly.

RAID is not a backup, and redundancy does not automatically guarantee uninterrupted recording. Hot spares, battery-backed controllers and redundant power are what allow recording to continue through failures, but only testing shows whether a particular configuration holds its write rate while degraded. Failover mechanisms should be tested rather than assumed, particularly where a design depends on automated failover and failback.

What causes SAN tests to pass but live CCTV recording to fail?

SAN tests pass but live CCTV recording fails when the test does not match real conditions: short durations, idle arrays, unrealistic stream patterns, untested failure states, or bottlenecks elsewhere in the chain that a storage benchmark never touches. The storage can be technically capable while the system as a whole is not.

The most common causes are:

  • Testing the array in isolation: recording servers, network switches, HBAs and VMS configuration are all part of the data path. A saturated network uplink or an undersized recording server will cause frame loss that the SAN never sees.
  • Ignoring analytics and metadata: AI video analytics generates additional metadata and read activity. If metadata shares volumes with video, it can increase fragmentation and latency over time.
  • Shared SAN resources: a SAN also serving other applications may meet its benchmark in a quiet test window, then suffer contention in production.
  • Bitrate underestimation: real cameras in busy scenes, low light or with increased motion produce higher bitrates than average-based design figures assume.
  • Thin provisioning and growth: performance on a freshly created, empty volume often differs from the same volume when heavily used.
  • Single-point testing: passing a healthy-state test says nothing about degraded behaviour.

Accurate bandwidth and storage calculations at the design stage reduce these surprises. Secure Logiq’s system design service covers end-to-end analysis of the recording architecture, so bottlenecks in the wider chain are identified before testing begins rather than after go-live.

What results confirm a SAN is ready for go-live?

A SAN is ready for go-live when it holds the full design write rate for at least 24 hours with no frame loss, keeps worst-case latency within the limits the VMS tolerates, and continues recording acceptably through drive failure, rebuild and failover tests. Results must be documented, repeatable and signed off against the original design targets.

Use this as a go-live checklist:

  • Sustained aggregate write throughput meets the design figure with agreed headroom, with no downward drift over the test period.
  • Zero unexplained dropped frames or recording gaps in the VMS logs and timelines.
  • Latency percentiles and maximum latency remain stable under combined recording and playback load.
  • Recording continues, within an agreed tolerance, during a drive failure and rebuild.
  • Controller, path and power failover events are demonstrated, with measured recovery times recorded.
  • Monitoring and alerting are configured and confirmed to flag capacity, degraded arrays and failed components.
  • Test configuration, tools, durations and results are recorded for future reference and fault-finding.

Treat the results as a baseline. Retaining them lets the team compare future performance against the commissioning state when a fault or expansion arises. For a real-world example of shared storage in a large deployment, the Qatar University SAN storage case study shows how this kind of infrastructure is applied in practice.

If you are planning a SAN-based surveillance storage deployment and want the design and test approach validated before go-live, talk to Secure Logiq’s technical team about your recording and retention requirements.

Frequently Asked Questions

How do I get started with a SAN performance test if I have never done one for a CCTV system?

Start by documenting your design targets: total camera count, bitrate per camera, aggregate write throughput, retention period and expected playback or export concurrency. Then run a raw benchmark with fio or Vdbench against the LUNs to establish baseline capability, followed by a VMS-level simulation using camera simulators or recorded streams. Build up to the 24-hour full-load run only after the baseline results show comfortable headroom against your design figures.

What headroom should I allow above my calculated aggregate camera bitrate?

A common practice is to design for at least 20-30% above the calculated average bitrate, because real cameras produce higher bitrates in busy scenes, low light or heavy motion. Also allow for RAID write overhead, playback and export activity, and the reduced performance of a degraded array. Test using worst-case or peak bitrates rather than averages to confirm the headroom is genuine.

What should I do if write throughput drops after several hours of testing?

A gradual or sudden drop usually points to cache exhaustion, thermal throttling, background tasks such as scrubs or snapshots, or thin-provisioned space being allocated. Correlate the timing of the drop with controller logs, cache statistics and scheduled jobs. Resolve it by adjusting or rescheduling background tasks, adding drives or cache tiers, or redesigning the RAID layout, then rerun the full test rather than assuming the fix worked.

Can I run these tests on a SAN that is shared with other applications?

You can, but the results will only be meaningful if you test while the other workloads are running at realistic or peak levels. A quiet test window can hide contention that appears in production. Ideally, isolate the surveillance workload on dedicated LUNs, disks or storage pools, and apply QoS limits where the SAN supports them to protect recording from other tenants.

How do I know whether frame loss is caused by the SAN or by something else in the recording chain?

Compare the VMS logs and recording timelines with storage-side metrics for the same time period. If storage latency spikes or throughput dips coincide with the gaps, the SAN is the likely cause. If storage metrics are clean, check network switch and uplink utilisation, HBA or NIC errors, recording server CPU and memory, and camera-side issues such as dropped network packets.

How often should I retest SAN performance after go-live?

Repeat a reduced version of the test after any significant change, such as adding cameras, increasing resolution or frame rate, enabling analytics, firmware upgrades or storage expansion. It is also good practice to review performance against your commissioning baseline at least annually. Continuous monitoring of latency, capacity and array health helps you catch degradation between formal retests.

Is it safe to simulate drive failures on a production-ready array?

Fault testing should be done before go-live, when no live footage is at risk, and with vendor guidance for the specific array. Confirm that hot spares are configured, controller batteries are healthy and any critical data is backed up before pulling drives or paths. Never run destructive failure tests on a live system without a rollback plan and a maintenance window agreed with stakeholders.

Related Articles