10.2 Data Pruning, Clipping & Archiving

10.2 Data Pruning, Clipping & Archiving

Long-Term Monitoring (LTM) generates unprecedented volumes of digital electrophysiological and multi-media data. An active 8-to-16 bed Epilepsy Monitoring Unit (EMU) or Pediatric/Adult Neuro-ICU cEEG service produces hundreds of gigabytes of raw data daily and tens of terabytes every month. Managing these massive data repositories requires standardized data pruning protocols, multi-tiered enterprise storage architectures, lossless physiological compression algorithms, cryptographic checksum hashing, and strict adherence to statutory medical record retention mandates.

The Certified Long Term Monitoring Technologist (CLTM) carries the primary clinical responsibility for reviewing continuous recordings, clipping critical electroclinical events, executing standardized pruning workflows, and safeguarding permanent master archives.

---

1. Storage Economics & Continuous Data Volume Calculations

To manage enterprise neurodiagnostic storage, technologists must understand the mathematical components contributing to continuous data generation.

+-----------------------------------------------------------------------------+
|                     24-HOUR LTM DATA FOOTPRINT CALCULATION                  |
|                                                                             |
|   [DIGITAL EEG STREAM]                                                      |
|   - 64 Channels @ 500 Hz Sampling Rate, 24-bit (3 bytes) Resolution        |
|   - Calculation: 64 ch * 500 samples/s * 3 bytes = 96,000 bytes/s (96 KB/s)|
|   - Daily Volume: 96 KB/s * 86,400 s/day = ~8.29 GB / day (Uncompressed)  |
|                                                                             |
|   [HIGH-DEFINITION AUDIO]                                                   |
|   - AAC / PCM Stereo @ 48 kHz, 16-bit = ~192 Kbps = ~2.07 GB / day         |
|                                                                             |
|   [1080p HD VIDEO STREAM]                                                   |
|   - H.264 / H.265 @ 30 fps, Medium Profile (Bitrate: ~3.5 Mbps)             |
|   - Daily Volume: (3.5 Mbps / 8) * 86,400 s = ~37.80 GB / day               |
|   -----------------------------------------------------------------------   |
|   [TOTAL UNPRUNED DAILY PAYLOAD PER BED]       --->  ~48.16 GB / day        |
|   [8-Bed EMU Daily Production]                 --->  ~385.3 GB / day        |
|   [8-Bed EMU Annual Storage Production]        --->  ~140.6 TERABYTES / yr  |
+-----------------------------------------------------------------------------+

Multi-Channel Data Math by Sampling Parameters

The raw storage requirements for electrophysiological signals scale linearly with channel count, sampling rate ($f_s$), and bit depth: $\text{Data Rate (Bytes/sec)} = N_{\text{channels}} \times f_s \;(\text{Hz}) \times \text{Bytes per Sample}$ $\text{Daily EEG Storage (GB)} = \frac{\text{Data Rate (Bytes/s)} \times 86,400\text{ s}}{1024^3}$

Without aggressive, protocolized data pruning, hospital storage networks quickly become exhausted, multi-day study retrieval times suffer severe latency, and disaster recovery backup windows fail to complete.

---

2. Systematic Data Pruning Protocols & Mandatory Inclusions

Data pruning is the systematic, physician-supervised curation process of removing non-diagnostic, redundant continuous baseline recording (such as hours of unremarkable resting wakefulness or stable non-REM sleep) while consolidating all clinically and electrophysiographically critical epochs into an immutable permanent master archive.

+-----------------------------------------------------------------------------+
|                        CLINICAL DATA PRUNING WORKFLOW                       |
|                                                                             |
|   [RAW 24-HOUR CONTINUOUS VIDEO-EEG RECORDING (40-60 GB)]                   |
|                                |                                            |
|                                v                                            |
|   [CLTM TECHNOLOGIST SYSTEMATIC EVENT SELECTION & REVIEW]                   |
|   - Identify baseline awake background (eyes open/closed, reactivity)       |
|   - Identify complete sleep architecture (N1, N2, N3, REM)                  |
|   - Capture activation procedures (Hyperventilation, Photic Stimulation)    |
|   - Isolate ALL clinical & subclinical seizures (Add Pre/Post Margins)      |
|   - Sample representative interictal epileptiform discharges (IEDs)         |
|   - Retain ALL patient/family push-button events (even if normal EEG)       |
|   - Retain daily bio-calibration and impedance logs                         |
|                                |                                            |
|                                v                                            |
|   [EXECUTE PRUNING / METADATA CONSOLIDATION ENGINE]                         |
|   - Redundant baseline video deleted or downsampled                         |
|   - Critical epochs preserved at full native resolution with SMPTE sync     |
|                                |                                            |
|                                v                                            |
|   [PERMANENT CLINICAL MASTER ARCHIVE (PRUNED RECORD: 2-5 GB)]               |
|   - Generates cryptographic checksum (SHA-256)                              |
|   - Transfers to Enterprise Nearline Archive / Long-Term Cold Storage       |
+-----------------------------------------------------------------------------+

Mandatory Inclusions for the Permanent Master Record

Every pruned LTM study must contain a standardized set of physiological samples to ensure diagnostic validity, legal compliance, and baseline comparability for future evaluations:

1. Baseline Waking Background:
   • A clean 10-to-20 minute epoch of alert wakefulness displaying the Posterior Dominant Rhythm (PDR / alpha rhythm), normal anterior beta activity, and physiological eye movements.
   • Documented state transitions with eye opening and eye closure demonstrating background reactivity and voltage attenuation.
2. Complete Sleep Architecture:
   • Representative 5-to-10 minute epochs of Stage N1 sleep (vertex sharp transients, slow rolling eye movements).
   • Stage N2 sleep (well-formed, symmetric sleep spindles [11–16 Hz] and K-complexes).
   • Stage N3 slow-wave sleep (high-voltage delta activity occupying ≥20% of epoch).
   • Stage REM sleep (low-voltage mixed-frequency background, rapid eye movements, and chin EMG muscle atonia).
3. Provocative & Activation Procedures:
   • The entire continuous duration of Hyperventilation (HV) plus at least 3 minutes of post-HV recovery.
   • The complete sequence of Intermittent Photic Stimulation (IPS) across all flash frequencies (1 Hz to 30+ Hz) with eyes-open and eyes-closed conditions.
4. All Electrographic and Electroclinical Seizures:
   • Pre-Event Context: Preserve enough synchronized baseline before electrographic or behavioral onset to show the earliest change. The clip margin is defined by the department protocol and the event—not a universal 60- or 120-second rule.
   • Ictal/Event Epoch: Preserve the complete behavioral and electrographic event without interruption.
   • Recovery Context: Preserve post-event EEG, video, cardiorespiratory data, and testing through clinically meaningful recovery or the protocol-defined endpoint. Extend the clip when slowing, suppression, hypoxemia, confusion, or another consequence persists.
5. Interictal Epileptiform Discharges (IEDs):
   • Representative samples of every unique spike, sharp wave, polyspike, and paroxysmal fast activity morphology during both waking and sleeping states.
6. All Patient Event-Button Presses & Staff Markers:
   • Strict Mandate: Every event marker triggered by the patient, family, or nursing staff must be permanently retained with full video and audio, regardless of whether any electrographic change is observed.
   • Clinical Rationale: Retaining push-button events with normal EEG is essential for diagnosing or ruling out Psychogenic Non-Epileptic Seizures (PNES), physiological syncope, panic attacks, paroxysmal kinesigenic dyskinesia, or subtle sensory auras arising from cortical areas too small to generate scalp-detectable potentials.
7. Daily Instrument Calibration & Impedance Logs:
   • Initial and daily electronic calibration signals, biological calibration ($F_{p1}-O_2$ montage check), and daily 24-hour electrode impedance audit tables (<5 kΩ).

---

3. Compression Technologies: Lossless vs. Lossy

Data compression reduces digital storage footprints, but the technologist must distinguish where compression is clinically safe and where it introduces diagnostic distortion.

+-----------------------------------------------------------------------------+
|                 COMPRESSION MECHANICS: LOSSLESS vs. LOSSY                   |
|                                                                             |
|   [LOSSLESS COMPRESSION (MANDATORY FOR EEG / ELECTROPHYSIOLOGY)]            |
|   - Algorithms: Delta Encoding, Huffman Coding, Arithmetic Coding, FLAC-EEG |
|   - Mechanism: Reversible mathematical bit-packing; ZERO data discarded     |
|   - Fidelity: 100% exact numerical voltage sample reconstruction (Bit-for-Bit)|
|   - Compression Ratio: 2:1 to 3:1 (Reduces raw EEG size by 50-65%)          |
|                                                                             |
|   [LOSSY COMPRESSION (STANDARD FOR VIDEO & AUDIO ONLY)]                     |
|   - Codecs: H.264 (MPEG-4 AVC), H.265 (HEVC), AAC Audio                     |
|   - Mechanism: Discards perceptually redundant spatial/temporal pixel data  |
|   - Fidelity: High visual fidelity, but mathematically irreversible         |
|   - Hazards: Severe pixel macroblocking, motion smearing, dropped frames    |
|   - Compression Ratio: 20:1 to 50:1 (Enables sustainable video storage)     |
+-----------------------------------------------------------------------------+

Physiological Signal Integrity

• Lossless Rule for Raw EEG: Medical device regulations and ACNS standards mandate that continuous electrophysiological numerical voltage samples must never undergo lossy compression. Lossy compression alters microvolt-level waveform amplitudes, distorts high-frequency oscillations (HFOs: 80–500 Hz), truncates spike sharpness, and introduces mathematical phase shifts.
• Lossy Video Codec Management: Video streams rely on H.264 or H.265 codecs. The technologist must ensure that video bitrates are not compressed below threshold (minimum 2.0–3.5 Mbps for 1080p), which prevents macroblocking artifacts during violent hypermotor or convulsive movements.

---

4. Enterprise Archiving Media & Tiered Storage Architecture

Healthcare information technology utilizes Information Lifecycle Management (ILM) to migrate neurophysiology data across physical storage tiers based on clinical access frequency and age.

+-----------------------------------------------------------------------------+
|                  HIERARCHICAL NEUROPHYSIOLOGY STORAGE TIERS                 |
|                                                                             |
|   [TIER 1: HOT STORAGE (Active Acquisition & Real-Time Monitoring)]         |
|   - Media: Enterprise NVMe / SAS SSDs on Bedside Carts & SAN Arrays         |
|   - Redundancy: RAID 10 (Striped Mirrors) for ultra-low latency & high IOPS |
|   - Retention: Active hospital admission (0 to 30 days)                     |
|   - Retrieval Latency: Sub-second (<1 ms)                                   |
|                                |                                            |
|                                v                                            |
|   [TIER 2: WARM STORAGE (Nearline Clinical Access & Pruning Curation)]      |
|   - Media: Network-Attached Storage (NAS) / SAN SAS HDD Arrays              |
|   - Redundancy: RAID 6 (Dual Distributed Parity: survives 2 drive losses)   |
|   - Retention: 30 days to 1 year post-discharge                             |
|   - Retrieval Latency: 1 to 5 seconds                                       |
|                                |                                            |
|                                v                                            |
|   [TIER 3: COLD STORAGE (Permanent Deep Archive & Compliance)]              |
|   - Media: Immutable Cloud Object Storage (AWS S3 Glacier, Azure Archive)   |
|            Enterprise LTO-8 / LTO-9 Magnetic Tape / Optical BDXL WORM       |
|   - Redundancy: Geographical multi-region replication & physical air-gapping|
|   - Retention: 7 to 28+ years (Statutory legal mandates)                    |
|   - Retrieval Latency: Minutes to hours                                     |
+-----------------------------------------------------------------------------+

Enterprise Storage Media Specifications

1. Local Solid-State Drive (SSD) Acquisition Buffer:
   • Every bedside acquisition workstation contains an internal 1–2 TB enterprise NVMe SSD configured as a circular First-In, First-Out (FIFO) write buffer. It holds at least 24 to 48 hours of unpruned raw video-EEG locally. If the hospital network fails, recording continues uninterrupted, syncing back to the SAN automatically upon reconnection.
2. Storage Area Networks (SAN) & Network-Attached Storage (NAS):
   • SAN (Block Storage): Utilizes high-speed Fibre Channel (FC) or iSCSI to connect servers directly to dedicated block arrays, delivering maximum I/O throughput for real-time multi-bed writing.
   • NAS (File Storage): Utilizes NFS or SMB/CIFS protocols over the hospital LAN for shared physician review access.
   • RAID Configurations: RAID 10 provides maximum write speed for active acquisition; RAID 6 provides dual-drive parity protection for high-capacity nearline repositories.
3. Linear Tape-Open (LTO) Magnetic Tape:
   • LTO-9 cartridges hold 18 TB uncompressed (up to 45 TB compressed) with a certified shelf lifespan exceeding 30 years. Physical tape cartridges provide an immutable "air-gapped" barrier against hospital ransomware attacks.
4. Write-Once-Read-Many (WORM) Optical Media:
   • Professional optical media (Archival Disc / BDXL 100–128 GB) use physical phase-change layers that cannot be magnetically altered, erased, or rewritten, satisfying strict medico-legal non-erasable record criteria.
5. Immutable Cloud Storage:
   • Enterprise cloud tiers (AWS S3 Glacier Deep Archive, Microsoft Azure Archive) offer immutable Object Locking in Compliance Mode, ensuring files cannot be overwritten or deleted by any user or administrator until the legal retention clock expires.

---

5. Cryptographic Integrity Hashing & Forensic Admissibility

When neurophysiology files are migrated between acquisition carts, network servers, and cloud cold storage, silent file corruption (bit rot, hardware sector failure, or dropped network packets) can compromise data integrity.

+-----------------------------------------------------------------------------+
|                 CRYPTOGRAPHIC CHECKSUM VERIFICATION (SHA-256)               |
|                                                                             |
|   [Raw Pruned Study File]  --->  [SHA-256 Algorithm]  ---> [Source Hash: a3f9...] |
|                                                                   |         |
|   (Network Migration / Cloud Transfer / Tape Write)               |         |
|                                                                   v         |
|   [Migrated Archive File]  --->  [SHA-256 Algorithm]  ---> [Dest Hash: a3f9...]   |
|                                                                   |         |
|   +---------------------------------------------------------------+         |
|   |                                                                         |
|   v                                                                         |
|   [MATCH: 100% Bit-Level Integrity Verified]                                |
|   [MISMATCH: File Corrupted -> Auto-Reject & Trigger Retransfer]            |
+-----------------------------------------------------------------------------+

Cryptographic Hash Standards

• SHA-256 (Secure Hash Algorithm 256-bit): The gold standard cryptographic algorithm. SHA-256 processes the entire binary content of an LTM study and generates a unique 64-character hexadecimal signature. If even a single binary bit (a 0 changed to a 1) is altered during migration, the resulting hash changes completely (the avalanche effect). Matching SHA-256 hashes proves 100% bit-level integrity.
• MD5 (Message Digest 5): A legacy 128-bit checksum. While computationally fast, MD5 is cryptographically vulnerable to mathematical hash collisions and is no longer recommended for forensic legal authentication, though it remains in use for rapid low-level network transfer verification.
• Legal Chain of Custody: In forensic litigation (e.g., medical malpractice defense, criminal responsibility evaluations, or brain death documentation), a preserved hash and audit trail can help demonstrate whether a retained file changed after hashing. Their evidentiary use depends on the full chain-of-custody process and governing legal rules; no checksum makes a record “unassailable.”

---

6. Legal and Regulatory Record Retention

There is no single federal rule that makes every adult LTM record “7–10 years” or every pediatric record “age of majority plus 7 years.” Retention depends on state law, the organization’s health-information-management schedule, payer/accreditation obligations, litigation holds, research consent, and whether the item is the legal medical record. The medical director and HIM/legal teams define which raw signals, synchronized video, clips, logs, and reports must be kept and for how long.

Before pruning, confirm the current approved schedule and any hold. Preserve all required habitual events, electrographic seizures, representative backgrounds and sleep, clinically important artifacts, activation procedures, technical interruptions, annotations, and recovery. Never delete merely because a segment seems uninteresting, and never infer a child’s destruction date from an age formula. Record what was pruned, by whom, when, under which policy, and verify the retained copy before source deletion.

7. Comparative Pruning & Storage Summary Matrix

[!IMPORTANT] Event-context rule: Preserve enough synchronized baseline to demonstrate the earliest electrographic or behavioral change, the complete event, bedside testing, and meaningful post-event recovery. The approved protocol may define minimum clip windows, but the event determines whether more context is needed.

[!CAUTION] Push-Button Event Deletion Hazard: Never delete a patient push-button event marker simply because the simultaneous EEG background is normal. Retaining the complete video, audio, and EEG of event-button presses is clinically mandatory to confirm or exclude non-epileptic spells and focal sensory auras.