Data Cleaning, Management, and Descriptive Statistics for Validity

Quick Answer: Data cleaning and management are the steps that turn raw collection into an analysis-ready dataset that is valid and reliable, and descriptive statistics are the first analytic pass that summarizes what you have — measures of center (mean, median, mode), spread (range, interquartile range, standard deviation), and shape (skew), plus frequencies and proportions for categorical variables.

Data Cleaning Workflow

Cleaning begins the moment data arrive and continues iteratively. The core steps:

1. Range checks: flag values outside plausible bounds (age 150, systolic blood pressure 0).
2. Consistency checks: cross-variable logic (pregnant male, birth date after death date).
3. Duplicate detection: remove or reconcile repeated records, keyed by study ID.
4. Missing data patterns: distinguish missing completely at random (MCAR), missing at random (MAR), and missing not at random (MNAR); decide between complete-case analysis, single imputation, or multiple imputation based on the mechanism and percent missing.
5. Outlier review: inspect, do not auto-delete; some outliers are true cases (extreme BMI in a diabetes registry) and deleting them biases estimates.
6. Recoding and categorization: collapse continuous variables only when justified; preserve the raw version for re-analysis.
7. Unit standardization: harmonize units (mg/dL versus mmol/L), date formats, and ICD code versions.

Data Management for Reproducibility

Reliable analyses require documentation, not just clean values. A data dictionary records each variable's name, label, type, allowed values, and source question. Version control (git on code, dated file copies for data) prevents silent overwrites. De-identification protects privacy and complies with HIPAA and IRB requirements. Access controls and audit logs support reproducibility and ethics. Public health teams should write analysis scripts that read from the locked clean dataset, never editing it manually, so every result is traceable to a specific file version.

Descriptive Statistics: Summarizing What You Have

Descriptive statistics do not test hypotheses; they characterize the sample. The choice of measure depends on the variable type and distribution.

For categorical variables (nominal or ordinal), use frequencies, proportions, and the mode. For a binary outcome such as smoking status, report the proportion of smokers and a confidence interval.

For continuous variables, begin with measures of central tendency. The mean is the arithmetic average and is optimal for symmetric distributions, but it is pulled by extreme values. The median is the 50th percentile and is preferred for skewed data such as income, length of stay, or biomarker concentrations. The mode is the most frequent value and is most informative for categorical or discrete data.

Measures of spread describe variability. The range (maximum minus minimum) is simple but sensitive to outliers. The interquartile range (IQR), the 25th to 75th percentile, is robust to outliers and pairs with the median. The variance averages squared deviations from the mean; the standard deviation (SD) is its square root and shares the variable's units, pairing with the mean. Empirically, in a normal distribution roughly 68 percent of values fall within plus or minus one SD and 95 percent within plus or minus two SD.

Distribution Shape Matters

Always visualize before summarizing. Skewness describes asymmetry: right-skewed data (income, emergency department visits) have a long right tail, pulling the mean above the median; left-skewed data have a long left tail. Kurtosis describes tail weight and peak sharpness. A histogram or boxplot is the fastest check; if the distribution is visibly skewed or has extreme outliers, report the median and IQR alongside any mean for transparency.

Worked Scenario

A program manager receives a dataset of 1,200 WIC participant hemoglobin values. A histogram shows right skew with a few very high values. Reporting only the mean would overstate the typical hemoglobin; the median with IQR is the better summary. Cleaning flags three records with hemoglobin of 0 (impossible, likely a missing-code convention) and one record with 25 g/dL (verified against the lab record as a data-entry typo for 12.5). The data dictionary is updated to record that 0 means "not drawn," and the corrected file is saved as a new dated version with the original retained for audit.

Exam Traps

• Reporting a mean for skewed data without checking the distribution. Income, costs, and biomarker concentrations usually require a median.
• Confusing standard deviation with standard error. SD describes variability in the sample; SE describes precision of an estimate and is always smaller.
• Auto-removing all outliers. Outliers may be the most important cases; document the reason for any exclusion.
• Using complete-case analysis when data are MAR. Multiple imputation is generally less biased than dropping records.
• Treating descriptive statistics as the final answer. They are the first pass; inferential statistics and interpretation follow, and NBPHE task 8 covers interpretation of quantitative and qualitative results.
• Categorizing continuous variables without preserving the raw version. Categorization loses information and should be reversible.