5.1 Character Functions for String Manipulation

5.1 Character Functions for String Manipulation

In SAS DATA step programming, character variables are widely used to store text data such as customer names, addresses, identification codes, and narrative descriptions. SAS provides a robust suite of character functions designed to search, extract, clean, format, and concatenate string values. Understanding how these functions operate on character arguments—and how they handle variable lengths and padding—is fundamental to data preparation and a primary topic on the SAS Certified Specialist Base Programming exam.

---

1. Substring Extraction & In-Place Assignment: SUBSTR

The SUBSTR (substring) function serves a dual purpose in SAS depending on whether it appears on the right side or the left side of an assignment statement.

Substring Extraction (Right Side of =)

When positioned on the right side of an assignment statement or within an expression, SUBSTR extracts a portion of a character string based on a starting position and an optional length.

$\text{Syntax: } \text{SUBSTR}(\textit{string}, \textit{position}, <\textit{length}>)$

• string: Specifies the character variable, literal, or expression to parse.
• position: Specifies the 1-based starting character position. If position exceeds the length of the string, SAS returns a blank string and writes a note to the SAS log.
• length: Specifies the number of characters to extract. If omitted, SUBSTR extracts all remaining characters from position to the end of the string.

data work.substr_extract;
    code = "ACCT-8904-US";
    dept = substr(code, 1, 4);    /* Extracts 'ACCT' */
    num  = substr(code, 6, 4);    /* Extracts '8904' */
    region = substr(code, 11);    /* Extracts 'US' (length omitted) */
run;

Substring Modification (Left Side of =)

When SUBSTR appears on the left side of an assignment statement, it functions as a variable modifier, replacing characters in an existing variable starting at the specified position.

$\text{Syntax: } \text{SUBSTR}(\textit{variable}, \textit{position}, <\textit{length}>) = \text{expression};$

data work.substr_modify;
    phone = "919-555-0100";
    /* Replace area code 919 with 984 */
    substr(phone, 1, 3) = "984";  /* Result: '984-555-0100' */
run;

[!IMPORTANT] When using SUBSTR on the left side of an assignment, the target variable must already be defined. The length of the target variable does not expand; if the replacement string is longer than the specified length, the extra characters are truncated.

---

2. Parsing Tokens with the SCAN Function

The SCAN function selects and returns a specific word or token from a character string based on specified delimiters.

$\text{Syntax: } \text{SCAN}(\textit{string}, \textit{count}, <'\textit{delimiters}'>)$

• count: Specifies the position of the word to extract. A positive integer selects words counting from left to right. A negative integer selects words counting from right to left.
• delimiters: A character string listing the characters that separate words. If omitted, SAS uses default delimiters: blank, period, left parenthesis, plus sign, ampersand, percent sign, vertical bar, dollar sign, asterisk, right parenthesis, semicolon, hyphen, minus sign, slash, comma, and double quotation mark.

data work.scan_example;
    fullname = "Jane Marie Van-Der-Bilt";
    
    first_name = scan(fullname, 1, " ");        /* 'Jane' */
    middle_name= scan(fullname, 2, " ");        /* 'Marie' */
    last_token = scan(fullname, -1, " -");      /* 'Bilt' (scans from right) */
    second_last= scan(fullname, -2, " -");      /* 'Der' */
run;

Key Behaviors of SCAN:

1. Consecutive delimiters are treated as a single delimiter.
2. Leading delimiters are ignored.
3. If count is greater than the total number of words in the string, SCAN returns a blank value.
4. In SAS 9.4, if the length of the target variable is not explicitly declared with a LENGTH statement, SAS automatically assigns a default length of 200 bytes to variables created with SCAN.

---

3. String Concatenation: CAT, CATS, CATT, and CATX

While the traditional concatenation operator (||) joins character strings together, it preserves all leading and trailing blanks. SAS provides dedicated CAT family functions to manage whitespace during concatenation.

data work.concatenation;
    length title $ 10 first $ 10 last $ 10 full_name $ 40;
    title = " Dr.";
    first = "Robert ";
    last  = "Miller ";
    
    /* Using CATX strips whitespace and inserts single space */
    full_name = catx(" ", title, first, last); /* Result: 'Dr. Robert Miller' */
run;

---

4. Searching & Measuring Strings: INDEX & LENGTH

Finding Substrings with INDEX

The INDEX function searches a source string for a target string and returns the character position of the target's first occurrence.

$\text{Syntax: } \text{INDEX}(\textit{source}, \textit{excerpt})$

data work.index_demo;
    text = "The quick brown fox jumps over the lazy dog";
    pos1 = index(text, "brown"); /* Returns 11 */
    pos2 = index(text, "cat");   /* Returns 0 (not found) */
run;

Measuring Length: LENGTH vs. LENGTHN

• LENGTH(string): Returns the position of the last non-blank character. If the string is completely blank, LENGTH returns 1.
• LENGTHN(string): Returns the position of the last non-blank character, but returns 0 if the string is completely blank.

data work.length_comparison;
    empty_str = "    ";
    len1 = length(empty_str);  /* Returns 1 */
    len2 = lengthn(empty_str); /* Returns 0 */
run;

---

5. Cleaning & Formatting Character Data

SAS offers several utility functions to standardize and clean messy character data. Two of them — TRIM and COMPRESS — are named explicitly in the A00-231 content guide alongside SCAN, SUBSTR, UPCASE, and LOWCASE.

• TRIM(string): Removes trailing blanks only; leading blanks are preserved. TRIM is what you reach for before concatenating with the || operator, because SAS pads every character variable to its full declared length. trim('Cary ') || ', NC' returns 'Cary, NC', whereas 'Cary ' || ', NC' returns 'Cary , NC'.
• COMPRESS(string, <characters>, <modifiers>): Removes characters from anywhere in the string, not just the ends. With one argument it strips every blank wherever it occurs: compress('9 1 9 5 5 5') returns '919555'. With a second argument it removes exactly the listed characters: compress('(919) 555-0100', '() -') returns '9195550100'. Modifiers add character classes, so compress(id, , 'kd') keeps only digits (k = keep, d = digits) — the standard one-liner for stripping punctuation out of phone numbers and account codes.
• STRIP(string): Removes both leading and trailing blanks.
• COMPBL(string): Compresses multiple consecutive internal blanks into a single blank space.
• PROPCASE(string, <delimiters>): Capitalizes the first letter of each word and converts all other letters to lowercase.
• UPCASE(string) / LOWCASE(string): Converts all characters to uppercase or lowercase.

data work.compress_demo;
    phone_raw = "(919) 555-0100";

    digits_only = compress(phone_raw, '() -');   /* '9195550100' */
    keep_digits = compress(phone_raw, , 'kd');   /* '9195550100' - keep digits only */
    no_blanks   = compress("9 1 9 5 5 5");        /* '919555'    - all blanks removed */

    city = "Cary   ";
    joined_bad  = city || ", NC";          /* 'Cary   , NC'  */
    joined_good = trim(city) || ", NC";    /* 'Cary, NC'     */
run;

data work.clean_data;
    raw_name = "   jOHN   wILLIAM   sMITH ";
    
    /* Step 1: Reduce multiple blanks to single spaces */
    comp_name = compbl(raw_name);      /* ' jOHN wILLIAM sMITH ' */
    
    /* Step 2: Strip leading/trailing blanks */
    stripped  = strip(comp_name);      /* 'jOHN wILLIAM sMITH' */
    
    /* Step 3: Proper capitalization */
    clean_name = propcase(stripped);   /* 'John William Smith' */
run;