7.3 Choosing an Effective Assessment Method and Criterion-Referenced Testing

7.3 Choosing an Effective Assessment Method and Criterion-Referenced Testing

Quick Answer: When deciding how to assess learner progress, FAA-H-8083-9B gives aviation instructors a four-step process: (1) determine level-of-learning objectives, (2) list indicators/samples of desired behaviors, (3) establish criterion objectives, and (4) develop criterion-referenced assessment items. The process runs general to specific and works for the cognitive and affective domains as well as psychomotor skill testing. A criterion objective adds conditions and criteria to the expected behavior — so if you wrote performance-based objectives into your lesson plans, your criterion objectives already exist. The ACS and PTS are criterion-referenced standards, which is why the instructor evaluates every task before recommending an applicant.

---

Why Instructors Get This Backward

The common failure is building the test first: an instructor writes twenty questions about carburetor icing, gives the quiz, and only afterward asks what the score was supposed to prove. The handbook's process runs the other direction — general to specific — and each step constrains the next. Start with what level of learning you intended to produce, and the test items fall out of it.

There is also a timing decision hidden in this. The handbook cautions that authentic assessment may not be as useful as traditional assessment in the early phases of training, because the learner "does not have enough information about the concepts or knowledge to participate fully." This tracks the knowledge-acquisition sequence from section 3.4: a learner exposed to a new topic first acquires and memorizes facts, then organizes them into understanding. Only once the learner possesses enough knowledge to analyze, synthesize, and evaluate — the application and correlation levels — can they participate fully in a collaborative, learner-centered assessment. Handing a first-lesson student a self-assessment worksheet is not learner-centered; it is abandonment.

---

Step 1: Determine Level-of-Learning Objectives

State the individual objectives as general, level-of-learning objectives. The objective should measure one of the learning levels in the cognitive, affective, or psychomotor domains. The cognitive levels named by the handbook are knowledge, comprehension, application, analysis, synthesis, and evaluation.

The handbook's worked example is a maintenance one, and it is worth following exactly because the FOI tests the boundaries between levels:

Understanding-level objective: "Describe how to perform a compression test on an aircraft reciprocating engine."

Notice what this objective does not require:

The general level-of-learning objective is "a good starting point for developing a test because it defines the scope of the learning task." Get the level wrong and every later step measures the wrong thing — which is exactly how an instructor ends up asking a pre-solo student to evaluate a crosswind technique they have only been taught to describe.

---

Step 2: List Indicators / Samples of Desired Behaviors

List the behaviors that give the best indication that the objective has been achieved. The instructor selects behaviors that (a) can be measured and (b) give the best evidence of learning.

Continuing the compression-test example, if the learner is expected to display the understanding level, the test answers should describe:

• appropriate tools and equipment;
• the proper equipment setup;
• appropriate safety procedures; and
• the steps used to obtain compression readings.

Two handbook constraints govern this step:

• The overall test must be comprehensive enough to give a true representation of the learning to be measured.
• It is not usually feasible to measure every aspect of a level-of-learning objective — so by carefully choosing samples of behavior, the instructor obtains adequate evidence of learning.

That pair of statements is the resolution to a classic FOI tension: comprehensiveness is a required characteristic of a good test, yet no test can cover everything. Sampling deliberately is the answer; sampling accidentally is not.

---

Step 3: Establish Criterion Objectives

A criterion objective is a performance-based objective. In addition to the behavior expected, it states:

1. the conditions under which the behavior is to be performed, and
2. the criteria that must be met.

The handbook makes the labor-saving point explicit: if the instructor developed performance-based objectives while creating lesson plans, the criterion objectives have already been formulated. The criterion objective then "provides the framework for developing the test items used to measure the level of learning objectives."

Handbook example: "The learner will demonstrate understanding of compression test procedures for reciprocating aircraft engines by completing a quiz with a minimum passing score of 70 percent."

Break that apart against the three-element structure taught in section 9.2:

---

Step 4: Develop Criterion-Referenced Assessment Items

Write items that measure the behaviors described in the criterion objective. In the compression-test example, the questions should cover every area necessary to give evidence of understanding the procedure. The diagnostic payoff comes afterward: the questions missed identify the areas that were not adequately covered — the test grades the instruction as well as the learner.

The handbook adds two practical applications:

• A pre-solo knowledge test measures whether the learner understands the regulations, the local area, the aircraft type, and the procedures to be used, so it must be written against those objectives. Individual instructors should develop their own tests to measure their own learners' progress.
• A readiness test for an FAA knowledge test should be based on the objectives of all the lessons the learner has received, not just the recent ones.

---

Performance Testing and the ACS/PTS as Criterion-Referenced Standards

Aviation training also involves performance tests for maneuvers or procedures. The flight instructor does not administer the practical test for a pilot certificate, and the maintenance instructor does not administer the AMT oral and practical exam — but instructors do exactly the same kind of skill or performance testing in training. Performance testing is desirable for evaluating training that involves an operation, a procedure, or a process, and the instructor's job is to prepare the learner for those tests.

Practical tests for pilots and maintenance technicians are criterion-referenced tests: the objective is for all successful applicants to meet the high standards of knowledge, skill, and safety required by the regulations — not to rank applicants against each other. The purpose of the ACS/PTS is to delineate the standards by which FAA inspectors, designated pilot examiners (DPEs), and designated maintenance examiners (DMEs) conduct tests, reflecting 14 CFR parts 61, 65, 91, the AIM, and pertinent advisory circulars and handbooks.

• Private pilot applicants are evaluated in all tasks of each area of operation.
• Flight instructor applicants are evaluated on one or more tasks in each area of operation, with certain required tasks identified by notes immediately following the area of operation titles.

Because an evaluator may cover every task, the handbook instructs that the instructor evaluate all of the tasks before recommending a pilot or maintenance applicant for the practical test. That evaluation is "not necessarily formal," but it should adhere to criterion-referenced testing — measured against the published standard, not against how the learner compares to last month's student.

---

Putting It Together: A Worked Flight-Training Example

Lesson: short-field takeoff and maximum performance climb, private pilot level.

---

FOI Exam Traps & Common Misconceptions

• Trap 1 (order of the four steps): Level-of-learning objectives → indicators of desired behavior → criterion objectives → criterion-referenced items. The pattern is general to specific; any option that starts with writing test questions is reversed.
• Trap 2 (criterion-referenced vs. norm-referenced): Criterion-referenced means measured against a published standard, with the goal that all successful applicants meet it. If an option describes ranking learners against each other or grading on a curve, it is describing norm-referenced testing and is wrong for the ACS/PTS.
• Trap 3 (when authentic assessment works): Authentic and collaborative assessment are less useful in the early phases of training; the learner can participate fully once they reach the application and correlation levels.
• Trap 4 (you already wrote the criterion objectives): If the stem mentions that performance-based objectives exist in the lesson plans, the keyed answer is that the criterion objectives have already been formulated.
• Trap 5 (who evaluates which tasks): Private pilot applicants are evaluated in all tasks of each area of operation; flight instructor applicants are evaluated on one or more tasks in each area of operation. Either way, the recommending instructor should evaluate all tasks first, because the evaluator may cover every one.