Reliability, Validity, and Traceability

Learning outcomes

After studying this subject, the learner will be able to:

• define reliability, validity, and traceability independently;
• explain why no single axis guarantees decision quality;
• detect consistent measurement of the wrong construct;
• detect a suitable judgment that cannot be reconstructed or defended;
• design a three-axis examination for a judge, card, or panel; and
• select development suited to the defect rather than one treatment for every problem.

Three axes of decision quality

An adjudicative decision requires three different conditions. Reliability asks whether the source produces a stable judgment when conditions are comparable. Validity asks whether the tool and judgment measure the property the competition claims to measure. Traceability asks whether the route to the outcome can be reconstructed.

The intersection symbol means that quality requires the axes to meet; it does not mechanically add their scores. A judgment may be reliable but invalid, valid in direction but unstable, or strong on both axes with an incomplete record.

Reliability

Reliability is the stability of a source or tool in producing similar results under comparable conditions. It includes:

• Within-judge reliability: similarity across a hidden repeat.
• Between-judge reliability: agreement after decision independence is verified.
• Temporal reliability: performance stability across comparable sessions or days.
• Procedural reliability: stability when the same protocol is applied.

Difference does not always indicate low reliability. Samples or context may have changed. D, C, and T are examined before an indicator is calculated.

Validity

Validity means that measurement represents its intended construct. A competition may claim to assess execution quality while its card awards most points to story, price, or prestigious ingredients. Even perfect panel agreement would not make that measurement valid.

CAMS uses three validity questions:

• Construct validity: Does the variable represent the intended concept?
• Content validity: Does the card cover necessary elements without inflating one and neglecting another?
• Decision validity: Is the final inference proportionate to the data and its limits?

Professional disagreement does not automatically remove validity. The relationship among observation, standard, and outcome must remain clear.

Traceability

Traceability is the capacity to know who recorded a value, for which sample, at what time and context, how it changed, and who authorized it. An outcome may be correct by coincidence while remaining untraceable. The institution would then be unable to prove fairness or reproduce success.

Traceability includes the CJE identifier, original state, modification history, formula version, authorized role, and institutional verification link.

Three-axis defect patterns

Every pattern requires a different response. Low reliability requires calibration, rest, or repetition. Low validity requires revision of definition, card, or interpretation. Weak traceability requires repair of system, record, and permissions.

Mind map

Worked case

A panel used an “innovation” criterion and consistently awarded high scores to unfamiliar samples. Hidden-repeat scores were close and the system accurately retained every event. Reliability and traceability appeared high.

Review of the definition showed that the card did not distinguish useful novelty from strangeness. An unfamiliar but poorly integrated dish received 91, while a dish that effectively developed a known technique received 76. The panel was consistent, but the variable did not represent professional innovation as declared.

The response is not training judges to agree; they already agree. Innovation must be redefined through added value, technical function, integration with dish identity, and improvement of experience. Calibration cases then teach the difference between novelty and strangeness.

Applied laboratory: three-axis audit

Purpose

Evaluate a card, judge, and panel across all axes and select development specific to each defect.

Procedure

• Receive two dish-evaluation cards, repeat records for two judges, and a digital audit history.
• Identify the construct each field claims to measure.
• Examine repetition, agreement, and independence.
• Link outcomes to primary data and transformations.
• Rate each axis as strong, acceptable, or requiring treatment.
• Write a development plan separating source training, tool redesign, and record repair.

Three-axis examination card

Practical assessment

The learner receives five short cases, identifies the damaged axis, provides evidence, selects corrective action, and explains why greater agreement alone is insufficient.

Assessment rubric

Core terms

• Reliability: measurement stability under comparable conditions.
• Validity: representation of the construct a measure claims to assess.
• Construct validity: sound relation between variable and concept.
• Content validity: sufficient coverage of domain elements.
• Decision validity: proportionality of inference to data.
• Traceability: capacity to reconstruct outcome origin and path.

Scientific conclusion

Agreement alone does not create science. A correct definition does not guarantee stability, and a complete record does not make the wrong measure valid. CAMS gives every axis its function and requires them to meet in professional decisions. The institution can then distinguish what must be trained, redesigned, or documented.