先看這個問題為什麼值得解Why this problem is worth solving
情境先行Start with the situation
一個醫療模型在固定測試集提高 2 分，但醫院換了儀器與病患族群。那 2 分還回答同一個問題嗎？ A medical model gains two points on a fixed test set, but the hospital changes scanners and patient mix. Do the two points answer the same question?
錯把 proxy 當目標，會讓精確的數字支持錯誤的決策。 Mistaking a proxy for the target lets precise numbers support the wrong decision.
先備知識Prerequisite
- 理解平均值與抽樣變異 Understand averages and sampling variability
讀完你能做到After this path
- 把抽象主張拆成構念（construct）與測量方式 Separate an abstract construct from its measurement
- 辨識評測資料污染與分布偏移 Identify benchmark contamination and distribution shift
- 用不確定性改寫過度確定的結論 Rewrite overconfident conclusions using uncertainty
閱讀框架Concept frame
- 核心問題Question
- 一個分數穩定提高，何時仍不能支持『系統更安全』？ When does a stable score gain still fail to support the claim that a system is safer?
- 對照軸Comparison axis
- 部署環境與測試集的差距 Deployment-to-test distribution gap
- 判讀方式Evidence to inspect
- 並排閱讀三種相對情境，再回到論文檢查消融實驗、反事實測試與適用範圍。Read three relative scenarios side by side, then inspect the paper's ablations, counterfactual tests, and scope statements.
回答後查看解析Answer, then view the explanation
你的預測Make a prediction
看到 benchmark 提升後，哪個問題最優先？ After seeing a benchmark gain, which question comes first?
看現象，不急著講故事Observe before narrating
三個診斷訊號Three diagnostic signals
這些是待檢查的診斷假說。下一步用條件比較、消融實驗或反事實測試排除其他解釋。These are diagnostic hypotheses to test. Use condition comparisons, ablations, or counterfactual tests to rule out alternatives.
一次看完三種情境Read three scenarios side by side
三種情境對照Three-scenario comparison
哪個結論在 shift 增加時必須先降級？ Which conclusion should be downgraded first as shift increases?
對照軸：Comparison axis: 部署環境與測試集的差距 Deployment-to-test distribution gap
內部效度較容易解讀，但仍需檢查污染。 Internal validity is easier to read, though contamination remains a concern.
應分層報告並做 sensitivity analysis。 Use stratified reporting and sensitivity analyses.
排行榜結論不能直接外推；需要新場域資料。 Leaderboard claims should not be extrapolated without in-domain evidence.
從直覺到機制From intuition to mechanism
把機制講清楚Explain the mechanism
Benchmark 是一支尺；先知道它刻了什麼，才知道 2 公分代表什麼。 A benchmark is a ruler; understand its markings before interpreting a two-centimeter gain.
輸入是系統輸出與評測樣本；輸出是分數與不確定性；目標是支持特定使用情境下的比較。 Input: system outputs and evaluation samples. Output: scores and uncertainty. Objective: support a comparison for a specified use context.
- 01 定義 construct 與使用情境。 Define the construct and use context.
- 02 檢查題目、標註與 metric 如何 operationalize 它。 Audit how items, labels, and metrics operationalize it.
- 03 測試 perturbation、subgroup、shift 與 uncertainty。 Test perturbations, subgroups, shifts, and uncertainty.
把統計顯著、實務重要與因果改善視為同一件事。 Conflating statistical significance, practical importance, and causal improvement.
可信評測要交代『量什麼、在哪裡量、誤差多大、能外推多遠』。 Trustworthy evaluation states what is measured, where, with how much error, and how far it transfers.
不用回看頁面的自我檢查Check your recall without looking back
三個回想題Three recall questions
先口頭回答，再展開解析。Answer aloud before opening each explanation.
01 construct 與 metric 有何不同？ How does a construct differ from a metric?
construct 是想理解的能力或風險；metric 是它的具體測量規則。 A construct is the capability or risk of interest; a metric is a concrete measurement rule.
02 為何 subgroup 報告重要？ Why report subgroups?
平均值可能掩蓋不同族群方向相反的變化。 Averages can hide changes in opposite directions across groups.
03 何時不能外推 benchmark 結果？ When should benchmark results not be extrapolated?
當 deployment 的資料、成本或錯誤後果超出評測支援範圍。 When deployment data, costs, or failure consequences fall outside the evaluation's support.
把工具帶回你的研究Take the tool back to your research
最後一個任務One final task
選一篇宣稱『更 robust』的論文，寫下 construct、perturbation family 與未覆蓋的 shift。 Choose a paper claiming greater robustness and write down its construct, perturbation family, and uncovered shifts.