先看這個問題為什麼值得解Why this problem is worth solving
情境先行Start with the situation
影片問題可由字幕猜中時，高分不代表模型看懂動作；它可能只讀到了語言捷徑。 When video questions can be guessed from subtitles, a high score does not prove action understanding; the model may exploit a language shortcut.
沒有反事實測試，modality contribution 很容易被 dataset shortcut 取代。 Without counterfactual tests, modality contribution is easily replaced by dataset shortcuts.
先備知識Prerequisite
- 知道編碼器（encoder）與表徵的角色 Know the role of encoders and representations
讀完你能做到After this path
- 畫出 modality 進入與融合的位置 Map where modalities enter and fuse
- 設計 modality ablation 與 counterfactual Design modality ablations and counterfactuals
- 區分 recognition、grounding 與 action Distinguish recognition, grounding, and action
閱讀框架Concept frame
- 核心問題Question
- 模型回答對了，它真的使用了另一種模態嗎？ If a model answers correctly, did it actually use the other modality?
- 對照軸Comparison axis
- 跨模態資訊的互補程度 Cross-modal complementarity
- 判讀方式Evidence to inspect
- 並排閱讀三種相對情境，再回到論文檢查消融實驗、反事實測試與適用範圍。Read three relative scenarios side by side, then inspect the paper's ablations, counterfactual tests, and scope statements.
回答後查看解析Answer, then view the explanation
你的預測Make a prediction
哪個測試最能檢查模型是否真的使用影像？ Which test best checks whether a model actually uses the image?
看現象，不急著講故事Observe before narrating
三個診斷訊號Three diagnostic signals
這些是待檢查的診斷假說。下一步用條件比較、消融實驗或反事實測試排除其他解釋。These are diagnostic hypotheses to test. Use condition comparisons, ablations, or counterfactual tests to rule out alternatives.
一次看完三種情境Read three scenarios side by side
三種情境對照Three-scenario comparison
資訊越互補，early fusion 一定越好嗎？ As modalities become more complementary, is early fusion always better?
對照軸：Comparison axis: 跨模態資訊的互補程度 Cross-modal complementarity
應先證明額外 modality 不是冗餘或捷徑。 First show the added modality is neither redundant nor a shortcut.
fusion 位置與缺失 modality robustness 值得比較。 Compare fusion points and missing-modality robustness.
關鍵問題轉向對齊、時間同步與 credit assignment。 Alignment, synchronization, and credit assignment become central.
從直覺到機制From intuition to mechanism
把機制講清楚Explain the mechanism
多模態像多位證人：重複同一句話與提供互補線索，是兩種不同價值。 Modalities are like witnesses: repeating one account and providing complementary clues have different value.
輸入是多個同步或非同步訊號；輸出是預測、文字或動作；目標是利用互補證據完成對應與決策。 Input: multiple synchronized or asynchronous signals. Output: predictions, text, or actions. Objective: use complementary evidence for grounding and decisions.
- 01 各 modality sensing 與 tokenization。 Sense and tokenize each modality.
- 02 在特定層級對齊或融合 representation。 Align or fuse representations at a chosen layer.
- 03 以任務 loss 把跨模態 evidence 傳到輸出。 Propagate cross-modal evidence to outputs through task losses.
把 co-occurrence 當 grounding，把 benchmark accuracy 當 modality use。 Treating co-occurrence as grounding and benchmark accuracy as proof of modality use.
好的多模態證據包含移除、替換、錯位與缺失 modality 測試。 Good multimodal evidence includes removal, replacement, misalignment, and missing-modality tests.
不用回看頁面的自我檢查Check your recall without looking back
三個回想題Three recall questions
先口頭回答，再展開解析。Answer aloud before opening each explanation.
01 什麼是 modality shortcut？ What is a modality shortcut?
模型用較容易但非目標的訊號完成任務。 The model solves a task with an easier, unintended signal.
02 counterfactual replacement 測什麼？ What does counterfactual replacement test?
只改變關鍵 modality 證據時，輸出是否以合理方向改變。 Whether outputs change appropriately when only decisive modal evidence changes.
03 fusion ablation 應固定什麼？ What should a fusion ablation hold fixed?
資料、encoder 容量、訓練預算與 downstream metric。 Data, encoder capacity, training budget, and downstream metrics.
把工具帶回你的研究Take the tool back to your research
最後一個任務One final task
為一篇多模態論文設計『移除、替換、錯位』三個最小 counterfactual。 Design minimal removal, replacement, and misalignment counterfactuals for one multimodal paper.