📚 Evidence Frameworks for EdTech Pilots and Official Conclusions | 教育科技试点的证据框架与官方结论
Education technology is now a permanent line in almost every school budget, yet the decision to buy a platform, a device fleet or an adaptive tutoring licence is still frequently justified by a polished demonstration rather than by decision-grade evidence. A pilot is only useful if it is designed to answer a specific question with a pre-specified comparison, outcome and decision rule. This article sets out the evidence frameworks that regulators, research foundations and ministries actually use, summarises the official conclusions they have reached, and offers a practical checklist for judging claims made by vendors.
教育科技如今几乎是每所学校预算中的固定科目，但购买平台、设备或自适应辅导授权这样的决定，往往仍由一场漂亮的演示而非可用于决策的证据来支撑。只有当试点围绕一个明确的问题，并预先设定好对照组、结果指标与决策规则时，它才真正有用。本文将梳理监管机构、研究基金会与教育主管部门实际采用的证据框架，总结它们已经得出的官方结论，并提供一套判断厂商宣传的实用清单。
1. Why EdTech Pilots Need an Evidence Framework | 为什么教育科技试点需要证据框架
A pilot without a framework produces anecdotes that can be quoted by anyone to support any conclusion. An evidence framework forces three commitments before the pilot begins: what claim is being tested, what the counterfactual is, and what result would trigger a decision. Without these, schools accumulate enthusiasm rather than knowledge, and the same tool is re-piloted every three years by a new head of department.
没有框架的试点只会产生轶事，而轶事可以被任何人引用以支持任何结论。证据框架要求在试点开始前作出三项承诺：要检验什么主张、反事实是什么、什么样的结果会触发决策。缺少这些，学校积累的是热情而不是知识，同一个工具每隔三年就会被新的学科负责人重新试点一次。
A minimal framework should specify:
一个最小可行的框架应当明确：
- the causal claim (for example, 'the platform raises Year 9 algebra attainment') | 因果主张（例如"该平台能提高九年级代数成绩"）；
- the comparison condition (business as usual, an active alternative, or a waitlist) | 对照条件（常态教学、另一种有效干预，或等待名单）；
- the primary outcome and the instrument used to measure it | 主要结果指标及其测量工具；
- the minimum effect that would justify the cost | 能够证明成本合理的最小效应；
- the decision rule and the person accountable for acting on it | 决策规则以及负责据此行动的人。
2. Defining the Counterfactual: Designs That Support Causal Claims | 界定反事实：能支持因果推论的研究设计
The counterfactual is what would have happened to the same pupils without the intervention. Only designs that construct a credible counterfactual can support causal language. Randomised controlled trials assign pupils, classes or schools at random; cluster randomisation is usually necessary in education because pupils share teachers and classrooms, and contamination across arms is otherwise severe.
反事实是指同一批学生在没有该干预的情况下会发生什么。只有能构建可信反事实的设计，才能支持因果性的表述。随机对照试验将学生、班级或学校随机分配；在教育场景中通常必须采用整群随机化，因为学生共享教师与教室，否则组间污染会非常严重。
Where randomisation is impossible, quasi-experimental designs such as regression discontinuity, difference-in-differences and matched comparison groups can be used, provided the underlying assumptions are tested and reported. Large research funders, including the Education Endowment Foundation, typically fund three stages: an efficacy trial under ideal conditions with developer involvement, an effectiveness trial under realistic conditions with independent evaluation, and a scale-up trial to test delivery at system level.
当无法随机化时，可使用断点回归、双重差分与匹配对照组等准实验设计，但前提是其假设经过检验并被如实报告。包括教育捐赠基金会（EEF）在内的主要研究资助方通常资助三个阶段：由开发者参与、在理想条件下进行的效能试验；由独立团队评估、在真实条件下进行的实效试验；以及在系统层面检验实施的可扩展试验。
3. Logic Models and Theory of Change | 逻辑模型与变革理论
A logic model maps inputs to activities, outputs, outcomes and longer-term impact. Its purpose is to expose the mechanism: is the active ingredient immediate corrective feedback, distributed practice, reduced teacher workload, or simply more time on task? If the theory of change is not stated, a null result cannot be interpreted, because nobody knows which link in the chain failed.
逻辑模型把投入、活动、产出、结果与长期影响串联起来，其目的是暴露作用机制：真正的活性成分是即时纠错反馈、分布式练习、教师负担的减轻，还是单纯增加了任务时间？如果变革理论没有被写明，零结果就无法解释，因为没人知道是链条上的哪一环失效了。
Many commercial products bundle several mechanisms together: content delivery, adaptive sequencing, gamified motivation, parental messaging and automated marking. When such a bundle fails, the honest conclusion is that the bundle failed, not that 'technology does not work', and the useful next step is to test the components separately.
许多商业产品把多种机制捆绑在一起：内容呈现、自适应排序、游戏化激励、家长短信与自动批改。当这样的组合失败时，诚实的结论是"这个组合失败了"，而不是"技术没有用"；更有价值的下一步是把各成分分开检验。
4. Outcome Measures, Engagement and Implementation Fidelity | 结果测量、参与度与实施保真度
Engagement data are not learning outcomes. Logins, minutes on task and completion rates measure exposure to the product, not changes in what pupils know or can do. A pilot that reports only usage metrics has measured marketing performance, not educational impact. Outcomes should be partly distal, using independent standardised or curriculum-aligned assessments administered under controlled conditions.
参与度数据不等于学习结果。登录次数、在线时长与完成率衡量的是对产品的接触程度，而不是学生知识与能力的变化。只报告使用数据的试点，测量的是营销表现而非教育影响。结果指标应部分具有远端性，使用在受控条件下实施的独立标准化测验或与课程对齐的测评。
Implementation fidelity is equally important, and should be reported alongside outcomes: dosage actually delivered, adherence to the intended model, quality of delivery, participant responsiveness and documented adaptations. Without fidelity data, a null result is ambiguous: the intervention may be ineffective, or it may never have been delivered as designed.
实施保真度同样重要，且应与结果一并报告：实际交付的剂量、对原定模式的遵循度、交付质量、参与者响应度以及有记录的本地化调整。缺少保真度数据时，零结果是含糊的：干预可能本身无效，也可能从未按设计交付过。
| Outcome type | 结果类型 | Example metric | 示例指标 | What it can support | 可支持的结论 |
|---|---|---|
| Proximal skill | 近端技能 | Weekly quiz score on the taught topic | 当周所学主题的小测成绩 | Short-term mastery only | 仅能支持短期掌握 |
| Distal attainment | 远端学业成就 | Standardised test at end of year | 学年末标准化测验 | Transferable learning claims | 可支持可迁移的学习主张 |
| Behavioural | 行为类 | Homework submission rate | 作业提交率 | Intermediate mechanism only | 仅能支持中间机制 |
| Usage | 使用类 | Logins, minutes, modules completed | 登录、时长、完成模块数 | Implementation only, not impact | 仅支持实施情况，不支持影响 |
5. Effect Sizes, Minimum Detectable Effects and Statistical Power | 效应量、最小可检测效应与统计功效
Effect sizes standardise the difference between groups so that results from different trials can be compared. The most common form for continuous outcomes is:
效应量把组间差异标准化，以便不同试验的结果可以比较。连续结果最常用的形式是：
d = (Mₜ − Mc) ÷ SD_pooled
Cohen's conventional benchmarks of 0.20, 0.50 and 0.80 were designed for psychological laboratory research and are widely misapplied in education. Analyses of large samples of education trials suggest median effects closer to 0.10 overall, and near 0.05 for standardised tests in large-scale randomised evaluations. An effect of 0.05 can still matter at system scale, but it requires large samples to detect.
科恩提出的 0.20、0.50、0.80 基准是为心理学实验室研究设计的，在教育领域被广泛误用。对大量教育试验的分析显示，总体中位效应更接近 0.10，而在大规模随机评价中使用标准化测验时接近 0.05。0.05 的效应在系统层面仍可能重要，但需要很大的样本才能检测到。
Because education trials randomise clusters, the required sample is inflated by the design effect:
由于教育试验按整群随机化，所需样本量会被设计效应放大：
DE = 1 + (m − 1)ρ
where m is the average cluster size and ρ is the intraclass correlation. School-level ICCs of roughly 0.10 to 0.20 are common in attainment data. The minimum detectable effect follows approximately:
其中 m 是平均群规模，ρ 是组内相关系数。在学业成绩数据中，学校层面的 ICC 通常在 0.10 至 0.20 左右。最小可检测效应近似为：
MDES ≈ 2.8 × SD × √1 ÷ nₜ + 1 ÷ nc
effects. A trial with 60 pupils per arm has very little chance of detecting d = 0.10, so a 'no significant difference' verdict from such a pilot is uninformative rather than negative.
这意味着小型试点通常无力检测真实存在的效应。每组 60 名学生的试验几乎不可能检测到 d = 0.10，因此这类试点得出的"无显著差异"结论是没有信息量的，而不是负面的。
6. Common Threats to Validity in School-Based Trials | 校本试验中常见的效度威胁
Most failures of EdTech evidence are design failures rather than statistical ones. Attrition is the most serious: if pupils who disengage also fail to sit the post-test, the remaining sample is no longer comparable. Contamination occurs when control teachers adopt the tool or share resources, diluting the contrast between arms.
教育科技证据的失败大多源于设计而非统计。最严重的问题是流失：如果退出干预的学生同时缺席后测，剩余样本就不再可比。污染则发生在对照组的教师也使用了该工具或共享资源时，这会稀释两组之间的对比。
Other recurring threats include:
其他反复出现的威胁包括：
- novelty and Hawthorne effects, where short-term gains reflect attention rather than the tool | 新奇效应与霍桑效应，短期增益来自被关注而非工具本身；
- teaching to the test when the outcome is the vendor's own assessment | 当结果指标是厂商自带的测评时出现应试化；
- multiple comparisons and outcome switching, producing spurious 'significant' subgroup findings | 多重比较与结果更换，产生虚假的"显著"子组发现；
- publication and reporting bias, since null trials are less likely to be published | 发表与报告偏倚，因为零结果试验更不容易发表；
- conflicts of interest where the evaluator is funded by the provider | 评估方由供应商资助所产生的利益冲突；
- Campbell's law, where a metric used for high-stakes decisions ceases to be a good measure | 坎贝尔定律：一旦指标被用于高利害
Published by TutorHao | Exam Prep Revision Series | aleveler.com
广告 / Sponsored英文原版 Oxford IB Diploma Programme IB Prepared Mathematics An…¥296到手价 · 以淘宝页面为准数学 备考教辅 · 正版现货 · 学习备考通用查看详情 / View on Taobao广告 / Sponsored英文原版 New Cambridge IGCSE & O Level Essential Biology Workbo…¥95到手价 · 以淘宝页面为准生物 备考教辅 · 正版现货 · 学习备考通用查看详情 / View on Taobao更多咨询请联系16621398022（同微信）
备考资讯每日更新，关注不错过考纲变化 · 科目分析 · 学习方法
在微信里长按二维码识别关注
或截图后用「扫一扫 → 相册」选择本图公众号：tutorhao
