EduEvidenceDataLab [Light]数据来源：真实文献 · 人工精编

我们准备在大学一年级 C 语言课程中允许学生使用生成式 AI 编程助手。它到底会不会提高学习效果？应该怎样引入？

模式：平台原生 · 生成时间：2026-08-24 · 证据 12 条 · 来源 8 个

先看结论

该不该做、置信度多高、最关键的证据边界在哪。

建议决策 试点验证 置信度 · 中

任务表现的正面证据 + 有据可查的无护栏风险 + 混合的质量/可用性信号 + 大学层面学习证据缺失 → 有界、护栏化、带评估的试点，而非全面采用。

最强支持结论

AI 编程助手在训练期稳定提升练习效率：69 名新手的随机对照中完成率 1.15 倍、用时 0.57 倍。

关键不确定性 / 反例

缺少大学层面的直接学习证据；唯一大规模试验显示，无护栏使用 GPT-4 的学生独立考试成绩下降 17%。

主要风险

无护栏使用会抬高练习表现却压低独立考试表现，而学习者往往意识不到这一落差。

下一步

开展分阶段 CS1 试点：给提示而非答案、每周实验课使用、并以无 AI 迁移考试作为可叫停的验收条件。

证据 / 来源 · 12 / 8

任务表现 ≠ 学习效果

只展示真正有解释力的结果分离；正向、负向与零效应按 effect_direction 编码。

结果分离 · 任务表现 ≠ 学习效果

将不同结果类型分开裁决，避免把训练时更快、更高分直接等同于真正学会。

效应方向来自 evidence.effect_direction；“支持某个主张”不等于“结果是正向”。

任务 / 近端表现

完成时间正向效应 2
代码质量零效应 1
作业成绩正向效应 2

学习 / 保持 / 迁移

知识获得正向效应 1
记忆保持零效应 1
独立问题解决负向效应 1零效应 2

风险 / 依赖

过度依赖负向效应 1

其他结果

元认知正向效应 1

这意味着什么：各 Outcome 的正向 / 负向 / 零效应证据条数对比。这里展示的是 effect_direction，不是“这条证据是否支持某个主张”。

证据裁决

支持、不确定、被反驳与缺失证据分开放置，不把长段落平铺在同一层。

决策：试点验证 · 置信度：中

可以主张

AI 编程助手在训练期间可靠地提升任务表现（完成速度、正确性）—— E-001、E-006。
E-001E-006
无护栏的生成式 AI 访问在移除工具后可能损害独立问题解决能力 —— E-004。
E-004
护栏设计（给提示而非给答案）能大幅缓解负面学习效应 —— E-005。
E-005

查看其余 4 条

任务表现提升并不自动等于学习提升 —— E-004 与 E-006 的研究内对照。
E-004E-006
工具能力可观：Codex 能解出约半数至四分之三的 CS1 考试风格题目 —— E-010。
E-010
职业开发者 RCT 显示 Copilot 带来约 55% 任务提速；但职业人群限制直接性 —— E-008。
E-008
LLM 代码讲解的质量评级与学生自撰讲解相当，可作支架材料 —— E-011。
E-011

尚不能主张

AI 编程助手能否真正改善或保持大学新手的编程学习——本证据集中没有大学层面的直接 RCT [无直接证据]
Kazemitabaar 2023 的一周中性保持性能否延伸到一个学期 —— E-003。
E-003
基准质量结论（E-009）与讲解质量评级（E-011）能否转化为课堂学习收益。
E-009E-011

查看其余 1 条

可用性研究所记录的理解/所有权困难（E-012）在整学期护栏条件下会如何演变。
E-012

被反驳的主张

'AI 工具总能提高学习'被 E-004 反驳（无护栏访问，独立考试 −17%）。
E-004
'速度收益等于学习收益'被 E-001/E-006/E-008 与 E-004 之间的任务-学习分离所反驳。
E-001E-006E-008E-004

缺失证据

在大学编程课程中带保持与无 AI 迁移测试的 RCT。
同一课程内变化 AI 使用政策的研究。
跨越一门课的 AI 依赖纵向数据。

查看其余 1 条

职业提速 RCT 的同行评审重复（Peng 等仍为预印本）。

从证据到行动

适用性、护栏、停止条件与评价连成一条可执行路径。

证据

AI 编程助手在训练期间可靠地提升任务表现（完成速度、正确性）—— E-001、E-006。

适用性

在大一 C 课程以护栏化使用政策开展试点

决策

试点验证

护栏

AI 使用分三档明确分级（解释 / 协作 / 无 AI 迁移）。照抄未审视的 AI 输出属学术诚信违规，并通过推理痕迹要求核查。

停止条件

迁移测验成绩显著低于基线同届预期; 推理痕迹中出现普遍诚信违规; 风险指标中 AI 依赖信号超阈值; 助教/教师工作量不可持续

评价

实验班独立问题解决非劣（差异在 5% 以内）且保持相当或更优、AI 依赖指数低于阈值；若独立问题解决下滑超过 10%，无论任务收益如何，试点均判为失败。

Lieflat 实证手作画廊

AI 按数据形状从 Lieflat 目录选型编排；每张图的数字都可溯源到 result.json。

发表年份 × 结果维度文献年历

气泡面积 ∝ 该格研究数 · 实心圆 = 有显著结果

仅当证据集携带发表年份与结果维度时绘制。

L9 Bubble Almanac · Evidence.year X Dimension

年份 × 结果维度证据密度

每格数字 = 该年份该结果维度的证据条数

当证据跨多个年份与结果维度时，展示研究密度的分布。

L16 Matrix Heat · Evidence.year X Outcome Counts

各结果类型效应方向分布

每 1 个圆点 = 1 条证据 · 绿 = 正向 · 灰 = 零效应 · 橙 = 负向

基于 effect_direction 计数，全部数值来自 result.json。

F5 Tick Rows · Outcomes.direction Counts

各结果类型的正负证据对照

左右两列分别汇总正向与负向证据条数

当同一结果同时存在正向与负向证据时，分列呈现避免相互抵消。

F6 Paired Rungs · Outcomes.paired Counts

各结果类型的净效应倾向

位置 =（正向 − 负向）÷ 方向计数 · 中点为中性

双极展示各结果构念整体偏向支持还是反对。

L7 Brand Spectrum · Outcomes.bipolar Axes

研究设计构成

每格 = 1 篇研究 · 显示证据来自哪些研究设计

当证据包含多种研究设计时，构成图比表格更快暴露设计偏斜。

L14 Hundred Field · Evidence.study Type Composition

关键来源

摘要页只列最关键的来源；完整溯源在完整报告中展开。

S-2023-kazemitabaar

Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming

T1 DOI 可验证论文 · 2023

S-2025-bastani

Generative AI without guardrails can harm learning: Evidence from high school mathematics

T1 DOI 可验证论文 · 2025

S-2024-marzuki

Impact of ChatGPT on ESL students' academic writing skills

T1 DOI 可验证论文 · 2024

S-2023-peng

The Impact of AI on Developer Productivity: Evidence from GitHub Copilot

Tier2 Academic Database · 2023

完整报告中还有 4 个来源可展开追溯。

完整报告

结论前置：全部可追溯证据与方法学细节都在这里，关键论证位置穿插有意义的可视化，每个数字都能回查到 result.json。

01 结论、裁决与研究边界

先明确最终裁决与研究边界，再解释为什么。

建议决策 试点验证 置信度 · 中

任务表现的正面证据 + 有据可查的无护栏风险 + 混合的质量/可用性信号 + 大学层面学习证据缺失 → 有界、护栏化、带评估的试点，而非全面采用。

最强支持结论

AI 编程助手在训练期稳定提升练习效率：69 名新手的随机对照中完成率 1.15 倍、用时 0.57 倍。

关键不确定性 / 反例

缺少大学层面的直接学习证据；唯一大规模试验显示，无护栏使用 GPT-4 的学生独立考试成绩下降 17%。

主要风险

无护栏使用会抬高练习表现却压低独立考试表现，而学习者往往意识不到这一落差。

下一步

开展分阶段 CS1 试点：给提示而非答案、每周实验课使用、并以无 AI 迁移考试作为可叫停的验收条件。

证据 / 来源 · 12 / 8

研究问题

我们准备在大学一年级 C 语言课程中允许学生使用生成式 AI 编程助手。它到底会不会提高学习效果？应该怎样引入？

目标学习者

教育阶段：大学一年级；专业：计算机科学与技术；先验知识：首次程序设计课程，无文本编程基础；学习者特征：混合能力大班（60 人）

课程情境

课程：C 语言程序设计；课程类型：讲授 + 实验课；课程周期：16 周（一学期）

AI 干预

干预方式：讲授 + 实验练习；AI 工具：生成式 AI 编程助手；允许使用：设计中（待证据评审）；使用频率：每周实验课；干预周期：一学期

比较条件

No AI Coding Assistant Control

结果构念

主要结果：独立问题解决、代码质量；次要结果：完成时间、记忆保持、知识获得；风险结果：AI 依赖、过度依赖、迁移下降

研究范围

时间范围：2021 2026；地域：Worldwide；研究设计：随机对照试验、准实验、观察性研究

决策成功条件

independent problem solving and code quality improve (or do not decline) while AI dependency risk stays controlled; evidence base supports a bounded pilot.

02 关键证据与结果分离

把任务表现、真实学习、保持与风险放在同一证据地图中，但不混为一谈。

纳入标准

Studies Of Generative AI Coding Tools In Learning To Program
Outcomes Measuring Learning Not Only Task Speed
University Or Novice Programming Populations

排除标准

Practitioner Anecdotes Without Data
Industry Professional Populations Only

证据来源覆盖

S-2023-kazemitabaar S-2025-bastani S-2024-marzuki S-2023-peng S-2023-yetistiren S-2022-finnie-ansley S-2023-explanations-compare S-2022-vaithilingam

当前报告只展示 result 中真实存在的检索与来源信息；没有流程计数时不伪造 PRISMA / funnel 数字。

结果分离 · 任务表现 ≠ 学习效果

将不同结果类型分开裁决，避免把训练时更快、更高分直接等同于真正学会。

效应方向来自 evidence.effect_direction；“支持某个主张”不等于“结果是正向”。

任务 / 近端表现

完成时间正向效应 2
代码质量零效应 1
作业成绩正向效应 2

学习 / 保持 / 迁移

知识获得正向效应 1
记忆保持零效应 1
独立问题解决负向效应 1零效应 2

风险 / 依赖

过度依赖负向效应 1

其他结果

元认知正向效应 1

结果类型	正向效应	负向效应	零效应	证据
知识获得knowledge_gain	1	0	0	`E-007`
记忆保持retention	0	0	1	`E-003`
独立问题解决independent_problem_solving	0	1	2	`E-002` `E-004` `E-005`
完成时间completion_time	2	0	0	`E-001` `E-008`
代码质量code_quality	0	0	1	`E-009`
作业成绩assignment_score	2	0	0	`E-006` `E-010`
元认知metacognition	1	0	0	`E-011`
过度依赖over_reliance	0	1	0	`E-012`

这意味着什么：各 Outcome 的正向 / 负向 / 零效应证据条数对比。这里展示的是 effect_direction，不是“这条证据是否支持某个主张”。

图 1. 各结果类型的正向 / 负向 / 零效应证据数量（基于 effect_direction，不等同于 Claim 是否被支持）。

ID	结果	效应	质量	主张	来源
`E-001`	完成时间	正向效应与主张关系: 支持	9	AI 编程助手在训练期间显著提升任务完成速度与完成率（完成率 1.15 倍、用时 0.57 倍、正确率 1.8 倍）。查看完整证据研究 ID STUDY-KAZEMITABAAR-2023 样本 ID SMPL-KAZEMITABAAR-2023-N69 研究标题 Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming 来源标题 Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming 年份 2023 研究设计随机对照试验教育阶段 K-12 Ages 10 17 研究人群 69 名 10-17 岁编程新手，此前无文本编程经验样本量 69 干预三臂 RCT：GPT Base（无护栏标准 ChatGPT 式界面）与 GPT Tutor（护栏版，教师设计提示、不给直接答案）用于数学练习对照 / 比较条件对照组（无 AI 传统教学）结果测量 Code Authoring Task Progress And Time 效应 / 结果 1.15x completion rate, 0.57x time, 1.8x correctness 效应方向正向效应与主张关系支持干预时长 3 Weeks Training 方法 controlled experiment with random assignment, immediate post-test and 1-week retention test 优势 randomized_controlled_design；immediate_post_test_and_retention_test；code_modification_task_guard 局限 non_university_population_ages_10_17；small_sample_69；self-paced environment differs from classroom 混杂因素 Prior Programming Competency Interaction 质量维度 D1 研究设计=2 · D2 样本质量=2 · D3 测量效度=2 · D4 时间强度=2 · D5 直接性=1 质量分 9.0 证据等级 strong 适用性 learner_match=partial_novice_programmers_but_younger · subject_match=introductory_programming · tool_match=codex_like_generative_ai · scope=task_performance_during_training 置信度 0.7 证据状态已支持完整主张 AI 编程助手在训练期间显著提升任务完成速度与完成率（完成率 1.15 倍、用时 0.57 倍、正确率 1.8 倍）。来源位置 https://dl.acm.org/doi/10.1145/3544548.3580919 可验证链接 https://dl.acm.org/doi/10.1145/3544548.3580919	`S-2023-kazemitabaar`
`E-002`	独立问题解决	零效应与主张关系: 中性	7	可使用 AI 生成代码并未降低学生在人工代码修改任务上的表现（组间差异不显著）。查看完整证据研究 ID STUDY-KAZEMITABAAR-2023 样本 ID SMPL-KAZEMITABAAR-2023-N69 研究标题 Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming 来源标题 Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming 年份 2023 研究设计随机对照试验教育阶段 K-12 Ages 10 17 研究人群 69 名 10-17 岁编程新手，此前无文本编程经验样本量 69 干预无护栏 GPT Base（类标准 ChatGPT 界面）课内练习；移除访问后参加独立考试对照 / 比较条件对照组（从未使用 AI）结果测量 manual code-modification tasks during training 效应 / 结果 no significant difference between groups 效应方向零效应与主张关系中性干预时长 3 Weeks Training 方法 controlled experiment, code-modification task followed each code-authoring task 优势 direct_test_of_transfer-adjacent_skill；same_session_measurement 局限 code modification is not full independent problem solving；non_university population 混杂因素 Practice Effect 质量维度 D1 研究设计=2 · D2 样本质量=2 · D3 测量效度=1 · D4 时间强度=1 · D5 直接性=1 质量分 7.0 证据等级 moderate 适用性 learner_match=partial · subject_match=introductory_programming · tool_match=codex · scope=short-term manual code modification 置信度 0.5 证据状态已支持完整主张可使用 AI 生成代码并未降低学生在人工代码修改任务上的表现（组间差异不显著）。来源位置 https://dl.acm.org/doi/10.1145/3544548.3580919 可验证链接 https://dl.acm.org/doi/10.1145/3544548.3580919	`S-2023-kazemitabaar`
`E-003`	记忆保持	零效应与主张关系: 中性	9	训练结束一周后，Codex 组与基线组的保持性差异未达统计显著（Codex 组略优）。查看完整证据研究 ID STUDY-KAZEMITABAAR-2023 样本 ID SMPL-KAZEMITABAAR-2023-N69 研究标题 Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming 来源标题 Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming 年份 2023 研究设计随机对照试验教育阶段 K-12 Ages 10 17 研究人群 69 名 10-17 岁编程新手，此前无文本编程经验样本量 69 干预 GPT Tutor（护栏版：教师设计提示、不给直接答案）用于数学练习对照 / 比较条件对照组（无 AI）与 GPT Base（无护栏）组结果测量 retention post-test one week after training 效应 / 结果 slightly better for Codex group but not significant 效应方向零效应与主张关系中性干预时长 3 Weeks Training Plus 1 Week Retention 方法 controlled experiment with delayed retention test 优势 delayed_test_included 局限 1-week retention window is short；small sample；non-university population 混杂因素 Prior Competency 质量维度 D1 研究设计=2 · D2 样本质量=2 · D3 测量效度=2 · D4 时间强度=2 · D5 直接性=1 质量分 9.0 证据等级 strong 适用性 learner_match=partial · subject_match=introductory_programming · tool_match=codex · scope=retention over one week 置信度 0.5 证据状态已支持完整主张训练结束一周后，Codex 组与基线组的保持性差异未达统计显著（Codex 组略优）。来源位置 https://dl.acm.org/doi/10.1145/3544548.3580919 可验证链接 https://dl.acm.org/doi/10.1145/3544548.3580919	`S-2023-kazemitabaar`
`E-004`	独立问题解决	负向效应与主张关系: 反驳	8	无护栏使用 GPT-4 的学生在练习表现更高的同时，独立考试成绩比对照组低 17%。查看完整证据研究 ID STUDY-BASTANI-2025 样本 ID SMPL-BASTANI-2025-N950 研究标题 Generative AI without guardrails can harm learning: Evidence from high school mathematics 来源标题 Generative AI without guardrails can harm learning: Evidence from high school mathematics 年份 2025 研究设计随机对照试验教育阶段高中研究人群土耳其近千名高中数学学生（约 1000 名学生，共 2848 次观测）样本量 950 干预训练阶段一半学习者可使用 OpenAI Codex 完成代码编写任务，任务后接代码修改任务对照 / 比较条件无 Codex 访问组结果测量 exam without access to AI resources after practice phase 效应 / 结果 negative_17_percent_on_independent_exam 效应方向负向效应与主张关系反驳干预时长 In Class Study Sessions 方法 large-scale randomized controlled trial, practice phase then closed-book exam 优势 large_scale_rct；independent_exam_without_ai；arm_wise_design 局限 high_school_mathematics_not_university_programming；single_country 混杂因素 Tool Design Difference 质量维度 D1 研究设计=2 · D2 样本质量=2 · D3 测量效度=2 · D4 时间强度=1 · D5 直接性=1 质量分 8.0 证据等级 strong 适用性 learner_match=partial_same_age_band_different_subject · subject_match=no_mathematics_vs_programming · tool_match=gpt4_chat_interface · scope=unguarded_general_chat_interface 置信度 0.75 证据状态已支持完整主张无护栏使用 GPT-4 的学生在练习表现更高的同时，独立考试成绩比对照组低 17%。来源位置 https://www.pnas.org/doi/10.1073/pnas.2422633122 可验证链接 https://www.pnas.org/doi/10.1073/pnas.2422633122	`S-2025-bastani`
`E-005`	独立问题解决	零效应与主张关系: 支持	8	AI 导师的护栏设计（给提示而非直接答案、教师参与设计提问）基本消除了负向学习效应，但未观察到正向效应。查看完整证据研究 ID STUDY-BASTANI-2025 样本 ID SMPL-BASTANI-2025-N950 研究标题 Generative AI without guardrails can harm learning: Evidence from high school mathematics 来源标题 Generative AI without guardrails can harm learning: Evidence from high school mathematics 年份 2025 研究设计随机对照试验教育阶段高中研究人群土耳其近千名高中数学学生（约 1000 名学生，共 2848 次观测）样本量 950 干预训练阶段使用 OpenAI Codex 完成代码编写任务对照 / 比较条件无 Codex 访问组；一周后评估后测结果测量 exam without access to AI resources after practice phase 效应 / 结果 negative effect essentially eradicated, no positive effect observed 效应方向零效应与主张关系支持干预时长 In Class Study Sessions 方法 large-scale randomized controlled trial, three arms 优势 direct_manipulation_of_tool_design；large_sample 局限 no_positive_learning_gain_even_with_guardrails；subject_mismatch 混杂因素 Prompt Engineering Effort 质量维度 D1 研究设计=2 · D2 样本质量=2 · D3 测量效度=2 · D4 时间强度=1 · D5 直接性=1 质量分 8.0 证据等级 strong 适用性 learner_match=partial · subject_match=no · tool_match=guardrailed_tutor_design · scope=guardrail_design_principle_transferable 置信度 0.75 证据状态已支持完整主张 AI 导师的护栏设计（给提示而非直接答案、教师参与设计提问）基本消除了负向学习效应，但未观察到正向效应。来源位置 https://www.pnas.org/doi/10.1073/pnas.2422633122 可验证链接 https://www.pnas.org/doi/10.1073/pnas.2422633122	`S-2025-bastani`
`E-006`	作业成绩	正向效应与主张关系: 支持	8	练习阶段使用 GPT-4 提升了任务表现（GPT Base 组 +48%、GPT Tutor 组 +127%），但该任务表现并未迁移到独立考试。查看完整证据研究 ID STUDY-BASTANI-2025 样本 ID SMPL-BASTANI-2025-N950 研究标题 Generative AI without guardrails can harm learning: Evidence from high school mathematics 来源标题 Generative AI without guardrails can harm learning: Evidence from high school mathematics 年份 2025 研究设计随机对照试验教育阶段高中研究人群土耳其近千名高中数学学生（约 1000 名学生，共 2848 次观测）样本量 950 干预学生在学术写作过程中使用 ChatGPT 的体验与策略（质性研究，无效应量测量）对照 / 比较条件无对照组（质性案例研究）结果测量 practice problem performance during study sessions 效应 / 结果 48-127 percent improvement on practice problems 效应方向正向效应与主张关系支持干预时长 In Class Study Sessions 方法 randomized controlled trial with practice and closed-book exam phases 优势 same_study_compares_task_and_learning；large_sample 局限 subject_mismatch_mathematics 混杂因素 Task Familiarity 质量维度 D1 研究设计=2 · D2 样本质量=2 · D3 测量效度=2 · D4 时间强度=1 · D5 直接性=1 质量分 8.0 证据等级 strong 适用性 learner_match=partial · subject_match=no · tool_match=gpt4 · scope=task_performance_vs_learning_separation 置信度 0.75 证据状态已支持完整主张练习阶段使用 GPT-4 提升了任务表现（GPT Base 组 +48%、GPT Tutor 组 +127%），但该任务表现并未迁移到独立考试。来源位置 https://www.pnas.org/doi/10.1073/pnas.2422633122 可验证链接 https://www.pnas.org/doi/10.1073/pnas.2422633122	`S-2025-bastani`
`E-007`	知识获得	正向效应与主张关系: 支持	6	以 ChatGPT 作为形成性反馈工具，对学生学术写作能力产生了显著的正向影响，学生评价亦为正面。查看完整证据研究 ID STUDY-MARZUKI-2024 样本 ID SMPL-MARZUKI-2024-N72 研究标题 Impact of ChatGPT on ESL students' academic writing skills 来源标题 Impact of ChatGPT on ESL students' academic writing skills 年份 2024 研究设计混合方法教育阶段 Undergraduate 研究人群印度某大学本科英语作为第二语言（ESL）学生，n=72 样本量 72 干预学生在学术写作过程中使用 ChatGPT 的体验与策略对照 / 比较条件无对照组（质性案例研究）结果测量 writing tests with pre-post-delayed design 效应 / 结果 significant positive impact on writing skills 效应方向正向效应与主张关系支持干预时长 6 Hours Intervention 方法 mixed methods intervention study, pre/post/delayed tests and focus groups 优势 delayed_post_test；mixed_methods_triangulation 局限 short_intervention_6_hours；single_institution；elite_private_university 混杂因素 Self Selection Consent 质量维度 D1 研究设计=1 · D2 样本质量=1 · D3 测量效度=2 · D4 时间强度=2 · D5 直接性=0 质量分 6.0 证据等级 moderate 适用性 learner_match=yes_undergraduate · subject_match=no_writing_not_programming · tool_match=chatgpt · scope=formative_feedback_writing 置信度 0.55 证据状态已支持完整主张以 ChatGPT 作为形成性反馈工具，对学生学术写作能力产生了显著的正向影响，学生评价亦为正面。来源位置 https://link.springer.com/article/10.1186/s40561-024-00295-9 可验证链接 https://link.springer.com/article/10.1186/s40561-024-00295-9	`S-2024-marzuki`
`E-008`	完成时间	正向效应与主张关系: 支持	8	随机对照实验（n=95）显示：使用 Copilot 的职业开发者完成标准化编码任务的用时比对照组缩短约 55%。查看完整证据研究 ID STUDY-PENG-2023 样本 ID SMPL-PENG-2023-N95 研究标题 The Impact of AI on Developer Productivity: Evidence from GitHub Copilot 来源标题 The Impact of AI on Developer Productivity: Evidence from GitHub Copilot 年份 2023 研究设计随机对照试验教育阶段 Professional Developers Not Students 研究人群 95 名经自由职业平台招募的职业开发者，完成标准化编码任务样本量 95 干预任务期间可使用 GitHub Copilot 对照 / 比较条件不可使用 Copilot 的对照组结果测量 Time To Complete Http Server Implementation 效应 / 结果 ~55.8% faster task completion in Copilot group 效应方向正向效应与主张关系支持干预时长 Single Task Session 方法 online randomized controlled experiment with objective completion-time metric 优势 randomized_controlled_design；objective_completion_time_metric 局限 professional_population_not_students；single_task_ecology；preprint_not_peer_reviewed 混杂因素 Task Familiarity；Platform Recruitment Self Selection 质量维度 D1 研究设计=2 · D2 样本质量=2 · D3 测量效度=2 · D4 时间强度=1 · D5 直接性=1 质量分 8.0 证据等级 moderate 适用性 learner_match=mismatch_professional_developers · subject_match=adjacent_web_development_task · tool_match=copilot_like_generative_ai · scope=task_performance_only_no_learning_outcome 置信度 0.6 证据状态已支持完整主张随机对照实验（n=95）显示：使用 Copilot 的职业开发者完成标准化编码任务的用时比对照组缩短约 55%。来源位置 https://doi.org/10.48550/arXiv.2302.06590 可验证链接 https://doi.org/10.48550/arXiv.2302.06590	`S-2023-peng`
`E-009`	代码质量	零效应与主张关系: 中性	7	系统性基准评估显示 Copilot 生成代码相对人类代码的质量结论不一：部分基准上正确性具竞争力，同时记录到安全相关缺陷。查看完整证据研究 ID STUDY-YETISTIREN-2023 样本 ID SMPL-YETISTIREN-2023-BENCH 研究标题 GitHub Copilot AI Pair Programmer: Asset or Liability? 来源标题 GitHub Copilot AI Pair Programmer: Asset or Liability? 年份 2023 研究设计观察性研究教育阶段 Not Applicable Code Artifacts 研究人群取自公开基准数据集的 Copilot 生成程序与人类编写程序干预 Copilot 生成的程序对照 / 比较条件相同基准上的人类编写程序结果测量 Correctness Security Maintainability Metrics On Benchmarks 效应 / 结果 mixed quality profile; no single-direction summary 效应方向零效应与主张关系中性干预时长 Not Applicable Artifact Study 方法 systematic empirical evaluation of generated code against human baselines on public benchmarks 优势 multi_dimensional_quality_metrics；reproducible_benchmark_protocol 局限 artifact_benchmark_not_classroom；no_learning_outcome；tool_version_from_2023 混杂因素 Benchmark Task Distribution 质量维度 D1 研究设计=1 · D2 样本质量=2 · D3 测量效度=2 · D4 时间强度=1 · D5 直接性=1 质量分 7.0 证据等级 moderate 适用性 learner_match=mismatch_no_learners_in_study · subject_match=introductory_adjacent_code_tasks · tool_match=copilot_like_generative_ai · scope=output_quality_only 置信度 0.55 证据状态已支持完整主张系统性基准评估显示 Copilot 生成代码相对人类代码的质量结论不一：部分基准上正确性具竞争力，同时记录到安全相关缺陷。来源位置 https://doi.org/10.1016/j.jss.2023.111734 可验证链接 https://doi.org/10.1016/j.jss.2023.111734	`S-2023-yetistiren`
`E-010`	作业成绩	正向效应与主张关系: 支持	7	Codex 在 CS1 考试风格题目上能给出通过水平的解答（依数据集约 50%–75%），表明新手手中存在可观的任务能力余量。查看完整证据研究 ID STUDY-FINNIEANSLEY-2022 样本 ID SMPL-FINNIEANSLEY-2022-QSETS 研究标题 Using GitHub Copilot to Solve Introductory Programming Problems 来源标题 Using GitHub Copilot to Solve Introductory Programming Problems 年份 2022 研究设计观察性研究教育阶段 University Year 1 Question Sets 研究人群 CS1 考试风格题目集，由 Codex 作答并与已发表的学生分数分布比较干预 Codex 对 CS1 题目作答生成对照 / 比较条件已发表的学生同届分数分布结果测量 Pass Rate On CS1 Exam Style Questions 效应 / 结果 passing solutions on ~50-75% of questions across datasets 效应方向正向效应与主张关系支持干预时长 Not Applicable Capability Probe 方法 capability benchmark against published student distributions; reproducible question sets 优势 public_reproducible_question_sets；directly_relevant_task_domain 局限 tool_solves_task_does_not_equate_student_learning；codex_2021_model_version_outdated 混杂因素 Question Leakage Into Training Data Possible 质量维度 D1 研究设计=1 · D2 样本质量=2 · D3 测量效度=2 · D4 时间强度=1 · D5 直接性=1 质量分 7.0 证据等级 moderate 适用性 learner_match=partial_measures_tool_not_students · subject_match=introductory_programming · tool_match=copilot_like_generative_ai · scope=tool_capability_headroom 置信度 0.55 证据状态已支持完整主张 Codex 在 CS1 考试风格题目上能给出通过水平的解答（依数据集约 50%–75%），表明新手手中存在可观的任务能力余量。来源位置 https://dl.acm.org/doi/10.1145/3545945.3569830 可验证链接 https://dl.acm.org/doi/10.1145/3545945.3569830	`S-2022-finnie-ansley`
`E-011`	元认知	正向效应与主张关系: 支持	7	受控比较发现 LLM 生成的代码讲解与学生自撰讲解相当（部分更优），适合作为解释性支架材料，而非替代学生的解释练习。查看完整证据研究 ID STUDY-EXPLCOMP-2023 样本 ID SMPL-EXPLCOMP-2023-RATINGS 研究标题 Comparing Code Explanations Created by Students and Large Language Models 来源标题 Comparing Code Explanations Created by Students and Large Language Models 年份 2023 研究设计观察性研究教育阶段 University Introductory 研究人群同一批短程序的学生版与 LLM 版讲解的受控对比干预 LLM 生成的代码讲解对照 / 比较条件学生撰写的同题讲解结果测量 Rated Explanation Quality And Comprehensibility 效应 / 结果 comparable-or-better rated quality vs student explanations 效应方向正向效应与主张关系支持干预时长 Single Session Ratings 方法 controlled comparison with blind rating of explanation pairs 优势 controlled_pairwise_comparison；learning_process_relevant_construct 局限 short_term_ratings_not_learning_gains；small_program_snippets_ecology 混杂因素 Rating Criteria Subjectivity 质量维度 D1 研究设计=1 · D2 样本质量=2 · D3 测量效度=2 · D4 时间强度=1 · D5 直接性=1 质量分 7.0 证据等级 moderate 适用性 learner_match=partial_scaffold_material_only · subject_match=introductory_programming · tool_match=llm_explanations · scope=scaffold_quality_not_effectiveness 置信度 0.55 证据状态已支持完整主张受控比较发现 LLM 生成的代码讲解与学生自撰讲解相当（部分更优），适合作为解释性支架材料，而非替代学生的解释练习。来源位置 https://dl.acm.org/doi/10.1145/3587102.3588785 可验证链接 https://dl.acm.org/doi/10.1145/3587102.3588785	`S-2023-explanations-compare`
`E-012`	过度依赖	负向效应与主张关系: 反驳	7	尽管首任务完成更快，参与者难以理解并调试 AI 生成的解法、对最终程序所有权感低——记录了纯速度指标遗漏的元认知与依赖风险。查看完整证据研究 ID STUDY-VAITHILINGAM-2022 样本 ID SMPL-VAITHILINGAM-2022-N24 研究标题 Expectation vs. Experience: Evaluating the Usability of Code Generation Tools 来源标题 Expectation vs. Experience: Evaluating the Usability of Code Generation Tools 年份 2022 研究设计质性研究教育阶段 Mixed Cs Students And Professionals 研究人群 24 名参与者参与的 Copilot 类工具组内可用性研究样本量 24 干预 Copilot 类工具辅助编程对照 / 比较条件不使用工具的组内基线结果测量 Understanding Ownership And Debugging Reports 效应 / 结果 documented comprehension/ownership difficulties despite speed gain 效应方向负向效应与主张关系反驳干预时长 Single Session 方法 within-subject usability study with tasks, observation and interviews 优势 rich_qualitative_process_data；constructs_missed_by_speed_metrics 局限 small_n_24；single_session；self_reported_understanding 混杂因素 Participant AI Familiarity 质量维度 D1 研究设计=1 · D2 样本质量=2 · D3 测量效度=2 · D4 时间强度=1 · D5 直接性=1 质量分 7.0 证据等级 moderate 适用性 learner_match=partial_includes_cs_students · subject_match=programming_adjacent · tool_match=copilot_like_generative_ai · scope=risk_identification 置信度 0.55 证据状态被反驳完整主张尽管首任务完成更快，参与者难以理解并调试 AI 生成的解法、对最终程序所有权感低——记录了纯速度指标遗漏的元认知与依赖风险。来源位置 https://dl.acm.org/doi/10.1145/3491101.3519665 可验证链接 https://dl.acm.org/doi/10.1145/3491101.3519665	`S-2022-vaithilingam`

03 证据可信度、反证与方法审计

检查证据为什么可信、哪里冲突，以及哪些结论必须降级。

审查目标：overall

关注

对照组通过

提出因果主张的量化研究均含对照条件：Bastani（E-004/E-005/E-006）为三臂随机对照（无护栏 GPT Base / 护栏 GPT Tutor / 无 AI 对照，约千名学生）；…

查看审计依据

提出因果主张的量化研究均含对照条件：Bastani（E-004/E-005/E-006）为三臂随机对照（无护栏 GPT Base / 护栏 GPT Tutor / 无 AI 对照，约千名学生）；Kazemitabaar（E-001/E-002/E-003）为随机对照（有/无 Codex，n=69）；Peng（E-008）为职业开发者随机对照（n=95）。不含对照的是不承担因果主张的研究：Marzuki（E-007）为混合方法，Vaithilingam（E-012）为组内可用性研究（n=24），Yetistiren（E-009）与Finnie-Ansley（E-010）为基准评估。

随机分配通过

Kazemitabaar（E-001/E-002/E-003）与 Bastani（E-004/E-005/E-006）均为随机分配。

前测通过

Kazemitabaar 有前测评估；Bastani 测量基线协变量。

后测通过

三项随机对照研究均报告即时后测。

保持测试部分

Kazemitabaar（E-003）有一周保持测；Bastani（E-004/E-005/E-006）无延迟测验；Marzuki（E-007）有延迟测量。

迁移测试部分

Kazemitabaar 的代码修改任务属迁移邻近任务；本证据集缺少完整的无 AI 迁移测验。

样本偏差通过

Bastani 样本接近千人；Kazemitabaar 样本偏小且年龄偏低（n=69，10-17 岁）。

自我选择偏差部分

Marzuki（E-007）基于知情同意招募，存在自我选择风险。

测量效度部分

完成率/正确率/速度作为'学习'代理的效度可疑：E-001 练习正确率、E-004 任务得分、E-011/E-012 完成速度与进展均在 AI 可访问条件下测得，AI 可直接产出答案抬高指标，无法区分'学会了'与'抄到了'（Bastani 机制数据 E-002：GPT Base 答对率 5…

查看审计依据

完成率/正确率/速度作为'学习'代理的效度可疑：E-001 练习正确率、E-004 任务得分、E-011/E-012 完成速度与进展均在 AI 可访问条件下测得，AI 可直接产出答案抬高指标，无法区分'学会了'与'抄到了'（Bastani 机制数据 E-002：GPT Base 答对率 51% 中 42% 为逻辑错误；Wermelinger S-2023 显示 Copilot 可首次尝试解决 24 道典型入门题中的 16 道，FETCH_PARTIAL 仅验证到机构库摘要）。自评测量不可靠（E-002 学生过度乐观、E-008 能力错觉）。效度较高的测量（无 AI 独立考试 E-002/E-003、延迟后测 E-005）恰恰给出负向或零结果——测量选择本身决定结论方向。

混杂因素部分

Kazemitabaar 中先验编程能力与 AI 收益存在交互。

教师效应不适用

Kazemitabaar 为自定进度；课堂类研究可能带有教师效应。

新奇效应部分

短周期干预易高估参与度；本证据集的研究均未控制新奇效应。

工具版本效应不适用

各研究只覆盖单一工具版本，工具迭代快，结论耐久性受限。

AI 使用规则部分

Bastani（E-001~E-003）是唯一直接操纵 AI 使用政策的研究：无护栏 GPT Base（类标准 ChatGPT 界面，可抄答案）vs 护栏 GPT Tutor（教师设计提示、不给直接答案），对应实证了'允许使用但无规则→独立考试 -17% 伤害'与'有护栏→练习 +127%…

查看审计依据

Bastani（E-001~E-003）是唯一直接操纵 AI 使用政策的研究：无护栏 GPT Base（类标准 ChatGPT 界面，可抄答案）vs 护栏 GPT Tutor（教师设计提示、不给直接答案），对应实证了'允许使用但无规则→独立考试 -17% 伤害'与'有护栏→练习 +127% 且负效应消除'的政策对比，与试点'禁止直接提交 AI 代码、实验课独立评测'的护栏设计同构。局限：护栏效果仅在单一情境（高中数学、教师设计提示）验证过，需在 C 语言场景复验；且各研究均未验证对照组依从性（对照组成员是否实际未使用 AI），政策污染未排除。

样本流失部分

Marzuki（E-007）报告了流失情况；其余研究未详细说明。

任务 vs 学习护栏：

查看审计依据

证据集本身未把任务表现等同学习效果：三类独立测量（E-002/E-003/E-005）与任务表现测量（E-001/E-004/E-011/E-012）被明确分开，且独立测量给出负向/零结果，恰是对照铁律（SKILL.md RULE 3：task performance 不得自动等同 learning effect）的正确执行。但风险在边界处：(a) 若下游综合以 E-001 的 +127% 或 E-011 的快 35% 作为'学习提升'证据，即违反铁律，证据天平会系统性偏向'允许使用'；(b) E-001 练习成绩提升与 E-002 独立考试伤害在同一研究中并存，任何只引其一的做法都会误导。frame 的 outcomes 已把'任务表现'与'学习能力'分开测量、并把期末无 AI 统一机试/笔试设为唯一成功判据，与本 guard 一致。

裁决说明：

分歧来自结果分离（任务 vs 学习）、工具设计（有护栏 vs 无护栏）与人群（K-12/职业者 vs 大学生）。随机实验与基准研究中任务表现证据一致为正；唯一测量移除 AI 后独立表现的研究显示无护栏时有害；可用性与工件研究补充依赖与质量警示而非解决学习问题。

决策：试点验证 · 置信度：中

可以主张

AI 编程助手在训练期间可靠地提升任务表现（完成速度、正确性）—— E-001、E-006。
E-001E-006
无护栏的生成式 AI 访问在移除工具后可能损害独立问题解决能力 —— E-004。
E-004
护栏设计（给提示而非给答案）能大幅缓解负面学习效应 —— E-005。
E-005
任务表现提升并不自动等于学习提升 —— E-004 与 E-006 的研究内对照。
E-004E-006
工具能力可观：Codex 能解出约半数至四分之三的 CS1 考试风格题目 —— E-010。
E-010
职业开发者 RCT 显示 Copilot 带来约 55% 任务提速；但职业人群限制直接性 —— E-008。
E-008
LLM 代码讲解的质量评级与学生自撰讲解相当，可作支架材料 —— E-011。
E-011

尚不能主张

AI 编程助手能否真正改善或保持大学新手的编程学习——本证据集中没有大学层面的直接 RCT [无直接证据]
Kazemitabaar 2023 的一周中性保持性能否延伸到一个学期 —— E-003。
E-003
基准质量结论（E-009）与讲解质量评级（E-011）能否转化为课堂学习收益。
E-009E-011
可用性研究所记录的理解/所有权困难（E-012）在整学期护栏条件下会如何演变。
E-012

被反驳的主张

'AI 工具总能提高学习'被 E-004 反驳（无护栏访问，独立考试 −17%）。
E-004
'速度收益等于学习收益'被 E-001/E-006/E-008 与 E-004 之间的任务-学习分离所反驳。
E-001E-006E-008E-004

缺失证据

在大学编程课程中带保持与无 AI 迁移测试的 RCT。
同一课程内变化 AI 使用政策的研究。
跨越一门课的 AI 依赖纵向数据。
职业提速 RCT 的同行评审重复（Peng 等仍为预印本）。

EvidenceFlow 协议

裁决信息图

C-001

在练习环节，无护栏的 GPT Base（类标准 ChatGPT 界面）使高中生的练习成绩相对对照组提高 48%，带护栏的 GPT Tutor 提高 127%（Table 1：practice 系数 0.137/0.361，对照均值 0.284）

↓

E-001完成时间正向效应主张关系: 支持Q 9.0→S-2023-kazemitabaar

C-002

移除 AI 访问后的无辅助独立考试中，GPT Base 组成绩比从未使用 AI 的对照组低 17%（统计显著），表明无护栏使用 AI 损害技能习得；机制上学生把 GPT 当'拐杖'直接抄答案（GPT Base 答对率仅 51%，其中 42% 逻辑错误、8% 算术错误），且学生自评过度乐观、未察觉学习受损

↓

E-002独立问题解决零效应主张关系: 中性Q 7.0→S-2023-kazemitabaar

C-003

带护栏的 GPT Tutor（教师设计提示而非直接答案）在练习成绩 +127% 的同时，移除访问后的独立考试负效应基本消除（-0.004，不显著），说明精心设计的护栏可兼得练习提升与学习保持

↓

E-003记忆保持零效应主张关系: 中性Q 9.0→S-2023-kazemitabaar

C-004

训练阶段使用 OpenAI Codex 的 10-17 岁新手在 45 道 Python 代码编写任务上表现显著提升：完成率提高 1.15 倍、得分提高 1.8 倍

↓

E-004独立问题解决负向效应主张关系: 反驳Q 8.0→S-2025-bastani

C-005

训练期使用 Codex 的学习者一周后评估后测成绩略好于对照组，但差异未达统计显著（保持力无显著差异）；Scratch 前测高分者若有 Codex 使用史，保持后测显著更好

↓

E-005独立问题解决零效应主张关系: 支持Q 8.0→S-2025-bastani

C-006

质性案例研究中，3 名 EFL 学生珍视 ChatGPT 的辅助价值（消除不确定性、澄清词汇、提供内容建议、语法/结构反馈，让学生专注于创意层面），并形成语言精修、观点生成与结构、校对与信心增强等使用策略

↓

E-006作业成绩正向效应主张关系: 支持Q 8.0→S-2025-bastani

C-007

同一质性研究中，学生担忧 AI 使用的学术真实性与过度依赖风险（建议过于复杂/正式、语气不符、文化刻板印象等局限），强调必须保持人的判断并寻求教师/同伴反馈，呼吁伦理指引与批判性思维培养

↓

E-007知识获得正向效应主张关系: 支持Q 6.0→S-2024-marzuki

C-008

随机对照实验（n=95）显示：使用 Copilot 的职业开发者完成标准化编程任务的用时比对照组缩短约 55%。

↓

E-008完成时间正向效应主张关系: 支持Q 8.0→S-2023-peng

C-009

系统性基准评估显示 Copilot 生成代码相对人类代码的质量结论不一：部分正确性具竞争力，同时记录到安全相关缺陷。

↓

E-009代码质量零效应主张关系: 中性Q 7.0→S-2023-yetistiren

C-010

Codex 在 CS1 考试风格题目上能给出通过水平的解答（依数据集约 50%–75%），表明新手手中存在可观的任务能力余量。

↓

E-010作业成绩正向效应主张关系: 支持Q 7.0→S-2022-finnie-ansley

C-011

受控比较发现 LLM 生成的代码讲解与学生自撰讲解相当（部分更优），适合作为解释性支架材料，而非替代学生的解释练习。

↓

E-011元认知正向效应主张关系: 支持Q 7.0→S-2023-explanations-compare

C-012

尽管首任务完成更快，参与者难以理解并调试 AI 生成的解法、对最终程序所有权感低——记录了纯速度指标遗漏的认知与依赖风险。

↓

E-012过度依赖负向效应主张关系: 反驳Q 7.0→S-2022-vaithilingam

这意味着什么：每个重要主张都必须能追到 Evidence ID 和原始来源。

04 适用范围与教学行动

把可外推范围、护栏和教学动作连接到具体证据。

目标人群：首次学习 C 语言编程的大一计算机专业学生

目标情境：16 周讲授课+实验课，60 人班级，助教支持，线下

适用于谁：在大一 C 课程以护栏化使用政策开展试点

不适用于：无使用政策的全面放开采用

适用条件：

护栏化 AI 使用政策（给提示不给答案，仿 GPT Tutor 组）
无 AI 迁移评估
助教支持

不可外推的结论

『AI 编程助手提高学习效果』—— 超出边界：缺少直接学习效应证据。
『AI 对所有人都有效』—— 超出边界：人群与学科错配。

证据

AI 编程助手在训练期间可靠地提升任务表现（完成速度、正确性）—— E-001、E-006。

适用性

在大一 C 课程以护栏化使用政策开展试点

决策

试点验证

护栏

AI 使用分三档明确分级（解释 / 协作 / 无 AI 迁移）。照抄未审视的 AI 输出属学术诚信违规，并通过推理痕迹要求核查。

停止条件

迁移测验成绩显著低于基线同届预期; 推理痕迹中出现普遍诚信违规; 风险指标中 AI 依赖信号超阈值; 助教/教师工作量不可持续

评价

目标学习者：大一 C 语言编程学生（60 人讲授课班） · 试点时长：8 周

AI 使用规则：AI 使用分三档明确分级（解释 / 协作 / 无 AI 迁移）。照抄未审视的 AI 输出属学术诚信违规，并通过推理痕迹要求核查。

阶段一 —— 独立基础

AI 规则：禁止完整代码生成；AI 仅可用于概念讲解

活动：

基线测验
前 2 周作业完全不使用 AI 代码生成

结果检查：基线任务表现与独立问题解决测量

阶段二 —— 只解释，不解题

AI 规则：AI 可以解释，但不得产出完整解题方案

活动：

第 3–4 周：允许 AI 解释错误、概念与调试思路

结果检查：期中无 AI 测验

阶段三 —— 结构化协作

AI 规则：允许部分代码生成；关键逻辑须书面解释；提交需附推理痕迹

活动：

第 5–7 周：允许 AI 生成部分代码；学生必须用自己的话解释每段 AI 生成代码

结果检查：每周实验完成率与代码质量量表

阶段四 —— 迁移检验

AI 规则：迁移评估期间不得使用 AI

活动：

第 8 周：在无 AI 环境完成新的编程任务

结果检查：迁移测验分数、独立问题解决

停止条件

迁移测验成绩显著低于基线同届预期
推理痕迹中出现普遍诚信违规
风险指标中 AI 依赖信号超阈值
助教/教师工作量不可持续

干预时间线信息图

05 试点设计、评估与停止条件

用独立学习结果验证试点，并预先写清停止条件。

研究问题：在大一 C 课程中，护栏化的 AI 编程助手（只解释→结构化协作）相比无 AI 教学，能否在不增加 AI 依赖的前提下提升独立问题解决能力？

基线：第 1 周无 AI 编程测验（独立问题解决、完成用时）

后测：第 8 周无 AI 编程测验（独立问题解决、代码质量）

保持测试：期末考试（第 16 周）—— 试点后 8 周的延迟测量

迁移测试：第 8 周严格无 AI 环境下的新编程任务

过程指标

每周实验完成率
AI 使用日志：提交的提示、复制代码块、推理痕迹
求助行为计数

学习指标

独立问题解决（无 AI 测验）
代码质量量表
期末考试保持
迁移任务得分

风险指标

AI 依赖指数（来自推理痕迹质量的『使用而无理解』）
学术诚信违规
自我报告的过度依赖
虚假信心（测后自信 vs 实际得分）

成功阈值：实验班独立问题解决非劣（差异在 5% 以内）且保持相当或更优、AI 依赖指数低于阈值；若独立问题解决下滑超过 10%，无论任务收益如何，试点均判为失败。

分析计划：预登记的实验班 vs 对照班基线调整学习指标比较（ANCOVA）；任务表现指标与学习指标分开报告；按先验编程能力做亚组分析；第 3、5、7 周监测停止条件。

评价设计信息图

基准图已抑制

result.json 未携带 benchmark.baselines，本图不绘制；基准表现见独立基准报告。

06 来源、溯源与附录

保留原始来源、URL、证据 ID 和获取信息，确保可回查。

来源列表

ID	标题	年份	权威级别	可验证位置
`S-2023-kazemitabaar`	Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming DOI 可验证查看来源与溯源原文标题 Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming 年份 2023 权威级别 T1 DOI 可验证论文来源位置 https://dl.acm.org/doi/10.1145/3544548.3580919 可验证链接 https://dl.acm.org/doi/10.1145/3544548.3580919	2023	T1 DOI 可验证论文	https://dl.acm.org/doi/10.1145/3544548.3580919
`S-2025-bastani`	Generative AI without guardrails can harm learning: Evidence from high school mathematics DOI 可验证查看来源与溯源原文标题 Generative AI without guardrails can harm learning: Evidence from high school mathematics 年份 2025 权威级别 T1 DOI 可验证论文来源位置 https://www.pnas.org/doi/10.1073/pnas.2422633122 可验证链接 https://www.pnas.org/doi/10.1073/pnas.2422633122	2025	T1 DOI 可验证论文	https://www.pnas.org/doi/10.1073/pnas.2422633122
`S-2024-marzuki`	Impact of ChatGPT on ESL students' academic writing skills DOI 可验证查看来源与溯源原文标题 Impact of ChatGPT on ESL students' academic writing skills 年份 2024 权威级别 T1 DOI 可验证论文来源位置 https://link.springer.com/article/10.1186/s40561-024-00295-9 可验证链接 https://link.springer.com/article/10.1186/s40561-024-00295-9	2024	T1 DOI 可验证论文	https://link.springer.com/article/10.1186/s40561-024-00295-9
`S-2023-peng`	The Impact of AI on Developer Productivity: Evidence from GitHub Copilot DOI 可验证查看来源与溯源原文标题 The Impact of AI on Developer Productivity: Evidence from GitHub Copilot 年份 2023 权威级别 Tier2 Academic Database 来源位置 https://arxiv.org/abs/2302.06590 可验证链接 https://doi.org/10.48550/arXiv.2302.06590	2023	Tier2 Academic Database	https://doi.org/10.48550/arXiv.2302.06590
`S-2023-yetistiren`	GitHub Copilot AI Pair Programmer: Asset or Liability? DOI 可验证查看来源与溯源原文标题 GitHub Copilot AI Pair Programmer: Asset or Liability? 年份 2023 权威级别 T1 DOI 可验证论文来源位置 https://doi.org/10.1016/j.jss.2023.111734 可验证链接 https://doi.org/10.1016/j.jss.2023.111734	2023	T1 DOI 可验证论文	https://doi.org/10.1016/j.jss.2023.111734
`S-2022-finnie-ansley`	Using GitHub Copilot to Solve Introductory Programming Problems DOI 可验证查看来源与溯源原文标题 Using GitHub Copilot to Solve Introductory Programming Problems 年份 2022 权威级别 T1 DOI 可验证论文来源位置 https://dl.acm.org/doi/10.1145/3545945.3569830 可验证链接 https://dl.acm.org/doi/10.1145/3545945.3569830	2022	T1 DOI 可验证论文	https://dl.acm.org/doi/10.1145/3545945.3569830
`S-2023-explanations-compare`	Comparing Code Explanations Created by Students and Large Language Models DOI 可验证查看来源与溯源原文标题 Comparing Code Explanations Created by Students and Large Language Models 年份 2023 权威级别 T1 DOI 可验证论文来源位置 https://dl.acm.org/doi/10.1145/3587102.3588785 可验证链接 https://dl.acm.org/doi/10.1145/3587102.3588785	2023	T1 DOI 可验证论文	https://dl.acm.org/doi/10.1145/3587102.3588785
`S-2022-vaithilingam`	Expectation vs. Experience: Evaluating the Usability of Code Generation Tools DOI 可验证查看来源与溯源原文标题 Expectation vs. Experience: Evaluating the Usability of Code Generation Tools 年份 2022 权威级别 T1 DOI 可验证论文来源位置 https://dl.acm.org/doi/10.1145/3491101.3519665 可验证链接 https://dl.acm.org/doi/10.1145/3491101.3519665	2022	T1 DOI 可验证论文	https://dl.acm.org/doi/10.1145/3491101.3519665

Fetch 溯源

搜索提供方：n/a

无逐条 fetch 记录（来源由研究管线直接提供）。

EduEvidenceDataLab [Light]Data: real studies · manually curated

我们准备在大学一年级 C 语言课程中允许学生使用生成式 AI 编程助手。它到底会不会提高学习效果？应该怎样引入？

Mode: Platform native · Generated: 2026-08-24 · Evidence: 12 · Sources: 8

Decision first

What to do, how confident we are, and the most important evidence boundary.

Recommended decision Pilot Confidence · Moderate

Positive task-performance evidence plus documented unguarded-access risk, mixed quality/usability signals, and missing university-level learning evidence → bounded, guardrailed pilot with evaluation, not full adoption.

Strongest supported conclusion

AI coding assistants reliably speed up practice work: completion rate 1.15x and time 0.57x in a randomised trial of 69 novices.

Key uncertainty / contradiction

No university-level RCT measures learning directly, and the one large trial that did - unguarded GPT-4 - saw independent exam scores fall 17%.

Main risk

Unguarded access can raise practice performance while lowering independent exam performance, and learners may not notice the gap.

Next action

Run a phased CS1 pilot with hints-not-answers guardrails, weekly lab use, and a no-AI transfer exam that can stop the pilot.

Evidence / sources · 12 / 8

Task performance ≠ learning

Only informative outcome separation; positive, negative and null effects use effect_direction.

Outcome Separation · Task performance ≠ learning

Outcomes are adjudicated separately so faster training performance is not silently treated as evidence of learning.

Effect direction comes from evidence.effect_direction; supporting a claim does not imply a positive outcome.

Task / proximal performance

Completion timePositive effect 2
Code qualityNull effect 1
Assignment scorePositive effect 2

Learning / retention / transfer

Knowledge gainPositive effect 1
RetentionNull effect 1
Independent problem solvingNegative effect 1Null effect 2

Risk / dependency

Over-relianceNegative effect 1

Other outcomes

MetacognitionPositive effect 1

What this means: Positive / negative / null effect-direction evidence counts per outcome. This visual encodes effect_direction, not whether evidence supports a claim.

Evidence tribunal

Supported, uncertain, contradicted and missing evidence stay separated instead of flattened into long prose.

Decision: Pilot · Confidence: Moderate

Can claim

AI coding assistants reliably increase task performance during training (completion speed, correctness) — E-001, E-006.
E-001E-006
Unguarded generative AI access can harm independent problem solving when access is removed — E-004.
E-004
Guardrail design (hints instead of answers) substantially mitigates the negative learning effect — E-005.
E-005

View 4 more

Task performance gains do not automatically imply learning gains — E-004 vs E-006 (within-study contrast).
E-004E-006
Tool capability is substantial: Codex solves roughly half to three-quarters of CS1 exam-style questions — E-010.
E-010
Professional-developer RCT shows ~55% faster task completion with Copilot; directness limited by professional population — E-008.
E-008
LLM code explanations rate comparable to student-authored explanations, viable as scaffold material — E-011.
E-011

Cannot yet claim

Whether AI coding assistants improve or preserve actual programming learning in university novices — no direct university-level RCT in reviewed set [无直接证据]
Whether one-week neutral retention (Kazemitabaar 2023) extends to a semester — E-003.
E-003
Whether benchmark quality findings (E-009) and explanation-quality ratings (E-011) translate into classroom learning gains.
E-009E-011

View 1 more

How comprehension/ownership difficulties documented in usability studies (E-012) behave over a full semester with guardrails.
E-012

Contradicted claims

The claim 'AI tools always improve learning' is contradicted by E-004 (unguarded access, -17% independent exam).
E-004
The claim 'speed gains equal learning gains' is contradicted by the task-vs-learning separation across E-001/E-006/E-008 vs E-004.
E-001E-006E-008E-004

Missing evidence

RCT of AI coding assistants in university programming courses with retention and no-AI transfer tests.
Studies varying AI usage policy within the same course.
Longitudinal data on AI dependency beyond one course.

View 1 more

Peer-reviewed replication of the professional speed RCT (Peng et al. remains a preprint).

Evidence to action

Applicability, guardrails, stop conditions and evaluation form one executable path.

Evidence

AI coding assistants reliably increase task performance during training (completion speed, correctness) — E-001, E-006.

Applicability

pilot in first-year C course with guardrailed usage policy

Decision

Pilot

Guardrails

AI usage is allowed in three explicitly graded modes (explain / collaborate / no-AI-transfer).…

Expand full explanation

AI usage is allowed in three explicitly graded modes (explain / collaborate / no-AI-transfer). Copying unexamined AI output is an academic integrity violation and is assessed via the reasoning-trace requirement.

Stop conditions

transfer-test scores drop significantly below baseline cohort expectations; widespread integrity violations in reasoning traces;…

Expand full explanation

transfer-test scores drop significantly below baseline cohort expectations; widespread integrity violations in reasoning traces; AI dependency signals exceed threshold in risk metrics; TA/teacher workload becomes unsustainable

Evaluation

treatment group shows non-inferior independent problem solving (delta within 5%) AND superior or equal retention AND AI dependency index below thresho…

Expand full explanation

treatment group shows non-inferior independent problem solving (delta within 5%) AND superior or equal retention AND AI dependency index below threshold; if independent problem solving declines >10%, the pilot is judged unsuccessful regardless of task-performance gains.

Lieflat Editorial Gallery

Charts selected and composed by AI from the Lieflat catalog; every number traces back to result.json.

Year × dimension evidence almanac

Bubble area ∝ study count · solid core = significant results

Drawn only when years and outcome dimensions exist.

L9 Bubble Almanac · Evidence.year X Dimension

Year × outcome evidence density

Each cell counts evidence items for that year and outcome

Shows where the evidence sits across years and outcomes.

L16 Matrix Heat · Evidence.year X Outcome Counts

Effect direction by outcome

One dot = one evidence item · green = positive · grey = null · orange = negative

Based on effect_direction counts; all numbers come from result.json.

F5 Tick Rows · Outcomes.direction Counts

Positive vs negative evidence by outcome

Two columns summarise positive and negative evidence counts

Splits positive and negative evidence so they never cancel out.

F6 Paired Rungs · Outcomes.paired Counts

Net effect direction by outcome

Position = (positive - negative) / directional count · centre is neutral

Bipolar view of whether each outcome leans supportive or against.

L7 Brand Spectrum · Outcomes.bipolar Axes

Study-design composition

One cell = one study · shows which designs produced the evidence

Reveals design skew faster than a table when several designs are present.

L14 Hundred Field · Evidence.study Type Composition

Key sources

Only the key sources in the brief; full traceability expands in the full report.

S-2023-kazemitabaar

Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming

Tier 1 DOI-verified paper · 2023

S-2025-bastani

Generative AI without guardrails can harm learning: Evidence from high school mathematics

Tier 1 DOI-verified paper · 2025

S-2024-marzuki

Impact of ChatGPT on ESL students' academic writing skills

Tier 1 DOI-verified paper · 2024

S-2023-peng

The Impact of AI on Developer Productivity: Evidence from GitHub Copilot

Tier2 Academic Database · 2023

4 more sources are traceable in the full report.

Full Report

Conclusions first: every traceable piece of evidence and method note lives here, with visuals only at points where they add meaning. Every number traces back to result.json.

01 Decision, Adjudication & Research Boundary

State the final adjudication and research boundary before explaining why.

Recommended decision Pilot Confidence · Moderate

Strongest supported conclusion

AI coding assistants reliably speed up practice work: completion rate 1.15x and time 0.57x in a randomised trial of 69 novices.

Key uncertainty / contradiction

No university-level RCT measures learning directly, and the one large trial that did - unguarded GPT-4 - saw independent exam scores fall 17%.

Main risk

Unguarded access can raise practice performance while lowering independent exam performance, and learners may not notice the gap.

Next action

Run a phased CS1 pilot with hints-not-answers guardrails, weekly lab use, and a no-AI transfer exam that can stop the pilot.

Evidence / sources · 12 / 8

Research question

我们准备在大学一年级 C 语言课程中允许学生使用生成式 AI 编程助手。它到底会不会提高学习效果？应该怎样引入？

Target learners

Education level: First-year undergraduate; Major: Computer science; Prior knowledge: First programming course, no prior text-based programming; Learner characteristics: Mixed-ability large class (60 students)

Course context

Subject: C programming; Course type: Lecture + lab; Duration: 16 weeks (one semester)

AI intervention

Method: Lecture with lab exercises; AI tool: Generative AI coding assistant; Allowed usage: Under design (pending evidence review); Frequency: Weekly lab sessions; Duration: One semester

Comparison

No AI Coding Assistant Control

Outcome constructs

Primary outcomes: Independent problem solving, Code quality; Secondary outcomes: Completion time, Retention, Knowledge gain; Risk outcomes: AI dependency, Over-reliance, Reduced transfer

Research scope

Time range: 2021 2026; Geography: Worldwide; Study designs: Randomized controlled trial, Quasi-experimental, Observational

Decision success condition

independent problem solving and code quality improve (or do not decline) while AI dependency risk stays controlled; evidence base supports a bounded pilot.

02 Key Evidence & Outcome Separation

Place task performance, actual learning, retention and risk on one evidence map without conflating them.

Inclusion criteria

Studies Of Generative AI Coding Tools In Learning To Program
Outcomes Measuring Learning Not Only Task Speed
University Or Novice Programming Populations

Exclusion criteria

Practitioner Anecdotes Without Data
Industry Professional Populations Only

Source coverage

S-2023-kazemitabaar S-2025-bastani S-2024-marzuki S-2023-peng S-2023-yetistiren S-2022-finnie-ansley S-2023-explanations-compare S-2022-vaithilingam

This report shows only retrieval metadata present in result; it does not fabricate PRISMA/funnel counts when none exist.

Outcome Separation · Task performance ≠ learning

Outcomes are adjudicated separately so faster training performance is not silently treated as evidence of learning.

Effect direction comes from evidence.effect_direction; supporting a claim does not imply a positive outcome.

Task / proximal performance

Completion timePositive effect 2
Code qualityNull effect 1
Assignment scorePositive effect 2

Learning / retention / transfer

Knowledge gainPositive effect 1
RetentionNull effect 1
Independent problem solvingNegative effect 1Null effect 2

Risk / dependency

Over-relianceNegative effect 1

Other outcomes

MetacognitionPositive effect 1

Outcome	Positive effect	Negative effect	Null effect	Evidence
Knowledge gainknowledge_gain	1	0	0	`E-007`
Retentionretention	0	0	1	`E-003`
Independent problem solvingindependent_problem_solving	0	1	2	`E-002` `E-004` `E-005`
Completion timecompletion_time	2	0	0	`E-001` `E-008`
Code qualitycode_quality	0	0	1	`E-009`
Assignment scoreassignment_score	2	0	0	`E-006` `E-010`
Metacognitionmetacognition	1	0	0	`E-011`
Over-relianceover_reliance	0	1	0	`E-012`

What this means: Positive / negative / null effect-direction evidence counts per outcome. This visual encodes effect_direction, not whether evidence supports a claim.

Fig. 1. Positive / negative / null effect counts per outcome (based on effect_direction, not claim support).

ID	Outcome	Effect	Quality	Claim	Source
`E-001`	Completion time	Positive effectClaim relation: Support	9	AI coding assistants significantly increase task completion speed and completion rate during training. View full evidence Study ID STUDY-KAZEMITABAAR-2023 Sample ID SMPL-KAZEMITABAAR-2023-N69 Study title Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming Source title Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming Year 2023 Study design Randomized controlled trial Education level K-12 Ages 10 17 Population 69 novices ages 10-17 with no prior text-based programming experience Sample size 69 Intervention Access To Openai Codex AI Coding Assistant During Training Comparison Baseline Group Without AI Coding Assistant Outcome measure Code Authoring Task Progress And Time Effect / result 1.15x completion rate, 0.57x time, 1.8x correctness Effect direction Positive effect Relation to claim Support Duration 3 Weeks Training Method controlled experiment with random assignment, immediate post-test and 1-week retention test Strengths randomized_controlled_design；immediate_post_test_and_retention_test；code_modification_task_guard Limitations non_university_population_ages_10_17；small_sample_69；self-paced environment differs from classroom Confounders Prior Programming Competency Interaction Quality dimensions D1 Study design = 2 · D2 Sample quality = 2 · D3 Measurement validity = 2 · D4 Temporal strength = 2 · D5 Directness = 1 Quality score 9.0 Evidence level strong Applicability learner_match=partial_novice_programmers_but_younger · subject_match=introductory_programming · tool_match=codex_like_generative_ai · scope=task_performance_during_training Confidence 0.7 Evidence status Supported Full claim AI coding assistants significantly increase task completion speed and completion rate during training. Source location https://dl.acm.org/doi/10.1145/3544548.3580919 Verifiable URL https://dl.acm.org/doi/10.1145/3544548.3580919	`S-2023-kazemitabaar`
`E-002`	Independent problem solving	Null effectClaim relation: Neutral	7	Access to AI code generation did not decrease performance on manual code-modification tasks. View full evidence Study ID STUDY-KAZEMITABAAR-2023 Sample ID SMPL-KAZEMITABAAR-2023-N69 Study title Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming Source title Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming Year 2023 Study design Randomized controlled trial Education level K-12 Ages 10 17 Population 69 novices ages 10-17 with no prior text-based programming experience Sample size 69 Intervention Access To Openai Codex AI Coding Assistant During Training Comparison Baseline Group Without AI Coding Assistant Outcome measure manual code-modification tasks during training Effect / result no significant difference between groups Effect direction Null effect Relation to claim Neutral Duration 3 Weeks Training Method controlled experiment, code-modification task followed each code-authoring task Strengths direct_test_of_transfer-adjacent_skill；same_session_measurement Limitations code modification is not full independent problem solving；non_university population Confounders Practice Effect Quality dimensions D1 Study design = 2 · D2 Sample quality = 2 · D3 Measurement validity = 1 · D4 Temporal strength = 1 · D5 Directness = 1 Quality score 7.0 Evidence level moderate Applicability learner_match=partial · subject_match=introductory_programming · tool_match=codex · scope=short-term manual code modification Confidence 0.5 Evidence status Supported Full claim Access to AI code generation did not decrease performance on manual code-modification tasks. Source location https://dl.acm.org/doi/10.1145/3544548.3580919 Verifiable URL https://dl.acm.org/doi/10.1145/3544548.3580919	`S-2023-kazemitabaar`
`E-003`	Retention	Null effectClaim relation: Neutral	9	One week after training, retention differences between Codex and baseline groups did not reach statistical significance. View full evidence Study ID STUDY-KAZEMITABAAR-2023 Sample ID SMPL-KAZEMITABAAR-2023-N69 Study title Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming Source title Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming Year 2023 Study design Randomized controlled trial Education level K-12 Ages 10 17 Population 69 novices ages 10-17 with no prior text-based programming experience Sample size 69 Intervention Access To Openai Codex AI Coding Assistant During Training Comparison Baseline Group Without AI Coding Assistant Outcome measure retention post-test one week after training Effect / result slightly better for Codex group but not significant Effect direction Null effect Relation to claim Neutral Duration 3 Weeks Training Plus 1 Week Retention Method controlled experiment with delayed retention test Strengths delayed_test_included Limitations 1-week retention window is short；small sample；non-university population Confounders Prior Competency Quality dimensions D1 Study design = 2 · D2 Sample quality = 2 · D3 Measurement validity = 2 · D4 Temporal strength = 2 · D5 Directness = 1 Quality score 9.0 Evidence level strong Applicability learner_match=partial · subject_match=introductory_programming · tool_match=codex · scope=retention over one week Confidence 0.5 Evidence status Supported Full claim One week after training, retention differences between Codex and baseline groups did not reach statistical significance. Source location https://dl.acm.org/doi/10.1145/3544548.3580919 Verifiable URL https://dl.acm.org/doi/10.1145/3544548.3580919	`S-2023-kazemitabaar`
`E-004`	Independent problem solving	Negative effectClaim relation: Contradict	8	Students with unguarded GPT-4 access performed 17% worse on the independent exam than the control group, despite higher practice performance. View full evidence Study ID STUDY-BASTANI-2025 Sample ID SMPL-BASTANI-2025-N950 Study title Generative AI without guardrails can harm learning: Evidence from high school mathematics Source title Generative AI without guardrails can harm learning: Evidence from high school mathematics Year 2025 Study design Randomized controlled trial Education level High school Population nearly a thousand high school math students in Turkey Sample size 950 Intervention Gpt4 Based Tutor Gpt Base Unguarded Comparison No Generative AI Control Outcome measure exam without access to AI resources after practice phase Effect / result negative_17_percent_on_independent_exam Effect direction Negative effect Relation to claim Contradict Duration In Class Study Sessions Method large-scale randomized controlled trial, practice phase then closed-book exam Strengths large_scale_rct；independent_exam_without_ai；arm_wise_design Limitations high_school_mathematics_not_university_programming；single_country Confounders Tool Design Difference Quality dimensions D1 Study design = 2 · D2 Sample quality = 2 · D3 Measurement validity = 2 · D4 Temporal strength = 1 · D5 Directness = 1 Quality score 8.0 Evidence level strong Applicability learner_match=partial_same_age_band_different_subject · subject_match=no_mathematics_vs_programming · tool_match=gpt4_chat_interface · scope=unguarded_general_chat_interface Confidence 0.75 Evidence status Supported Full claim Students with unguarded GPT-4 access performed 17% worse on the independent exam than the control group, despite higher practice performance. Source location https://www.pnas.org/doi/10.1073/pnas.2422633122 Verifiable URL https://www.pnas.org/doi/10.1073/pnas.2422633122	`S-2025-bastani`
`E-005`	Independent problem solving	Null effectClaim relation: Support	8	Guardrail design of the AI tutor (hints instead of answers, teacher-informed prompts) largely eliminated the negative learning effect. View full evidence Study ID STUDY-BASTANI-2025 Sample ID SMPL-BASTANI-2025-N950 Study title Generative AI without guardrails can harm learning: Evidence from high school mathematics Source title Generative AI without guardrails can harm learning: Evidence from high school mathematics Year 2025 Study design Randomized controlled trial Education level High school Population nearly a thousand high school math students in Turkey Sample size 950 Intervention Gpt4 Tutor With Teacher Designed Guardrails Comparison No Generative AI Control Outcome measure exam without access to AI resources after practice phase Effect / result negative effect essentially eradicated, no positive effect observed Effect direction Null effect Relation to claim Support Duration In Class Study Sessions Method large-scale randomized controlled trial, three arms Strengths direct_manipulation_of_tool_design；large_sample Limitations no_positive_learning_gain_even_with_guardrails；subject_mismatch Confounders Prompt Engineering Effort Quality dimensions D1 Study design = 2 · D2 Sample quality = 2 · D3 Measurement validity = 2 · D4 Temporal strength = 1 · D5 Directness = 1 Quality score 8.0 Evidence level strong Applicability learner_match=partial · subject_match=no · tool_match=guardrailed_tutor_design · scope=guardrail_design_principle_transferable Confidence 0.75 Evidence status Supported Full claim Guardrail design of the AI tutor (hints instead of answers, teacher-informed prompts) largely eliminated the negative learning effect. Source location https://www.pnas.org/doi/10.1073/pnas.2422633122 Verifiable URL https://www.pnas.org/doi/10.1073/pnas.2422633122	`S-2025-bastani`
`E-006`	Assignment score	Positive effectClaim relation: Support	8	Access to GPT-4 during practice improves task performance (48% for GPT Base, 127% for GPT Tutor) — but this task performance does not transfer to independent exam performance. View full evidence Study ID STUDY-BASTANI-2025 Sample ID SMPL-BASTANI-2025-N950 Study title Generative AI without guardrails can harm learning: Evidence from high school mathematics Source title Generative AI without guardrails can harm learning: Evidence from high school mathematics Year 2025 Study design Randomized controlled trial Education level High school Population nearly a thousand high school math students in Turkey Sample size 950 Intervention Gpt4 Tutor Access During Practice Comparison No Generative AI Control Outcome measure practice problem performance during study sessions Effect / result 48-127 percent improvement on practice problems Effect direction Positive effect Relation to claim Support Duration In Class Study Sessions Method randomized controlled trial with practice and closed-book exam phases Strengths same_study_compares_task_and_learning；large_sample Limitations subject_mismatch_mathematics Confounders Task Familiarity Quality dimensions D1 Study design = 2 · D2 Sample quality = 2 · D3 Measurement validity = 2 · D4 Temporal strength = 1 · D5 Directness = 1 Quality score 8.0 Evidence level strong Applicability learner_match=partial · subject_match=no · tool_match=gpt4 · scope=task_performance_vs_learning_separation Confidence 0.75 Evidence status Supported Full claim Access to GPT-4 during practice improves task performance (48% for GPT Base, 127% for GPT Tutor) — but this task performance does not transfer to independent exam performance. Source location https://www.pnas.org/doi/10.1073/pnas.2422633122 Verifiable URL https://www.pnas.org/doi/10.1073/pnas.2422633122	`S-2025-bastani`
`E-007`	Knowledge gain	Positive effectClaim relation: Support	6	ChatGPT as a formative feedback tool produced a significant positive impact on students' academic writing skills with positive student perceptions. View full evidence Study ID STUDY-MARZUKI-2024 Sample ID SMPL-MARZUKI-2024-N72 Study title Impact of ChatGPT on ESL students' academic writing skills Source title Impact of ChatGPT on ESL students' academic writing skills Year 2024 Study design Mixed methods Education level Undergraduate Population undergraduate ESL students at an Indian university Sample size 72 Intervention Chatgpt As Formative Feedback Tool Comparison Traditional Instruction Control Outcome measure writing tests with pre-post-delayed design Effect / result significant positive impact on writing skills Effect direction Positive effect Relation to claim Support Duration 6 Hours Intervention Method mixed methods intervention study, pre/post/delayed tests and focus groups Strengths delayed_post_test；mixed_methods_triangulation Limitations short_intervention_6_hours；single_institution；elite_private_university Confounders Self Selection Consent Quality dimensions D1 Study design = 1 · D2 Sample quality = 1 · D3 Measurement validity = 2 · D4 Temporal strength = 2 · D5 Directness = 0 Quality score 6.0 Evidence level moderate Applicability learner_match=yes_undergraduate · subject_match=no_writing_not_programming · tool_match=chatgpt · scope=formative_feedback_writing Confidence 0.55 Evidence status Supported Full claim ChatGPT as a formative feedback tool produced a significant positive impact on students' academic writing skills with positive student perceptions. Source location https://link.springer.com/article/10.1186/s40561-024-00295-9 Verifiable URL https://link.springer.com/article/10.1186/s40561-024-00295-9	`S-2024-marzuki`
`E-008`	Completion time	Positive effectClaim relation: Support	8	Professional developers with Copilot access completed a standardized coding task about 55% faster than the control group (RCT, n=95). View full evidence Study ID STUDY-PENG-2023 Sample ID SMPL-PENG-2023-N95 Study title The Impact of AI on Developer Productivity: Evidence from GitHub Copilot Source title The Impact of AI on Developer Productivity: Evidence from GitHub Copilot Year 2023 Study design Randomized controlled trial Education level Professional Developers Not Students Population 95 recruited professional developers completing a standardized coding task on a freelance platform Sample size 95 Intervention Access To Github Copilot During Task Comparison Control Group Without Copilot Outcome measure Time To Complete Http Server Implementation Effect / result ~55.8% faster task completion in Copilot group Effect direction Positive effect Relation to claim Support Duration Single Task Session Method online randomized controlled experiment with objective completion-time metric Strengths randomized_controlled_design；objective_completion_time_metric Limitations professional_population_not_students；single_task_ecology；preprint_not_peer_reviewed Confounders Task Familiarity；Platform Recruitment Self Selection Quality dimensions D1 Study design = 2 · D2 Sample quality = 2 · D3 Measurement validity = 2 · D4 Temporal strength = 1 · D5 Directness = 1 Quality score 8.0 Evidence level moderate Applicability learner_match=mismatch_professional_developers · subject_match=adjacent_web_development_task · tool_match=copilot_like_generative_ai · scope=task_performance_only_no_learning_outcome Confidence 0.6 Evidence status Supported Full claim Professional developers with Copilot access completed a standardized coding task about 55% faster than the control group (RCT, n=95). Source location https://doi.org/10.48550/arXiv.2302.06590 Verifiable URL https://doi.org/10.48550/arXiv.2302.06590	`S-2023-peng`
`E-009`	Code quality	Null effectClaim relation: Neutral	7	Systematic benchmark evaluation reports mixed quality results for Copilot-generated code relative to human code: correctness competitive on parts of the benchmark while security-relevant defects are documented. View full evidence Study ID STUDY-YETISTIREN-2023 Sample ID SMPL-YETISTIREN-2023-BENCH Study title GitHub Copilot AI Pair Programmer: Asset or Liability? Source title GitHub Copilot AI Pair Programmer: Asset or Liability? Year 2023 Study design Observational Education level Not Applicable Code Artifacts Population Copilot-generated and human-written programs drawn from published benchmark datasets Intervention Copilot Generated Programs Comparison Human Written Programs On Same Benchmarks Outcome measure Correctness Security Maintainability Metrics On Benchmarks Effect / result mixed quality profile; no single-direction summary Effect direction Null effect Relation to claim Neutral Duration Not Applicable Artifact Study Method systematic empirical evaluation of generated code against human baselines on public benchmarks Strengths multi_dimensional_quality_metrics；reproducible_benchmark_protocol Limitations artifact_benchmark_not_classroom；no_learning_outcome；tool_version_from_2023 Confounders Benchmark Task Distribution Quality dimensions D1 Study design = 1 · D2 Sample quality = 2 · D3 Measurement validity = 2 · D4 Temporal strength = 1 · D5 Directness = 1 Quality score 7.0 Evidence level moderate Applicability learner_match=mismatch_no_learners_in_study · subject_match=introductory_adjacent_code_tasks · tool_match=copilot_like_generative_ai · scope=output_quality_only Confidence 0.55 Evidence status Supported Full claim Systematic benchmark evaluation reports mixed quality results for Copilot-generated code relative to human code: correctness competitive on parts of the benchmark while security-relevant defects are documented. Source location https://doi.org/10.1016/j.jss.2023.111734 Verifiable URL https://doi.org/10.1016/j.jss.2023.111734	`S-2023-yetistiren`
`E-010`	Assignment score	Positive effectClaim relation: Support	7	Codex produced passing-level solutions for roughly half to three-quarters of CS1 exam-style questions depending on the dataset, indicating substantial task-capability headroom available to novices. View full evidence Study ID STUDY-FINNIEANSLEY-2022 Sample ID SMPL-FINNIEANSLEY-2022-QSETS Study title Using GitHub Copilot to Solve Introductory Programming Problems Source title Using GitHub Copilot to Solve Introductory Programming Problems Year 2022 Study design Observational Education level University Year 1 Question Sets Population CS1 exam-style question sets answered by Codex and compared against published student score distributions Intervention Codex Answer Generation On CS1 Questions Comparison Published Student Cohort Score Distributions Outcome measure Pass Rate On CS1 Exam Style Questions Effect / result passing solutions on ~50-75% of questions across datasets Effect direction Positive effect Relation to claim Support Duration Not Applicable Capability Probe Method capability benchmark against published student distributions; reproducible question sets Strengths public_reproducible_question_sets；directly_relevant_task_domain Limitations tool_solves_task_does_not_equate_student_learning；codex_2021_model_version_outdated Confounders Question Leakage Into Training Data Possible Quality dimensions D1 Study design = 1 · D2 Sample quality = 2 · D3 Measurement validity = 2 · D4 Temporal strength = 1 · D5 Directness = 1 Quality score 7.0 Evidence level moderate Applicability learner_match=partial_measures_tool_not_students · subject_match=introductory_programming · tool_match=copilot_like_generative_ai · scope=tool_capability_headroom Confidence 0.55 Evidence status Supported Full claim Codex produced passing-level solutions for roughly half to three-quarters of CS1 exam-style questions depending on the dataset, indicating substantial task-capability headroom available to novices. Source location https://dl.acm.org/doi/10.1145/3545945.3569830 Verifiable URL https://dl.acm.org/doi/10.1145/3545945.3569830	`S-2022-finnie-ansley`
`E-011`	Metacognition	Positive effectClaim relation: Support	7	Controlled comparisons find LLM-generated code explanations comparable to (in places better than) student-authored explanations, suggesting viability as explanatory scaffold material rather than as a replacemen… View full evidence Study ID STUDY-EXPLCOMP-2023 Sample ID SMPL-EXPLCOMP-2023-RATINGS Study title Comparing Code Explanations Created by Students and Large Language Models Source title Comparing Code Explanations Created by Students and Large Language Models Year 2023 Study design Observational Education level University Introductory Population Student-produced versus LLM-produced explanations of short programs under controlled comparison Intervention LLM Generated Code Explanations Comparison Student Generated Explanations Of Same Programs Outcome measure Rated Explanation Quality And Comprehensibility Effect / result comparable-or-better rated quality vs student explanations Effect direction Positive effect Relation to claim Support Duration Single Session Ratings Method controlled comparison with blind rating of explanation pairs Strengths controlled_pairwise_comparison；learning_process_relevant_construct Limitations short_term_ratings_not_learning_gains；small_program_snippets_ecology Confounders Rating Criteria Subjectivity Quality dimensions D1 Study design = 1 · D2 Sample quality = 2 · D3 Measurement validity = 2 · D4 Temporal strength = 1 · D5 Directness = 1 Quality score 7.0 Evidence level moderate Applicability learner_match=partial_scaffold_material_only · subject_match=introductory_programming · tool_match=llm_explanations · scope=scaffold_quality_not_effectiveness Confidence 0.55 Evidence status Supported Full claim Controlled comparisons find LLM-generated code explanations comparable to (in places better than) student-authored explanations, suggesting viability as explanatory scaffold material rather than as a replacement for student explanation practice. Source location https://dl.acm.org/doi/10.1145/3587102.3588785 Verifiable URL https://dl.acm.org/doi/10.1145/3587102.3588785	`S-2023-explanations-compare`
`E-012`	Over-reliance	Negative effectClaim relation: Contradict	7	Despite faster first-task completion, participants struggled to understand and debug AI-generated solutions and reported low ownership of the final program - documenting metacognitive and dependence risks that… View full evidence Study ID STUDY-VAITHILINGAM-2022 Sample ID SMPL-VAITHILINGAM-2022-N24 Study title Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Source title Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Year 2022 Study design Qualitative Education level Mixed Cs Students And Professionals Population 24 participants in a within-subjects usability study of Copilot-style tools Sample size 24 Intervention Copilot Assisted Program Writing Comparison Within Subject Baseline Without Tool Outcome measure Understanding Ownership And Debugging Reports Effect / result documented comprehension/ownership difficulties despite speed gain Effect direction Negative effect Relation to claim Contradict Duration Single Session Method within-subject usability study with tasks, observation and interviews Strengths rich_qualitative_process_data；constructs_missed_by_speed_metrics Limitations small_n_24；single_session；self_reported_understanding Confounders Participant AI Familiarity Quality dimensions D1 Study design = 1 · D2 Sample quality = 2 · D3 Measurement validity = 2 · D4 Temporal strength = 1 · D5 Directness = 1 Quality score 7.0 Evidence level moderate Applicability learner_match=partial_includes_cs_students · subject_match=programming_adjacent · tool_match=copilot_like_generative_ai · scope=risk_identification Confidence 0.55 Evidence status Contradicted Full claim Despite faster first-task completion, participants struggled to understand and debug AI-generated solutions and reported low ownership of the final program - documenting metacognitive and dependence risks that pure speed metrics miss. Source location https://dl.acm.org/doi/10.1145/3491101.3519665 Verifiable URL https://dl.acm.org/doi/10.1145/3491101.3519665	`S-2022-vaithilingam`

03 Evidence Quality, Counterevidence & Method Audit

Examine why evidence is credible, where it conflicts, and which conclusions require downgrading.

Audit target: overall

Concern

Control groupMet

All three studies include a no-AI control group.

RandomizationMet

Kazemitabaar 2023 and Bastani 2025 use randomized assignment.

Pre-testMet

Kazemitabaar 2023 has a pre-study evaluation; Bastani 2025 measures baseline covariates.

Post-testMet

Immediate post-tests present in all studies.

Retention testPartial

Kazemitabaar 2023 has 1-week retention; Bastani 2025 has no delayed test; Marzuki 2024 has delayed test.

Transfer testPartial

Kazemitabaar 2023 code-modification task is transfer-adjacent; no full no-AI transfer task.

Sample biasMet

Bastani 2025 nearly 1000 students; Kazemitabaar 2023 small (69) young sample.

Self-selection biasPartial

Marzuki 2024 consent-based participation risks self-selection.

Measurement validityPartial

Practice/task performance is not equated to learning; independent exams present in Bastani 2025 only.

ConfoundersPartial

Prior programming competency interacts with AI benefit in Kazemitabaar 2023.

Instructor effectN/A

Kazemitabaar 2023 is self-paced; classroom studies may carry instructor effects.

Novelty effectPartial

Short interventions likely inflate engagement; none of the studies controlled for novelty.

Tool-version effectN/A

Single tool versions studied; rapid tool change limits durability.

AI usage policyPartial

Bastani 2025 explicitly contrasts unguarded vs guardrailed usage policies.

AttritionPartial

Marzuki 2024 reports attrition; others not detailed.

Task vs learning guard:

Bastani 2025 demonstrates the danger of equating the two: +48-127% practice performance coexisted with -17% independent exam performance.

Tribunal note:

Expand full explanation

Disagreement comes from outcome separation (task vs learning), tool design (guarded vs unguarded), and population (K-12 / professionals vs university). Task-performance evidence is consistently positive across randomized and benchmark studies; the only study measuring independent performance after AI removal shows harm without guardrails; usability and artifact studies add dependence and quality caveats rather than resolving the learning question.

Decision: Pilot · Confidence: Moderate

Can claim

AI coding assistants reliably increase task performance during training (completion speed, correctness) — E-001, E-006.
E-001E-006
Unguarded generative AI access can harm independent problem solving when access is removed — E-004.
E-004
Guardrail design (hints instead of answers) substantially mitigates the negative learning effect — E-005.
E-005
Task performance gains do not automatically imply learning gains — E-004 vs E-006 (within-study contrast).
E-004E-006
Tool capability is substantial: Codex solves roughly half to three-quarters of CS1 exam-style questions — E-010.
E-010
Professional-developer RCT shows ~55% faster task completion with Copilot; directness limited by professional population — E-008.
E-008
LLM code explanations rate comparable to student-authored explanations, viable as scaffold material — E-011.
E-011

Cannot yet claim

Whether AI coding assistants improve or preserve actual programming learning in university novices — no direct university-level RCT in reviewed set [无直接证据]
Whether one-week neutral retention (Kazemitabaar 2023) extends to a semester — E-003.
E-003
Whether benchmark quality findings (E-009) and explanation-quality ratings (E-011) translate into classroom learning gains.
E-009E-011
How comprehension/ownership difficulties documented in usability studies (E-012) behave over a full semester with guardrails.
E-012

Contradicted claims

The claim 'AI tools always improve learning' is contradicted by E-004 (unguarded access, -17% independent exam).
E-004
The claim 'speed gains equal learning gains' is contradicted by the task-vs-learning separation across E-001/E-006/E-008 vs E-004.
E-001E-006E-008E-004

Missing evidence

RCT of AI coding assistants in university programming courses with retention and no-AI transfer tests.
Studies varying AI usage policy within the same course.
Longitudinal data on AI dependency beyond one course.
Peer-reviewed replication of the professional speed RCT (Peng et al. remains a preprint).

EvidenceFlow Protocol

Tribunal infographic

C-001

AI coding assistants significantly increase task completion speed and completion rate during training.

↓

E-001Completion timePositive effectclaim relation: SupportQ 9.0→S-2023-kazemitabaar

C-002

Access to AI code generation did not decrease performance on manual code-modification tasks.

↓

E-002Independent problem solvingNull effectclaim relation: NeutralQ 7.0→S-2023-kazemitabaar

C-003

One week after training, retention differences between Codex and baseline groups did not reach statistical significance.

↓

E-003RetentionNull effectclaim relation: NeutralQ 9.0→S-2023-kazemitabaar

C-004

Students with unguarded GPT-4 access performed 17% worse on the independent exam than the control group, despite higher practice performance.

↓

E-004Independent problem solvingNegative effectclaim relation: ContradictQ 8.0→S-2025-bastani

C-005

Guardrail design of the AI tutor (hints instead of answers, teacher-informed prompts) largely eliminated the negative learning effect.

↓

E-005Independent problem solvingNull effectclaim relation: SupportQ 8.0→S-2025-bastani

C-006

Access to GPT-4 during practice improves task performance (48% for GPT Base, 127% for GPT Tutor) — but this task performance does not transfer to independent exam performance.

↓

E-006Assignment scorePositive effectclaim relation: SupportQ 8.0→S-2025-bastani

C-007

ChatGPT as a formative feedback tool produced a significant positive impact on students' academic writing skills with positive student perceptions.

↓

E-007Knowledge gainPositive effectclaim relation: SupportQ 6.0→S-2024-marzuki

C-008

Professional developers with Copilot access completed a standardized coding task about 55% faster than the control group (RCT, n=95).

↓

E-008Completion timePositive effectclaim relation: SupportQ 8.0→S-2023-peng

C-009

Systematic benchmark evaluation reports mixed quality results for Copilot-generated code relative to human code: correctness competitive on parts of the benchmark while security-relevant defects are documented.

↓

E-009Code qualityNull effectclaim relation: NeutralQ 7.0→S-2023-yetistiren

C-010

Codex produced passing-level solutions for roughly half to three-quarters of CS1 exam-style questions depending on the dataset, indicating substantial task-capability headroom available to novices.

↓

E-010Assignment scorePositive effectclaim relation: SupportQ 7.0→S-2022-finnie-ansley

C-011

Controlled comparisons find LLM-generated code explanations comparable to (in places better than) student-authored explanations, suggesting viability as explanatory scaffold material rather than as a replacement for student explanation prac…

Expand full explanation

↓

E-011MetacognitionPositive effectclaim relation: SupportQ 7.0→S-2023-explanations-compare

C-012

Despite faster first-task completion, participants struggled to understand and debug AI-generated solutions and reported low ownership of the final program - documenting metacognitive and dependence risks that pure speed metrics miss.

↓

E-012Over-relianceNegative effectclaim relation: ContradictQ 7.0→S-2022-vaithilingam

What this means: Every important claim must resolve to Evidence IDs and original sources.

04 Applicability & Teaching Action

Connect applicability, guardrails and teaching actions to specific evidence.

Target population: university first-year computer science students learning C programming for the first time

Target context: 16-week lecture-lab course, 60 students, TA support, offline

Suitable for: pilot in first-year C course with guardrailed usage policy

Not suitable for: unrestricted AI adoption without usage policy

Conditions:

guardrailed AI usage policy (hints not answers, modeled on GPT Tutor arm)
no-AI transfer assessment
TA support

Claims beyond the evidence boundary

Claiming 'AI coding assistants improve learning' exceeds the boundary: direct learning-effect evidence is missing.
Claiming 'AI works for everyone' exceeds the boundary: population and subject mismatch.

Evidence

AI coding assistants reliably increase task performance during training (completion speed, correctness) — E-001, E-006.

Applicability

pilot in first-year C course with guardrailed usage policy

Decision

Pilot

Guardrails

AI usage is allowed in three explicitly graded modes (explain / collaborate / no-AI-transfer).…

Expand full explanation

Stop conditions

transfer-test scores drop significantly below baseline cohort expectations; widespread integrity violations in reasoning traces;…

Expand full explanation

Evaluation

treatment group shows non-inferior independent problem solving (delta within 5%) AND superior or equal retention AND AI dependency index below thresho…

Expand full explanation

Target learners: university first-year C programming students (60-student lecture-lab class) · Pilot duration: 8_weeks

AI usage policy: AI usage is allowed in three explicitly graded modes (explain / collaborate / no-AI-transfer). Copying unexamined AI output is an academic integrity violation and is assessed via the reasoning-trace requirement.

Phase 1 — Independent Foundation

AI rule: full code generation forbidden; AI allowed only for conceptual explanations

Activities:

baseline assessment
first 2 weeks of assignments completed without any AI code generation

Outcome check: baseline task-performance and independent-problem-solving measures

Phase 2 — Explain, Don't Solve

AI rule: AI may explain but must not produce full solutions

Activities:

weeks 3-4: AI allowed to explain errors, concepts and debugging approaches

Outcome check: mid-pilot quiz without AI

Phase 3 — Structured Collaboration

AI rule: partial code generation allowed; mandatory written explanation of key logic; submission requires reasoning trace

Activities:

weeks 5-7: AI allowed to generate partial code; students must explain every AI-generated block in their own words

Outcome check: weekly lab completion and code-quality rubric

Phase 4 — Transfer Check

AI rule: no AI access during transfer assessment

Activities:

week 8: new programming task completed in a no-AI environment

Outcome check: transfer test score, independent problem solving

Stop conditions

transfer-test scores drop significantly below baseline cohort expectations
widespread integrity violations in reasoning traces
AI dependency signals exceed threshold in risk metrics
TA/teacher workload becomes unsustainable

Intervention timeline infographic

05 Pilot, Evaluation & Stop Conditions

Validate the pilot with independent learning outcomes and pre-specified stop conditions.

Research question: In a first-year C programming course, does a guardrailed AI coding assistant (explain-only → structured collaboration) improve independent problem solving without increasing AI dependency, compared to no-AI instruction?

Baseline: week-1 no-AI programming quiz (independent problem solving, completion time)

Post test: week-8 no-AI programming quiz (independent problem solving, code quality)

Retention: final exam (week 16) — delayed measure 8 weeks after pilot

Transfer: week-8 novel programming task in a strictly no-AI environment

Process metrics

weekly lab completion rate
AI usage logs: prompts submitted, copied blocks, reasoning traces
help-seeking behavior counts

Learning metrics

independent problem solving (no-AI quizzes)
code quality rubric
retention on final exam
transfer task score

Risk metrics

AI dependency index (usage without understanding, from reasoning-trace quality)
academic integrity violations
self-reported over-reliance
false confidence (post-quiz confidence vs actual score)

Success threshold: treatment group shows non-inferior independent problem solving (delta within 5%) AND superior or equal retention AND AI dependency index below threshold; if independent problem solving declines >10%, the pilot is judged unsuccessful regardless of task-performance gains.

Analysis plan: pre-registered comparison of treatment vs comparison sections on baseline-adjusted learning metrics (ANCOVA); task-performance metrics reported separately from learning metrics; subgroup analysis by prior programming competency; stop-condition monitoring at weeks 3, 5, 7.

Evaluation design infographic

Benchmark visual suppressed

result.json carries no benchmark.baselines, so this visual is omitted; see the standalone benchmark report.

06 Sources, Traceability & Appendix

Preserve original sources, URLs, evidence IDs and retrieval metadata for auditability.

Source list

ID	Title	Year	Authority	Verifiable location
`S-2023-kazemitabaar`	Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming DOI verified View source & provenance Original title Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming Year 2023 Authority Tier 1 DOI-verified paper Source location https://dl.acm.org/doi/10.1145/3544548.3580919 Verifiable link https://dl.acm.org/doi/10.1145/3544548.3580919	2023	Tier 1 DOI-verified paper	https://dl.acm.org/doi/10.1145/3544548.3580919
`S-2025-bastani`	Generative AI without guardrails can harm learning: Evidence from high school mathematics DOI verified View source & provenance Original title Generative AI without guardrails can harm learning: Evidence from high school mathematics Year 2025 Authority Tier 1 DOI-verified paper Source location https://www.pnas.org/doi/10.1073/pnas.2422633122 Verifiable link https://www.pnas.org/doi/10.1073/pnas.2422633122	2025	Tier 1 DOI-verified paper	https://www.pnas.org/doi/10.1073/pnas.2422633122
`S-2024-marzuki`	Impact of ChatGPT on ESL students' academic writing skills DOI verified View source & provenance Original title Impact of ChatGPT on ESL students' academic writing skills Year 2024 Authority Tier 1 DOI-verified paper Source location https://link.springer.com/article/10.1186/s40561-024-00295-9 Verifiable link https://link.springer.com/article/10.1186/s40561-024-00295-9	2024	Tier 1 DOI-verified paper	https://link.springer.com/article/10.1186/s40561-024-00295-9
`S-2023-peng`	The Impact of AI on Developer Productivity: Evidence from GitHub Copilot DOI verified View source & provenance Original title The Impact of AI on Developer Productivity: Evidence from GitHub Copilot Year 2023 Authority Tier2 Academic Database Source location https://arxiv.org/abs/2302.06590 Verifiable link https://doi.org/10.48550/arXiv.2302.06590	2023	Tier2 Academic Database	https://doi.org/10.48550/arXiv.2302.06590
`S-2023-yetistiren`	GitHub Copilot AI Pair Programmer: Asset or Liability? DOI verified View source & provenance Original title GitHub Copilot AI Pair Programmer: Asset or Liability? Year 2023 Authority Tier 1 DOI-verified paper Source location https://doi.org/10.1016/j.jss.2023.111734 Verifiable link https://doi.org/10.1016/j.jss.2023.111734	2023	Tier 1 DOI-verified paper	https://doi.org/10.1016/j.jss.2023.111734
`S-2022-finnie-ansley`	Using GitHub Copilot to Solve Introductory Programming Problems DOI verified View source & provenance Original title Using GitHub Copilot to Solve Introductory Programming Problems Year 2022 Authority Tier 1 DOI-verified paper Source location https://dl.acm.org/doi/10.1145/3545945.3569830 Verifiable link https://dl.acm.org/doi/10.1145/3545945.3569830	2022	Tier 1 DOI-verified paper	https://dl.acm.org/doi/10.1145/3545945.3569830
`S-2023-explanations-compare`	Comparing Code Explanations Created by Students and Large Language Models DOI verified View source & provenance Original title Comparing Code Explanations Created by Students and Large Language Models Year 2023 Authority Tier 1 DOI-verified paper Source location https://dl.acm.org/doi/10.1145/3587102.3588785 Verifiable link https://dl.acm.org/doi/10.1145/3587102.3588785	2023	Tier 1 DOI-verified paper	https://dl.acm.org/doi/10.1145/3587102.3588785
`S-2022-vaithilingam`	Expectation vs. Experience: Evaluating the Usability of Code Generation Tools DOI verified View source & provenance Original title Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Year 2022 Authority Tier 1 DOI-verified paper Source location https://dl.acm.org/doi/10.1145/3491101.3519665 Verifiable link https://dl.acm.org/doi/10.1145/3491101.3519665	2022	Tier 1 DOI-verified paper	https://dl.acm.org/doi/10.1145/3491101.3519665

Fetch provenance

Search provider: n/a

No per-source fetch records (sources provided directly by the research pipeline).