20 Nov 2026
Performing with AI isn't learning: new evidence from consultants and 52 K-12 studies. Plus: why your evals and A/B tests can disagree
LDPro teamSTAFF
Welcome
Welcome to edition 3 of the LDPro Newsletter. At LDPro we think learning should be AI-driven, procedural, and always evolving, with people still in charge of quality. This week's theme is a distinction every L&D team should build into its measurement: doing better with AI is not the same as getting better because of AI. Every claim below links to its source, so you can check the detail yourself.
Research: learning evidence
GenAI as an "exoskeleton": consultants performed better with ChatGPT, but didn't learn the skill
A peer-reviewed randomised trial in Nature Human Behaviour (published 26 Oct 2026) gave 487 management consultants ChatGPT access and training for data-science tasks outside their expertise. The study's authors include researchers from BCG.
With ChatGPT, consultants outscored controls on three tasks, though they still didn't match expert data scientists.
Afterwards, without ChatGPT, they were no better than controls at answering technical questions.
Why it matters: this is adult, workplace evidence, close to corporate L&D. "Performance with AI" and "learning" are different outcomes, and a programme that only measures the first can look like a success while building no lasting capability.
Caveats: one firm, one skill domain (data science) and short-term tasks. The data is available only for non-commercial academic review.
Try this: in your pilots, separate "assisted task performance" from "unassisted performance after a delay". Design lessons so learners retrieve and produce on their own before AI help kicks in.
Stanford SCALE's evidence review now covers 52 causal studies of AI in K-12
Stanford SCALE's November 2026 evidence review (published 10 Nov 2026) grew its base of causal studies from 20 papers in its March 2026 review to 52. Key findings:
AI helps more when students use it to understand material than to produce work, and offloading tasks to AI can reduce learning.
Positive learner experiences often don't translate into better outcomes.
Tool features and implementation may matter more than tool type. Interactive formats, adaptive sequencing and guidance on how to use the tool show promise.
AI drafts helped educators give more and better feedback without extra time. But one broad teacher-support chatbot lowered achievement for students of lower-performing teachers.
Why it matters: it's a credible, neutral summary to bring into design decisions and stakeholder conversations, and a clear warning that satisfaction scores aren't evidence of learning.
Caveats: it covers K-12 studies published Oct 2025 to Jun 2026, and the authors say it shouldn't be treated as broadly generalisable. It's a synthesis, not new trials.
Try this: don't make learner satisfaction your main pilot metric. Adopt "use AI to learn, not to produce" as a design principle for learner-facing AI features.
Research: evaluation
Why offline benchmarks and live A/B tests can rank models differently
A new arXiv preprint (Nov 2026) by Hannah Li shows formally that when users adapt to a model (changing their prompts and the difficulty of what they attempt after seeing its responses), offline benchmarks and live A/B tests measure different things, and can disagree on which model is better. The paper gives a sufficient condition for when they agree, and links user adaptation to reward hacking and novelty effects.
Why it matters: any team that improves its learning content or models over time will lean on both a fixed evaluation set and live learner data. This explains why the two can conflict, and why learner behaviour is part of the result.
Caveats: a single-author preprint, not peer reviewed. The main results are theoretical, built on a stylised model.
Try this: when you compare a new model or content version against the old one, log how learners behave (question types, difficulty attempted, help requests) alongside outcome scores. Don't ship on offline scores alone.
Models
Qwen4-Max is generally available, API only
Following the preview covered in edition 2, Alibaba launched Qwen4-Max on 16 Nov 2026 on QwenCloud (API and Token Plan):
A 2.4T-parameter mixture-of-experts model with 71B active parameters, a 1M-token context, and image and video input.
List price: US$1.50 per 1M input tokens and $4.50 per 1M output tokens, with cached input at $0.075 per 1M.
Alibaba claims about 60% lower cost per task than Qwen3.8-Max on its own Work360 benchmark (vendor-reported).
There are still no open weights. Alibaba says only that smaller open-weight Qwen4 models are coming "soon", with no sizes or dates.
Why it matters: it isn't something you can fine-tune yourself. It could serve as a hosted "teacher" model for drafting or grading learning content.
Caveats: benchmarks and cost claims are Alibaba's own. If you plan to use its outputs to train another model, check QwenCloud's terms first.
IBM's Granite 4.1 LLMON preview: small Apache 2.0 models built to resist prompt injection
IBM released Granite 4.1 LLMON preview models (12 Nov 2026) in 3B and 8B sizes (8B model card), under Apache 2.0. They're trained on a new markup format that strictly separates trusted instructions from untrusted data such as documents, user content and tool results. IBM reports that prompt-injection attack success drops to nearly zero on some benchmarks, while general capability holds relative to Granite 4.1 Instruct.
Why it matters: any pipeline that turns uploaded documents into courses is exposed to instructions hidden in those documents. This is a concrete design pattern for that risk.
Caveats: IBM calls these experimental previews, not for production, and the results are IBM's own.
Try this: whatever model you use, keep source documents strictly in a "data" role, and add a few documents with hidden instructions to your evaluation set.
The takeaway for L&D teams
Two independent pieces of evidence this week point the same way: AI can lift performance in the moment without building lasting skill, and learners can enjoy a tool that doesn't help them learn. The fix is in the measurement. Test without AI, after a delay, and watch how learner behaviour changes, not just the scores. That's the kind of always-evolving, evidence-led learning we're building at LDPro.
Sources are linked inline. Preprints and vendor-reported figures are labelled as such. The LDPro Newsletter is a digest, not an endorsement of any model or vendor.
Questions about this?
Comments are off while the community is read-only. Ask us and we reply within 48 hours.
Ask us