Automating Storyline Doesn't Make You a Learning Engineer

A software simulation breaks when the software changes. Someone has to update the steps, screenshots, triggers, and feedback. Of course teams want AI to help. That is a real maintenance problem, and it is worth solving.

01The automation opportunity is real

Devlin Peck recently described a team interested in an agent that detects software changes, updates Storyline simulations, and sends the work to instructional designers and SMEs for review. The human approval step matters. This is not an argument against that idea. It is an argument for asking what happens after the agent detects a change. A changed button label may require a simple text update. A changed approval sequence may invalidate an entire decision path. Those are not equivalent maintenance tasks.

02Maintenance is not instructional validation

An agent can identify differences between versions of an interface. It cannot establish learning effectiveness merely by producing a matching new screen. The instructional question is whether the experience still rehearses the decisions people must make at work.

Automating the production of learning experiences is not the same as engineering better learning systems. Before approving an update, a designer should be able to trace the changed workflow to a performance requirement, learning objective, decision point, and validation test.

03The missing questions

What evidence shows that the simulation was the right intervention in the first place? Which changes are cosmetic, and which affect behavior? What happens when the agent misses a critical change? Who owns the acceptance criteria? And does the total time saved survive SME review, rework, and testing? Without answers, a faster maintenance pipeline can also become a faster way to preserve an ineffective solution.

04Build a decision system, not just an update bot

Imagine a bounded workflow: compare approved software versions; map differences to tasks and decisions; flag potential instructional impact; present evidence and uncertainty; require human approval; then generate a change plan and test cases. No silent publishing. No invented confidence masquerading as verification. That is the direction we're exploring in The Lab with a small, transparent experiment. It starts with a fictional workflow, not a promise that AI can safely rewrite every Storyline file.

05Test the claim

We'll compare manual and assisted change analysis using the same scenarios. We will measure critical-change recall, false alarms, review time, corrections, and whether the resulting practice still targets the right decisions. A demo is not evidence of impact. A measured experiment is a better beginning. Devlin's example raises a useful possibility. The challenge for learning designers is to push it beyond production efficiency and into accountable learning-system decisions. Explore the Simulation Maintenance experiment or read AI Can Build a Course. Can It Maintain One?.

Discussion reference: Devlin Peck's LinkedIn post. This article discusses the proposal, not a verified implementation or its results.