Gen AI Workflows

Co-designing Characters and Storyboards with GenAI and people with Parkinson's

2024-2025

Prompt Engineering, Human-AI interaction, Digital Health, Storyboarding, TORUS project

GenAI tools like Midjourney can rapidly produce large volumes of visual content from text prompts, including photorealistic scenes with multiple consistent characters situated in realistic context. As such, they are being actively explored as tools for character and storyboard creation by UX professionals and researchers. This work is a prompt engineering design experiment involving myself and a team of Parkinson’s patients, whereby we co-designed AI-generated characters to develop storyboards depicting interaction scenarios of a home monitoring system (part of EPSRC-funded TORUS project).

In principle, GenAI tools offers clear opportunities for cognitive offloading and job effort reduction to rapidly create materials for design teams. Moreover, as GenAI opens new avenues for collaborative content generation through co-shaping prompts and output, further opportunities for teamwork emerge that include non-experts. As part of doing research within digital the health domain, I involved people living with Parkinson’s in co-creating GenAI characters could ensure their values and experiences are better reflected, leading to richer, more respectful and empowering depictions.

The design experiment concluded with the design of an interactive storyboard (in Figma) to be further used to conduct online studies. Myself and my co-authors authored a paper, presenting findings from the three-week design experiment, reflecting on key learnings from the design process, drawing on theories of distributed cognition to unpack some of the complexities at play in the use of GenAI tools for character and storyboard creation. We particularly focus on the socially distributed nature of the cognitive activity, addressing the ways in which a GenAI tool can increase cognitive demand through increasing complexity of the task, and through a lack of appropriately socialised cognition (or Theory of Mind).

Above: The final storyboard - Figma prototype.

Above: Short video-clip showcasing part of the storyboard



Above: The design experiment - a diagram showing the amount of content generated and key design phases.



Above: An example of MidJourney prompting exploration.

In the paper we report on findings from studying and critically reflecting upon the design process and outcomes, seeking to form implications for cognitive tools to support groups and workflows using GenAI for characters and storyboard. In the paper, we discuss how cognitive loading/ offloading took place within different content generation phases; on AI hallucination and how it impacted cognitive load, design fixation, and creativity. Moreover, we unpacks the dynamics and complexities of human-AI collaboration as a creative co-regulation process that evolved as part of employing diverse prompting and tool-use strategies. We unpack how GenAI can support, distort, or reconfigure human values, and the potential cognitive biases at play. We highlight the mismatch between human values, intentions, and the intuitive understanding of social context, and the language and output of the algorithm, and point to potential solutions and tools to mitigate issues.

This work is seeking to inform cognitive workflows within teams involving GenAI for storyboard-led research.