DesignPrompt: Using Multimodal Interaction for Design Exploration with Generative AI - Archive ouverte HAL
Communication Dans Un Congrès Année : 2024

DesignPrompt: Using Multimodal Interaction for Design Exploration with Generative AI

Xiaohan Peng
Janin Koch
Wendy E Mackay

Résumé

Visually oriented designers often struggle to create effective generative AI (GenAI) prompts. A preliminary study identified specific issues in composing and fine-tuning prompts, as well as needs in accurately translating intentions into rich input. We developed DesignPrompt, a moodboard tool that lets designers combine multiple modalities — images, color, text — into a single GenAI prompt and tweak the results. We ran a comparative structured observation study with 12 professional designers to better understand their intent expression, expectation alignment and transparency perception using DesignPrompt and text input GenAI. We found that multimodal prompt input encouraged designers to explore and express themselves more effectively. Designer’s interaction preferences change according to their overall sense of control over the GenAI and whether they are seeking inspiration or a specific image. Designers developed innovative uses of DesignPrompt, including developing elaborate multimodal prompts and creating a multimodal prompt pattern to maximize novelty while ensuring consistency.
Fichier principal
Vignette du fichier
DIS_24_DesignPrompt_Author_Version.pdf (7.76 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Licence
Copyright (Tous droits réservés)

Dates et versions

hal-04652020 , version 1 (17-07-2024)

Licence

Copyright (Tous droits réservés)

Identifiants

Citer

Xiaohan Peng, Janin Koch, Wendy E Mackay. DesignPrompt: Using Multimodal Interaction for Design Exploration with Generative AI. DIS 2024 - Designing Interactive Systems Conference, ACM, Jul 2024, Copenhagen, Denmark. pp.1-15, ⟨10.1145/3643834.3661588⟩. ⟨hal-04652020⟩
129 Consultations
107 Téléchargements

Altmetric

Partager

More