Sangho Suh

Students often take digital notes during live lectures, but current methods can be slow when capturing information from lecture slides or the instructor’s speech, and require them to focus on their devices, leading to distractions and missing important details. This paper explores supporting live lecture note-taking with mixed reality (MR) to quickly capture lecture information and take notes while staying engaged with the lecture. A survey and interviews with university students revealed common note-taking behaviors and challenges to inform the design. We present MaRginalia to provide digital note-taking with a stylus tablet and MR headset. Students can take notes with an MR representation of the tablet, lecture slides, and audio transcript without looking down at their device. When preferred, students can also perform detailed interactions by looking at the physical tablet. We demonstrate the feasibility and usefulness of MaRginalia and MR-based note-taking in a user study with 12 students.

Novice programmers are increasingly relying on Large Language Models (LLMs) to generate code for learning programming concepts. However, this interaction can lead to superficial engagement, giving learners an illusion of learning and hindering skill development. To address this issue, we conducted a systematic design exploration to develop seven cognitive engagement techniques aimed at promoting deeper engagement with AI-generated code. In this paper, we describe our design process, the initial seven techniques and results from a between-subjects study (N=82). We then iteratively refined the top techniques and further evaluated them through a within-subjects study (N=42). We evaluate the friction each technique introduces, their effectiveness in helping learners apply concepts to isomorphic tasks without AI assistance, and their success in aligning learners’ perceived and actual coding abilities. Ultimately, our results highlight the most effective technique: guiding learners through the step-by-step problem-solving process, where they engage in an interactive dialog with the AI, prompting what needs to be done at each stage before the corresponding code is revealed.

This workshop provides a forum to discuss, brainstorm, and prototype the next generation of interfaces that leverage the dynamic experiences enabled by recent advances in AI and the generative capabilities of foundation models. These models simplify complex tasks by generating outputs in various representations (e.g., text, images, videos) through diverse input modalities like natural language, voice, and sketch. They interpret user intent to generate and transform representations, potentially changing how we interact with information and express ideas. Inspired by this potential, technologists, theorists, and researchers are exploring new forms of interaction by building demos and communities dedicated to concretizing and advancing the vision of working with dynamic abstractions. This UIST workshop provides a timely space to discuss AI’s impact on how we might design and use cognitive tools (e.g., languages, notations, diagrams). We will explore the challenges, critiques, and opportunities of this space by thinking through and prototyping use cases across various domains.

Research shows that many people learn about privacy and security risks from anecdotal stories shared by others, which influence their perceptions and actions. However, no studies have been conducted with children, who may also gain significant knowledge about security and privacy from peers, trusted adults, and social networks. To promote the creation and sharing of privacy stories, we created PrivacyToon, a conceptdriven storytelling tool that facilitates the visual production of privacy stories and visualizations. The tool provides users with creative and technical drawing support, where a comic story can be created, downloaded, and shared while improving the reflection of privacy issues in the process. A comic authoring tool centers users in the creation process to tell their own narratives that express their lived digital experiences orlessons from stories. We discuss our ongoing research on PrivacyToon and its potential as a security and privacy learning interface for children.

Thanks to their generative capabilities, large language models (LLMs) have become an invaluable tool for creative processes. These models have the capacity to produce hundreds and thousands of visual and textual outputs, offering abundant inspiration for creative endeavors. But are we harnessing their full potential? We argue that current interaction paradigms fall short, guiding users towards rapid convergence on a limited set of ideas, rather than empowering them to explore the vast latent design space in generative models. To address this limitation, we propose a framework that facilitates the structured generation of design space in which users can seamlessly explore, evaluate, and synthesize a multitude of responses. We demonstrate the feasibility and usefulness of this framework through the design and development of an interactive system, Luminate, and a user study with 14 professional writers. Our work advances how we interact with LLMs for creative tasks, introducing a way to harness the creative potential of LLMs.

Programmers increasingly rely on Large Language Models (LLMs) for code generation. However, current LLM-driven code assistants lack sufficient scaffolding to help programmers construct intentions from their overarching goals, translate these intentions into natural language prompts, and further refinement on the intention by editing prompts or code. To address this gap, we adopted an iterative design process to gain insights into programmers’ strategies when using LLMs for programming. Building on our findings, we created CoLadder a system that supports programmers by facilitating hierarchical task decomposition, direct code segment manipulation, and result evaluation during prompt authoring. A user study with 12 experienced programmers showed that CoLadder is effective in helping programmers externalize their problem-solving intentions flexibly, improving their ability to evaluate and modify code across various abstraction levels, from goal to final code implementation.

People are increasingly turning to large language models (LLMs) for complex information tasks like academic research or planning a move to another city. However, while they often require working in a nonlinear manner — e.g., to arrange information spatially to organize and make sense of it, current interfaces for interacting with LLMs are generally linear to support conversational interaction. To address this limitation and explore how we can support LLM-powered exploration and sensemaking, we developed Sensecape, an interactive system designed to support complex information tasks with an LLM by enabling users to (1) manage the complexity of information through multilevel abstraction and (2) switch seamlessly between foraging and sensemaking. Our within-subject user study reveals that Sensecape empowers users to explore more topics and structure their knowledge hierarchically, thanks to the externalization of levels of abstraction. We contribute implications for LLM-based workflows and interfaces for information tasks.

Science writers commonly use extended metaphors to communicate unfamiliar concepts in a more accessible way to a wider audience. However, creating metaphors for science writing is challenging even for professional writers; according to our formative study (n=6), finding inspiration and extending metaphors with coherent structures were critical yet significantly challenging tasks for them. We contribute Metaphorian, a system that supports science writers with the creation of scientific metaphors by facilitating the search, extension, and iterative revision of metaphors. Metaphorian uses a large language model-based workflow inspired by the heuristic rules revealed from a study with six professional writers. A user study (n=16) revealed that Metaphorian significantly enhances satisfaction, confidence, and inspiration in metaphor writing without decreasing writers’ sense of agency. We discuss design implications for creativity support for figurative writing in science.

Our work explores the idea of teaching computing by having learners create, arrange, and design comic panels. We designed comic-based learning toolkits, guided by the following research question: \emphHow can we support the informal learning of basic computing concepts for elementary school learners through a physical comic-based learning toolkit? This question emerged as a result of our partnership with a community organization that teaches art to elementary school learners through the production and distribution of art \emphsubscription boxes. Subscription boxes contain art materials and instruction manuals that learners can use to create artistic artifacts at home. The organization was interested in teaching computing through art activities. This led us to design a special subscription box containing materials that include paper comic panels, coloring pens, and magnets and an activity manual for a comic creation activity. Our learning toolkits guide learners to use computational concepts in the story-crafting process: \emphdecomposing a narrative within each comic panel, \emphsequencing comic panels to create a narrative flow, using \emphif-else for character decision-making within the story, and \emphiterating or \emphrefining the comic to create a cohesive narrative flow.

Recent work demonstrated how we can design and use coding strips, a form of comic strips with corresponding code, to enhance teaching and learning in programming. However, creating coding strips is a creative, time-consuming process. Creators have to generate stories from code (code->story) and design comics from stories (story->comic). We contribute CodeToon, a comic authoring tool that facilitates this code-driven storytelling process with two mechanisms: (1) story ideation from code using metaphor and (2) automatic comic generation from the story. We conducted a two-part user study that evaluates the tool and the comics generated by participants to test whether CodeToon facilitates the authoring process and helps generate quality comics. Our results show that CodeToon helps users create accurate, informative, and useful coding strips in a significantly shorter time. Overall, this work contributes methods and design guidelines for code-driven storytelling and opens up opportunities for using art to support computer science education.

With privacy-related concepts often abstract and difficult to define, comics can be an effective visual storytelling medium for explaining and raising awareness about privacy. However, existing privacy and security educational comics do not support content creation. To address this, we contribute PrivacyToon, a comic-based authoring tool that leverages concept-driven storytelling and ideation cards to help users create customizable privacy-related visual content. Our exploratory user study with 23 students and teachers shows PrivacyToon’s potential as a creative tool for communicating privacy concepts and stories. Our results show that a wide range of creativity preferences and contexts must be considered when designing systems that integrate ideation card-based design processes.

We explore how generative conversational AI can assist students’ learning, creative, and sensemaking process in a visual programming environment where users can create comics from code. The process of visualizing code in terms of comics involves mapping programming language (code) to natural language (story) and then to visual language (of comics). While this process requires users to brainstorm code examples, metaphors, and story ideas, the recent development in generative models introduces an exciting opportunity for learners to harness their creative superpower and researchers to advance our understanding of how generative conversational AI can augment our intelligence in creative learning contexts. We provide an overview of our system and discuss interaction scenarios to demonstrate ways we can partner with generative conversational AI in the context of learning computer programming.

Recent work investigated the potential of comics to support the teaching and learning of programming concepts and suggested several ways coding strips, a form of comic strip with its corresponding code, can be used. Building on this work, we tested the recommended use cases of coding strip in an undergraduate introductory computer science course at a large comprehensive university. At the end of the course, we surveyed students to assess their experience and found they benefited in various ways. Our work contributes a demonstration of the various ways comics can be used in introductory CS courses and an initial understanding of benefits and challenges with using comics in computing education gleaned from an analysis of students’ survey responses and code submissions.

The abstract nature of programming makes learning to code a daunting undertaking for many novice learners. In this work, we advocate the use of comics—a medium capable of presenting abstract ideas in a concrete, familiar way—for introducing programming concepts. Particularly, we propose a design process and related tools to help students and teachers create coding strips, a form of comic strips that are associated with a piece of code. We conducted two design workshops with students and high school computer science teachers to evaluate our design process and tools. We find that our design process and tools are effective at supporting the design of coding strips and that both students and teachers are excited about using coding strip as a tool for learning and teaching programming concepts.

Recently, deep learning has become exceptionally popular due to its outstanding performances in various machine learning and artificial intelligence applications. Convolutional neural network (CNN), a representative model of deep learning, has been successfully applied to solve computationally burdening tasks in various applications like computer vision. Despite its outstanding capability, training a CNN model properly is time-consuming and prone to overfitting and/or bad local minima. To address these issues, this study aims at improving the interpretation of the training process and using it for subsequent human intervention, specifically steering the training process of CNN model in the real time. In this paper, we present ReVACNN, a real-time visual analytic system for CNN, where (1) the overall training process (e.g., the amount of activations and that of changes each filter/layer has at a particular iteration/epoch) is visualized in network view and (2) the 2D embedding of trained filters within layers is visualized to show the relationships between filters as well as layers. In particular, ReVACNN allows users to perform several interactions in real time: (1) skipping the gradient descent update on the sub-part of a CNN model to reduce the subsequent training time and (2) steering filters interactively in the 2D embedding view to avoid bad local minima. At the end, we present several use cases that demonstrate the benefits users can gain from ReVACNN.

Nonnegative matrix factorization (NMF) has been widely applied in many domains. In document analysis, it has been increasingly used in topic modeling applications, where a set of underlying topics are revealed by a low-rank factor matrix from NMF. However, it is often the case that the resulting topics give only general topic information in the data, which tends not to convey much information. To tackle this problem, we propose a novel ensemble model of nonnegative matrix factorization for discovering high-quality local topics. Our method leverages the idea of an ensemble model, which has been successful in supervised learning, into an unsupervised topic modeling context. That is, our model successively performs NMF given a residual matrix obtained from previous stages and generates a sequence of topic sets. Our algorithm for updating the input matrix has novelty in two aspects. The first lies in utilizing the residual matrix inspired by a state-of-the-art gradient boosting model, and the second stems from applying a sophisticated local weighting scheme on the given matrix to enhance the locality of topics, which in turn delivers high-quality, focused topics of interest to users. We evaluate our proposed method by comparing it against other topic modeling methods, such as a few variants of NMF and latent Dirichlet allocation, in terms of various evaluation measures representing topic coherence, diversity, coverage, computing time, and so on. We also present qualitative evaluation on the topics discovered by our method using several real-world data sets.