Hugo Romat
Hugo Romat, Microsoft Research engineer inventing new mediums for the age of AI

Inventor · Builder · Researcher

Hugo Romat

Senior Research Engineer, Ph.D. · Microsoft Research (EPIC), Seattle

I invent new mediums for the age of AI: places where people and AI collaborate, reason, discover, and create together, and where content comes alive, responds to you, and is shaped as you interact with it. I build these ideas into working prototypes and ship them in Microsoft products like Copilot, OneNote, and Xbox. I’m passionate about building the future: making bold ideas real enough to experience, then reflecting on what those experiences reveal to shape research, products, and the real world.

Showreel

What if…

Projects exploring possible futures for generative AI: interfaces that come alive, respond, and evolve in real time.

Each prototype asks what happens when generated content stays live: something you can touch, steer, and reshape as you go, instead of a one-shot answer. Many of these ideas became papers at CHI and UIST, and some found their way into products.

Research areas

Journey

Making content respond to people: first data, then ink, now generative AI.

2023 — today

Media that come alive

Generative AI as an interactive, living medium.

Generative UIInteractive diffusionSpatial promptingGenerative ARVibe gaming

Why it matters

Chat unlocked the first wave of AI adoption. But a text box is a narrow window onto models that can now generate images, video, interfaces, and entire worlds. The next wave will come from mediums people can touch, steer, and reshape continuously. When AI becomes interactive, more people can use it, for more of their work, and with real control over the result.

Impact & influential work

I push AI beyond the chat box, toward generated content you can touch, steer, and reshape. I proved it in research, with three CHI and UIST Honorable Mentions, eight patent filings, and live demos to Microsoft leadership. I proved it in product too, leading image generation in the Copilot app, the generative pipeline behind Xbox Design Lab controllers, and image understanding in Copilot.

At EPIC I turned this vision into infrastructure: a common codebase and 20+ containerized generative models that take researchers and interns from idea to working prototype in days.

Vibe gaming: a game world that follows the player's imagination
Vibe gaming, shown at the Research Showcase
Texterial: shaping text like a material with gestures
Texterial: text as a material you shape
AI-Instruments: a generative container filling with art-style variations
AI-Instruments: prompts become tools

Key papers

  • ImaginationVellumUIST 2025 ★ Honorable Mention

    An ideation canvas where spatial layout, generative strokes, and history become the prompt.

  • AI-InstrumentsCHI 2025 ★ Honorable Mention

    Prompts become reusable instruments you apply directly to content, like tools.

  • TexterialCHI 2026 ★ Honorable Mention

    Text as a material you shape with gestures, with an LLM in the loop.

  • Intent TaggingCHI 2025

    Micro-prompts attached to parts of a work to steer generation precisely.

  • Live Artifacts2026

    Dynamic media built from live layers that keep their generative specification.

  • Brushing & Interpretive LinkingGI 2026

    Brushing and linking, reinvented for knowledge work with LLMs.

Product impact

  • Copilot · Drawing in the Copilot app

    Principal investigator for image generation in the Copilot app. I drove the Stable Diffusion technology end to end, ramped up the engineering team, and led responsible-AI communication for sketch-to-image.

  • Microsoft Paint · Cocreator

    Shaped the live AI sketching canvas that turns scribbles into art in Paint.

  • Copilot · Image understanding

    Led the analysis of how people use image input in Copilot, and shipped a small vision-language model to analyze images and suggest actions.

  • Copilot · Computer-using agent

    Built a computer-using agent that runs in a custom Linux container, and took it from leadership demo to product.

  • Copilot Notebooks

    Our hackathon with the Loop design team, on creating pages from existing assets and AI summaries, shaped Copilot Notebooks.

  • Xbox · Design Lab

    Principal investigator of the generative image pipeline that lets people design custom Xbox controllers.

Talks, demos & showcases

  • 2026MLADS · Generative UI and the next interactive AI experiences
  • 2026UX for Generative AI conference · Making AI interactive
  • 2025Microsoft Research Showcase · Vibe Gaming, plus an 8-person hackathon on generative games presented to Xbox leadership
  • 2025Microsoft Research Fair · ImaginationVellum, with private demos to Office AI and Gaming AI leaders
  • 2024Research Demo Fair · Intent Tagging and micro-prompting with Copilot
  • 2024New Future of Work report · AI-Instruments, Intent Tagging, and the intelligent canvas featured
  • 2025–26Organizer · Generative UI workshop (Project Green) and the GUIDE Forum

2019 — 2025

Ink & new input

Pen, touch, and devices that bring interaction closer to thought.

Digital inkPen + touchHandwriting AICross-deviceAccessibility

Why it matters

Ink is the fastest way to get a thought out of your head: no menus, no syntax. Yet digital ink has mostly stayed as static marks on a page. Making ink understood, legible, and generative turns the pen into a first-class way to work with data and AI, and opens it to people with motor impairments or who never learned to draw.

Impact & influential work

I made the case that ink should be a first-class input for AI, not a static mark on a page. My research, including a CHI Honorable Mention and work at CHI, UIST, and IEEE VR, defined how ink can act on data, reshape documents, and travel across devices.

On Microsoft’s Pen & Ink team I was the bridge from research to product. I led the prototypes that brought ink and AI into OneNote, Loop, and Copilot, which led to ink straightening and sketch-to-image in OneNote, a winning Office AI hackathon, and an ink canvas in Loop. Along the way I mentored Microsoft Research interns on cross-device and pen interaction.

ActiveInk: annotating a data visualization with a pen
ActiveInk: thinking with data, in ink
SpaceInk: document content reflowing to make space for handwritten notes
SpaceInk: documents make room for your notes
Flashpen: a high-precision pen for virtual reality
Flashpen: precise inking in VR

Key papers

  • ActiveInkCHI 2019 ★ Honorable Mention

    Ink annotations that act on the data they mark, for active reading of visualizations.

  • SpaceInkUIST 2019

    Document content reflows to make space for in-context handwritten annotations.

  • Tilt-Responsive TechniquesUIST 2020

    Digital drawing boards that adapt tools and content to how they are tilted.

  • Style BlinkCHI 2022

    Handcrafted styling as a first-class object for structured ink, like journaling.

  • FlashpenIEEE VR 2021

    A high-fidelity, high-precision multi-surface pen for virtual reality.

  • AirConstellationsUIST 2021

    In-air device formations for cross-device interaction and collaboration.

  • TapTypeCHI 2022

    Ten-finger typing on everyday surfaces, decoded with Bayesian inference.

Product impact

  • OneNote · Ink straightening

    Designed and implemented the straightening algorithms that make handwriting more legible, shipped them in the OneNote codebase, and led the end-to-end evaluation through rollout.

  • OneNote · Sketch to image

    Turned our prototypes into sketch-to-image on the OneNote canvas with the Pen & Ink incubation team.

  • Loop · Ink as a first-class citizen

    1st place in the Office AI hackathon (2025): ink on the lock screen, in Loop, and in Copilot chat. The product team then started building an ink canvas into Loop.

  • Office · Sketch beautification

    Led a top-three hackathon project (2024) on beautifying sketches, picked up as a new experience across Office apps.

  • Generative ink & handwriting legibility

    Tech lead on handwriting generation, and on new ML approaches that segment ascenders, descenders, and baselines to make ink more legible and accessible.

2016 — 2020

Data that responds

Expressive, interactive visualization for sensemaking.

Information visualizationGraph drawingAnimationPictographsData comics

Why it matters

Data only matters if people can see it, question it, and tell its story. During my Ph.D. at Inria, and in collaboration with Microsoft Research, I designed systems where visual representations are authored and shaped directly, rather than configured through menus. That idea, content you shape by hand, is the seed of everything I’ve built since.

Impact & influential work

My Ph.D. challenged the default look of network visualizations: plain circles, lines, and static edges. I showed that expressive, animated visuals can encode more data and engage readers. This work won the INTERACT 2019 Best Doctoral Student Paper award and appeared at CHI and IEEE TVCG, and my collaborations earned a CHI Best Paper.

The idea came full circle with generative AI. I turned charts into constraints for image generation, which became the AI infographics pipeline in Microsoft Designer, and I launched a joint internship bringing data comics to Microsoft Security.

Graphies: expressive node-link diagrams authored directly
Graphies: expressive node-link diagrams
Animated particles encoding data along graph edges
Animated particles that encode data
Data-driven image generation: charts turned into imagery
Full circle: data-driven image generation

Key papers

  • GraphiesIEEE TVCG 2021

    Expressive authoring of node-link diagrams, beyond default circles and lines.

  • Animated Edge TexturesCHI 2018

    A design space for animating edges to encode data in network visualizations.

  • Particle Speed PerceptionINTERACT 2019 ★ Best Doctoral Student Paper

    How color and size change the perceived speed of animated particles.

  • Dear PictographCHI 2020

    Personalization and immersion for consuming and enjoying visualizations.

  • Sketchnote StrategiesCHI 2021 ★ Best Paper

    Components, dimensions, and strategies for effective visual note taking.

  • Interactive Data ComicsIEEE TVCG 2022

    Comics that let readers interact with the data behind the story.

Product impact

  • Microsoft Designer · AI infographics

    Invented using a chart’s depth map to constrain image generation, and prototyped it on our Stable Diffusion + ControlNet pipeline. It drove a hackathon with the Designer team and became the SDXL pipeline for AI infographics.

  • Microsoft Security · Data comics

    Launched a joint internship with Microsoft Security that tells the story of security incidents through data comics.

Awards & service

Recognition

Paper awards

Patents & hackathons

  • 9 patents on generative interaction, spatial prompting, digital materials, intent tags, and ink
  • 1st place, Office AI hackathon 2025 · Ink as a first-class citizen
  • Top 3, organization hackathon 2024 · Sketch beautification

Mentoring

  • Mentored 10+ research interns at Microsoft Research, leading to papers at CHI, UIST, and GI
  • Mentored a Vibe Gaming intern to an MSR fellowship on collaborative generative UI, with MSR Cambridge
  • Launched joint internships between Microsoft Research and product teams

Service

  • Reviewer for CHI, UIST, DIS, and ICDAR
  • Organizer, “AI Manifestations in User Experiences” workshop
  • Organizer, Generative UI workshop and GUIDE Forum (2025, 2026)

Career

Experience

Senior Research Engineer

Microsoft Research, EPIC · Seattle

Building the foundations for AI-native experiences: generative UI, real-time image generation, and the shared platform behind the team’s prototypes.

Senior Applied Scientist

Microsoft, Pen & Ink team (Experiences & Devices) · Seattle

Bridged Microsoft Research and product, bringing ink and generative AI into OneNote, Copilot, and Paint.

Postdoctoral Researcher

ETH Zurich, SIP Lab · Zurich

Sensing and input for mixed reality and mobile: Flashpen and TapType.

Research

Publications

My peer-reviewed research spans Generative AI, Human-Computer Interaction (HCI), Human-AI Interaction, interactive data visualization, and novel input techniques. The publications below are drawn from my DBLP record (see also Google Scholar). Expand any entry to copy its BibTeX citation.

2026

  1. Live Artifacts: Authoring Dynamic Media via Live Layers Encapsulating Generative Specifications

    Leixian Shen, Haotian Li, Hugo Romat, Fanny Chevalier, Nicolai Marquardt, Nathalie Riche

    arXiv preprint 2026

    arXivPDF
  2. Brushing and Interpretive Linking for Knowledge Work

    Zheng Ning, Michel Pahud, Hugo Romat, Nicolai Marquardt, Fanny Chevalier, Jeevana Priya Inala, Chenglong Wang, David Brown, Toby Jia-Jun Li, Ken Hinckley, Nathalie Riche

    GI 2026

    PDF
  3. Texterial: A Text-as-Material Interaction Paradigm for LLM-Mediated Writing

    Jocelyn J. Shen, Nicolai Marquardt, Hugo Romat, Ken Hinckley, Nathalie Riche, Fanny Chevalier

    ACM CHI 2026 Honorable Mention

    ACMarXivPDF

2025

  1. Intent Tagging: Exploring Micro-Prompting Interactions for Supporting Granular Human-GenAI Co-Creation Workflows

    Frederic Gmeiner, Nicolai Marquardt, Michael Bentley, Hugo Romat, Michel Pahud, David Brown, Asta Roseway, Nikolas Martelaro, Kenneth Holstein, Ken Hinckley, Nathalie Riche

    ACM CHI 2025

    ACMarXivPDF
  2. ImaginationVellum: Generative-AI Ideation Canvas with Spatial Prompts, Generative Strokes, and Ideation History

    Nicolai Marquardt, Asta Roseway, Hugo Romat, Payod Panda, Michel Pahud, Gonzalo A. Ramos, Steven Mark Drucker, Andrew D. Wilson, Ken Hinckley, Nathalie Riche

    ACM UIST 2025 Honorable Mention

    ACMPDF
  3. AI-Instruments: Embodying Prompts as Instruments to Abstract & Reflect Graphical Interface Commands as General-Purpose Tools

    Nathalie Riche, Anna Offenwanger, Frederic Gmeiner, David Brown, Hugo Romat, Michel Pahud, Nicolai Marquardt, Kori Inkpen, Ken Hinckley

    ACM CHI 2025 Honorable Mention

    ACMarXivPDF

2024

  1. Opportunistic Nudges for Task Migration Between Personal Devices

    Nikhita Joshi, Richard Li, Jiannan Li, Leonardo Pavanatto, Michel Pahud, Jatin Sharma, Bongshin Lee, Hugo Romat, William Buxton, Nicolai Marquardt, Ken Hinckley, Nathalie Henry Riche

    ACM CHI Extended Abstracts 2024

    ACMPDF
  2. Memory Recall for Data Visualizations in Mixed Reality, Virtual Reality, 3D and 2D

    Christophe Hurter, Bernice E. Rogowitz, Guillaume Truong, Tiffany Andry, Hugo Romat, Ludovic Gardy, Fereshteh Amini, Nathalie Henry Riche

    IEEE TVCG 2024

    DOIPDF

2023

  1. Escapement: A Tool for Interactive Prototyping with Video via Sensor-Mediated Abstraction of Time

    Molly Jane Nicholas, Nicolai Marquardt, Michel Pahud, Nathalie Riche, Hugo Romat, Christopher Collins, David Ledo, Rohan Kadekodi, Badrish Chandramouli, Ken Hinckley

    ACM CHI 2023

    ACMPDF
  2. AdHocProx: Sensing Mobile, Ad-Hoc Collaborative Device Formations using Dual Ultra-Wideband Radios

    Richard Li, Teddy Seyed, Nicolai Marquardt, Eyal Ofek, Steve Hodges, Mike Sinclair, Hugo Romat, Michel Pahud, Jatin Sharma, William A. S. Buxton, Ken Hinckley, Nathalie Riche

    ACM CHI 2023

    ACMPDF

2022

  1. Understanding Multi-Device Usage Patterns: Physical Device Configurations and Fragmented Workflows

    Ye Yuan, Nathalie Riche, Nicolai Marquardt, Molly Jane Nicholas, Teddy Seyed, Hugo Romat, Bongshin Lee, Michel Pahud, Jonathan Goldstein, Rojin Vishkaie, Christian Holz, Ken Hinckley

    ACM CHI 2022

    ACMPDF
  2. TapType: Ten-finger text entry on everyday surfaces via Bayesian inference

    Paul Streli, Jiaxi Jiang, Andreas Rene Fender, Manuel Meier, Hugo Romat, Christian Holz

    ACM CHI 2022

    ACMarXivPDF
  3. Style Blink: Exploring Digital Inking of Structured Information via Handcrafted Styling as a First-Class Object

    Hugo Romat, Nicolai Marquardt, Ken Hinckley, Nathalie Henry Riche

    ACM CHI 2022

    ACMPDF
  4. Interactive Data Comics

    Zezhong Wang, Hugo Romat, Fanny Chevalier, Nathalie Henry Riche, Dave Murray-Rust, Benjamin Bach

    IEEE TVCG 2022

    DOIPDF
  5. Demonstrating TapType for mobile ten-finger text entry anywhere

    Paul Streli, Jiaxi Jiang, Andreas Rene Fender, Manuel Meier, Hugo Romat, Christian Holz

    ACM CHI Extended Abstracts 2022

    ACM

2021

  1. Sketchnote Components, Design Space Dimensions, and Strategies for Effective Visual Note Taking

    Rebecca Zheng, Marina Fernández Camporro, Hugo Romat, Nathalie Henry Riche, Benjamin Bach, Fanny Chevalier, Ken Hinckley, Nicolai Marquardt

    ACM CHI 2021 Best Paper

    ACMPDF
  2. Flashpen: A High-Fidelity and High-Precision Multi-Surface Pen for Virtual Reality

    Hugo Romat, Andreas Fender, Manuel Meier, Christian Holz

    IEEE VR 2021

    DOIPDF
  3. Expressive Authoring of Node-Link Diagrams With Graphies

    Hugo Romat, Caroline Appert, Emmanuel Pietriga

    IEEE TVCG 2021

    DOIPDF
  4. Demonstration of Flashpen: A high-fidelity and high-precision multi-surface pen for Virtual Reality

    Hugo Romat, Andreas Rene Fender, Manuel Meier, Christian Holz

    ACM CHI Extended Abstracts 2021

    ACM
  5. Demonstrating High-Precision and High-Fidelity Digital Inking for Virtual Reality

    Hugo Romat, Andreas Fender, Manuel Meier, Christian Holz

    IEEE VR Workshops 2021

    DOI
  6. AirConstellations: In-Air Device Formations for Cross-Device Interaction via Multiple Spatially-Aware Armatures

    Nicolai Marquardt, Nathalie Henry Riche, Christian Holz, Hugo Romat, Michel Pahud, Frederik Brudy, David Ledo, Chunjong Park, Molly Jane Nicholas, Teddy Seyed, Eyal Ofek, Bongshin Lee, William A. S. Buxton, Ken Hinckley

    ACM UIST 2021

    ACMPDF

2020

  1. Tilt-Responsive Techniques for Digital Drawing Boards

    Hugo Romat, Christopher Collins, Nathalie Henry Riche, Michel Pahud, Christian Holz, Adam Riddle, Bill Buxton, Ken Hinckley

    ACM UIST 2020

    ACMPDF
  2. Dear Pictograph: Investigating the Role of Personalization and Immersion for Consuming and Enjoying Visualizations

    Hugo Romat, Nathalie Henry Riche, Christophe Hurter, Steven Mark Drucker, Fereshteh Amini, Ken Hinckley

    ACM CHI 2020

    ACMPDF

2019

  1. SpaceInk: Making Space for In-Context Annotations

    Hugo Romat, Emmanuel Pietriga, Nathalie Henry Riche, Ken Hinckley, Caroline Appert

    ACM UIST 2019

    ACMPDF
  2. Influence of Color and Size of Particles on Their Perceived Speed in Node-Link Diagrams

    Hugo Romat, Dylan Lebout, Emmanuel Pietriga, Caroline Appert

    IFIP INTERACT 2019 Best Doctoral Student Paper

    DOIPDF
  3. From data exploration to presentation : designing new systems and interaction techniques to enhance the sense-making process. (De l'exploration des données à la présentation : concevoir de nouveaux systèmes et techniques d'interaction pour améliorer la création de sens à partir de données)

    Hugo Romat

    Ph.D. Dissertation · 2019

    DOIPDF
  4. ActiveInk: (Th)Inking with Data

    Hugo Romat, Nathalie Henry Riche, Ken Hinckley, Bongshin Lee, Caroline Appert, Emmanuel Pietriga, Christopher Collins

    ACM CHI 2019 Honorable Mention

    ACMPDF

2018

  1. Animated Edge Textures in Node-Link Diagrams: a Design Space and Initial Evaluation

    Hugo Romat, Caroline Appert, Benjamin Bach, Nathalie Henry Riche, Emmanuel Pietriga

    ACM CHI 2018

    ACMPDF

2016

  1. Natural Human-Robot Interaction Using Social Cues

    Hugo Romat, Mary-Anne Williams, Xun Wang, Benjamin Johnston, Henry Bard

    ACM/IEEE HRI 2016

    DOI

Background

Education

Ph.D. in Computer Science

Inria · Paris

From data exploration to presentation: new systems and interaction techniques for sensemaking.

M.Sc. in Computer Science

ENSC Bordeaux INP · Université Laval, Canada

Human–computer interaction, computer science, robotics, and cognitive science.