Free your hands for what matters

A task-conducting AI agent embed in Windows OS

Timeline

Device

Team

Responsibilities

Jun. - Aug. 2024
Desktop
PM: Yu Shi
AI Researcher: Jay Huang
Engineer Lead: Chaoyun Zhang
Wireframe, Design iteration, Hi-fi mockup, Prototyping, Usability testing, Literature review

Project Overview

Windows system are exploring how to enhance work efficiency and free users' hands for more meaningful tasks.

In the summer of 2024, I interned at Microsoft and worked on an innovative AI project that explored the potential of AI technology beyond conversational AI. Collaborating with the AI research team, we investigated various approaches to productize AI and image recognition technology, ultimately developing a task-conducting AI agent embedded within the Windows OS.

Adopting new technology often presents challenges in building trust between users and the system. Following Microsoft's Human-AI Interaction principles, I explored various methods to foster trust in a system that could potentially take control of users' devices.

Project context

AI agents are being applied to different apps and scenarios. What if there were an AI agent that could help users complete all types of tasks?

As a leader in AI technology, Microsoft continues to push the boundaries of what's possible. After launching Copilot, the company is now striving to transcend the limitations of conversational AI and develop an AI agent designed to truly assist users in completing tasks.

Design problem

This is an engineer-led project aimed at exploring the possibilities of technology. As the only product designer involved, my role is to explore using scenarios, understand users' perceptions and expectations of the new technology, and design an MVP product prototype for exploration. Thus, the design task I was handed was designing an intuitive AgentOS MVP experience for users to input task, to monitor task conducting, and to control the outcome quality. For users, the key concerns revolve around the control and trustworthiness of the AI product.

The Problem Statement

How might we build user trust in an AI that manages repetitive, non-decision-making tasks in their workflow?

What I design

Tracking task conducting status in PiP

Since the AI agent relies on image recognition to complete tasks on the desktop, I adopted the picture-in-picture (PiP) approach to enable users to monitor the agent's work in real time. The PiP window is adjustable, allowing users to resize it to suit their needs while seamlessly multitasking.

Learn more about this project

Contact me if you want to talk about this project!