02 A MODERN PLATFORM FOR AI AGENTS

Summary

AgentZ is Amazon’s internal automation framework enabling teams to create AI agents. The agents, powered by LLMs, are intended to solve a complex task for both technical and non-technical users.

Customer problem

AgentZ was an early-stage product that, while functional, was usable only by highly technical AWS users. For the tool to be widely adopted across Amazon, the interface needed a complete redesign to make it user-friendly for people with different levels of technical experience. This was the problem I was brought in to solve.

Old design - Create agent screen

Old design - Create agent screen workflow

Why build an AI agent

Most teams spend their time performing Knowledge Operations Work (KOW) like annotating images and videos, reviewing listings for ads in compliance with Amazon’s policies, and answering customer/seller questions. By creating a self-service platform for AI agents, we could enable numerous Amazon teams to automate Knowledge Operations Work (KOW).

My process

I began with a comprehensive audit of the existing workflow using Nielsen’s 10 heuristic principles to identify usability issues. I also encouraged the product partner to share existing users’ feedback, which served as a great qualitative data point for my research.

I analyzed different design systems within Amazon and recommended the one that had the most extensive documentation, flexibility, and support from internal teams (e.g., Slack). The stakeholders loved my recommendation. They were eager for a modern UI refresh with shorter build times.

Challenge

Being a very technical product, I had to spend a lot of time understanding the concept and devote many hours to reading the documentation. I met with the product partner multiple times each week and created numerous collaborative wireframes to understand the workflow. I also collaborated with the engineers to understand the technology, which helped me suggest improvements to the workflow.

The most hotly debated topic was whether to make the agents conversational. A study of other AI agents showed that most agents in the market were heavily conversational. Upon reviewing the workflow, I decided that introducing conversation would not be effective and would prove to be cumbersome, causing user fatigue. On the other hand, I insisted on a guided tour experience for first-time users and advocated relying on smart defaults and helpful inline text within the agent.

Designs

Namespace selection interface showing the initial setup flow for creating AI agents

Create agent screen with improved user guidance

Core capabilities interface showing available capabilities that can be added to the agent

AI review feature displaying intelligent suggestions and feedback for agent optimization

Design feedback

I presented my design to eight users who were early adopters of AgentZ:

1. Overall, users liked the workflow but wanted to know the state of the agent when executing their tasks. Based on this feedback, I added states to the AI agent when it was executing or updating.
2. Users wanted more reasoning from the AI review feature, so I added more helpful text when the AI was producing suggestions.

The users were eager and happy to participate in the exercise and share their experience. They also requested more nuanced features like version control of the agent.

Outcome

After three weeks of designing the agent workflow, I presented the designs to the Director of the Applied AI organization and other critical stakeholders. The stakeholders agreed that the designs simplified the user experience and elevated the design maturity of the product. This design was approved for General Audience (GA) launch.