Copilot Authoring

UI/ UX, UX research, Design research, Design workshops

UI/ UX, UX research, Design research,
Design workshops

Role: Lead Designer

Feature crew: 1 Designer, 1 PM, 1 UX Researcher, 4 Engs

Feature crew: 1 Designer, 1 PM,
1 UX Researcher, 4 Engs

Final MVP design

Opportunity

Our recent telemetry shows that 2.6% of authors complete a Pulse after starting a survey from the homepage. Drop-off rates are unclear but previous research indicates managers have trouble getting   in and out of the question library and struggle to craft precise questions that will get them the results   they desire. With such a high drop off rate, Pulse leadership prioritized looking at Copilot integration   in survey creation as a possible accelerator in question customization while providing assistance   in crafting precise wording from questions pulled from the latest ChatGpt o1 model and from our   backend of People science-backed questions.

Goals

Leverage Copilot with assisting with   question creation
in a template and rewriting for tone or clarity.

Leverage Copilot with assisting with   question creation in a template and rewriting for tone or clarity.

Use this phase to test, workshop, and inform
future   Copilot design enhancements.

Use this phase to test, workshop,
and inform future   Copilot design enhancements.

Success metrics

Adoption rate: 30% of users try Copilot in the first
2 months

Adoption rate: 30% of users try Copilot in the first 2 months

Satisfaction rating: 60%+ of users rate Copilot
positively

Satisfaction rating: 60%+ of users rate Copilot positively

Time saved: 20% reduction time to complete a survey draft

Edit rate: <50% of Copilot-generated questions
need editing

Edit rate: <50% of Copilot-generated questions need editing

User roles

Author

• Authors cover a variety of industries and are mainly M1 or M2 managers and team leads.

• Authors want to reach out to their team   in moments that matter that require feedback to support productivity, team connectedness, and their employee’s wellbeing and happiness at work.

Recipient

• Recipients are ICs (individual contributors) and FLWs (Front-line workers) who cover   a variety of industries.

Recipients want to feel safe, supported, and connected when leaving feedback   for surveys.

Recipients want to feel safe, supported, and connected when leaving feedback   for surveys.

My approach

With such a high turnaround for designs, I quickly put initial P0 designs through User testing and created a comparative study to gain qualitative feedback. One of the key things that we were trying to find was if we should use Copilot sidecar, as was advised by the Viva Copilot team, or if we should create an On-Canvas experience. In parallel to design research, we engaged cross team to see what other Viva and Microsoft teams were doing to integrate Copilot. I also dived into research efforts from the past two years that covered usability issues with Copilot across Microsoft Office, Loop, and Viva apps.

With such a high turnaround for designs,
I quickly put initial P0 designs through User testing and created a comparative study to gain qualitative feedback. One of the key things that we were trying to find was if we should use Copilot sidecar, as was advised by the Viva Copilot team, or if we should create an On-Canvas experience. In parallel to design research, we engaged cross team to see what other Viva and Microsoft teams were doing to integrate Copilot. I also dived into research efforts from the past two years that covered usability issues with Copilot across Microsoft Office, Loop, and Viva apps.

User Testing P0 Designs

These were the A/B test designs that went through User testing to understand better how gen pop users understood Copilot integration into our existing survey making platform.

Copilot integration inline in question cards

Copilot sidecar with Draft, Inspire, Ask prompts

Key takeaways from UT studies

Usability study structure

• Three separate UT design studies which covered   zero prompt state, rewrite inline, discoverability   scenarios  

• 32 unmoderated, 5-50 min tests. (The length varied   according to the intent of study)

• 32 unmoderated, 5-50 min tests.
(The length varied   according to the intent of study)

• Users were prompted with tasks as they walked   through clickable prototypes

• Seniors (12), Managers (14), Vice President (1),   Directors (3), Owners(1)

• All US based; half women and half men

High level findings

• In both sidecar and on-canvas experience, users   were delighted with Copilot assisting them writing   new questions or rewriting tone

• People found the prompts to be distinct and were   great in guiding them as they were just starting out   and attempting to draft and edit questions

• There was usability confusion when opening sidecar   and then trying to write questions on-canvas

• Users stated that if used in their workflow the benefits included:saving time, writing more effective questions   to their team, and encouraging them to create surveys   more often because of ease of use.

“Overall I really liked it. It made a ton of sense. You knew exactly which prompt to use, descriptions were really good. Very easy to make use of them from a process standpoint”

“It doesn’t get any easier than this”

“It's pretty good. Each prompt is pretty distinct and you can understand what you can do in each area.”

“I use Copilot a lot to collect my thoughts so this is very streamline. Very easy.”

Developing a strategic approach: Sidecar or On-Canvas experience

Collaborate cross-team

• We had discussions with the Microsoft Forms designer/engineers who were pioneers in hitting our scenarios of adding and rewriting questions within their framework with ChatGPT and Avalon. Also had discussions around how they were implementing some of common controls in the Copilot chat library and experiencing perf issues.

• Led discussions with Viva Insights and Glint designers who both were dealing with issues with building sidecar experience at an accelerated speed for coherence. They had discoverability issues and were adding skittles to the main page so users would engage with sidecar.

• Reviewed work with Viva Copilot design team to gain any feedback for our distinct scenarios and approach. With these discussions and reading research over the last two years on problems across M365 pushed for design POV to build On-Canvas experience.

Workshop with Pulse engs

• Close-collaboration with Pulse engs to figure out how the product will scale   

• Workshop features that are manual vs Copilot editing  

• Address any eng concerns, edge cases as they are integrating Forms framework and doing prompt engineering

New On-Canvas entry point for MVP

Outcome

• +16% increase in daily engagement during preview

• +28% increase in Pulses sent during preview

• Drove leadership alignment on the role of AI-assisted survey creation

• Defined the UX direction adopted for Viva Pulse's AI roadmap

North Star Desktop

North Star Desktop

North Star Mobile