Copilot Authoring
Copilot Authoring
UI/ UX, UX research, Design research, Design workshops
UI/ UX, UX research, Design research, Design workshops
Role:
Role:
Lead designer
Lead designer
Feature crew: 1 Designer, 1 PM, 1 UX researcher, 4 Engs
Feature crew: 1 Designer, 1PM, 1UX Researcher, 4 Engs
Opportunity
Opportunity
Our recent telemetry shows that 2.6% of authors complete a Pulse after starting a survey from the homepage. Drop-off rates are unclear but previous research indicates managers have trouble getting in and out of the question library and struggle to craft precise questions that will get them the results they desire.
With such a high drop off rate, Pulse leadership prioritized looking at Copilot integration in survey creation as a possible accelerator in question customization while
providing assistance in crafting precise wording from questions pulled from the latest ChatGPT o1 model and from our backend of People science-backed questions.
This was one of two features that accelerated out of a FHL (Fix Hack Learn) project in the Spring 2024 and made it on the roadmap.
Our recent telemetry shows that 2.6% of authors complete a Pulse after starting a survey from the homepage. Drop-off rates are unclear but previous research indicates managers have trouble getting
in and out of the question library and struggle to craft precise questions that will get them the results they desire.
With such a high drop off rate, Pulse leadership prioritized looking at Copilot integration in survey creation as a possible accelerator in customization while providing assistance in crafting precise wording from questions pulled from the latest ChatGPT o1 model and from our backend of People science-backed questions.
This was one of two features that came out of a FHL (Fix Hack Learn) project in Spring 2024 and made it on the roadmap.
Our recent telemetry shows that 2.6% of authors complete a Pulse after starting a survey from the homepage.
Drop-off rates are unclear but previous research indicates managers have trouble getting in and out of the
question library and struggle to craft precise questions that will get them the results they desire.
With such a high drop off rate, Pulse leadership prioritized looking at Copilot integration in survey creation as
a possible accelerator in customization while providing assistance in crafting precise wording from questions
pulled from the latest ChatGPT o1 model and from our backend of People science-backed questions.
This was one of two features that came out of a FHL (Fix Hack Learn) project in Spring 2024 and made it
on the roadmap.
Final Designs
Final Designs
Add a question
Add a question
Rewrite question text
Rewrite question text
Goals
Goals
Leverage Copilot to accelerate user goals by assisting with question
creation in a template and rewriting for tone or clarity.
Use this phase to test, workshop, and inform future Copilot design enhancements.
Use this phase to test, workshop, and inform future Copilot design enhancements.
Success metrics: MVP Benchmarks
Success metrics: MVP Benchmarks
Adoption rate: 30% of users try Copilot in the first 2 months
Adoption rate: 30% of users try Copilot
in the first 2 months
Satisfaction rating: 60%+ of users rate Copilot positively
Satisfaction rating: 60%+ of users rate
Copilot positively
Time saved: 20% reduction time to complete a survey draft
Time saved: 20% reduction time to complete
a survey draft
Edit rate: <50% of Copilot-generated questions need editing
Edit rate: <50% of Copilot-generated
questions need editing
User roles
User roles
Author
Author
Authors cover a variety of industries and are mainly M1 or M2 managers and team leads. M1 managers are managers of small teams of 10 or less. M2 managers are skip-level managers who have teams of 10+.
Authors want to effectively communicate with their team and empower their directs to speak their mind and feel supported.
Authors want to reach out to their team in moments that matter that require feedback to support productivity, team connectedness, and their employee’s wellbeing and happiness at work.
Authors cover a variety of industries and are mainly M1 or M2 managers and team leads. M1 managers are managers of small teams of 10 or less. M2 managers are skip-level managers who have teams of 10+.
Authors want to effectively communicate with
their team and empower their directs to speak their mind and feel supported.
Authors want to reach out to their team in moments that matter that require feedback to support productivity, team connectedness, and their employee’s wellbeing and happiness at work.
Authors cover a variety of industries and are mainly M1 or M2 managers and team leads. M1 managers are managers of small teams of 10 or less. M2 managers are skip-level managers who
have teams of 10+.
Authors want to effectively communicate with their team and empower their directs to speak
their mind and feel supported.
Authors want to reach out to their team in moments that matter that require feedback to support productivity, team connectedness, and their employee’s wellbeing and happiness at work.
Recipient
Recipient
Recipients are ICs (individual contributors) who cover a variety
of industries. We are designing for IWs (information workers) and FLWs (front-line workers).
We are designing for recipients so they feel safe, supported, and connected when leaving feedback for surveys.
Recipients are ICs (individual contributors) who cover a variety of industries.
We are designing for IWs (information workers)
and FLWs (front-line workers).
We are designing for recipients so they feel safe, supported, and connected when leaving feedback
for surveys.
Recipients are ICs (individual contributors) who cover a variety of industries. We are designing for IWs (information workers) and FLWs (front-line workers).
We are designing for recipients so they feel safe, supported, and connected when leaving feedback for surveys.
My approach
My approach
With such a high turnaround for designs, I quickly put initial P0 designs through user testing and created a comparative study to gain qualitative feedback. One of the key things that we were trying to find out was if we should use Copilot sidecar, as was advised by the Viva Copilot team, or if we should create an On-Canvas experience with visual cues in the flow of user's actions. In parallel to design research, we engaged cross-team to see what other Viva and Microsoft teams were doing to integrate Copilot.
I also dived into existing research efforts from the past two years that covered usability issues with Copilot across Microsoft Office, Loop, and Viva apps.
With such a high turnaround for designs, I quickly put initial P0 designs through User testing and created a comparative study to gain qualitative feedback. One of the key things that we were trying to find was if we should use Copilot sidecar, as was advised by the Viva Copilot team, or if we should create an On-Canvas experience with visual cues
in the flow of user's actions. In parallel to design research, we engaged cross team to see what other Viva and Microsoft teams were doing to integrate Copilot. I also dived into existing research efforts from the past two years that covered usability issues with Copilot across Microsoft Office, Loop, and Viva apps.
UT P0 designs
UT P0 designs
These were the A/B test designs that went through User testing to understand better how gen pop users understood Copilot integration into our existing survey making platform.
These were the A/B test designs that went through User testing to understand better how gen pop users understood Copilot integration into our existing survey making platform.
Copilot integration inline in question cards
Copilot sidecar with Draft, Inspire, Ask prompts
Key takeaways from UT studies
Key takeaways from UT studies
Usability study structure
Usability study structure
• Three separate UT design studies which covered zero prompt state,
rewrite inline, discoverability scenarios
• 32 unmoderated, 5-50 min tests. (The length varied according to the
intent of study)
• Users were prompted with tasks as they walked through clickable
prototypes
• Seniors (12), Managers (14), Vice President (1), Directors (3), Owners(1)
• All US based; half women and half men
• Three separate UT design studies which covered zero prompt state, rewrite inline, discoverability scenarios
• 32 unmoderated, 5-50 min tests. (The length varied according to the intent of study)
• Users were prompted with tasks as they walked through clickable prototypes
• Seniors (12), Managers (14), Vice President (1), Directors (3), Owners(1)
• All US based; half women and half men
• Three separate UT design studies which covered zero prompt state, rewrite inline, discoverability scenarios
• 32 unmoderated, 5-50 min tests. (The length varied according to the intent of study)
• Users were prompted with tasks as they walked through clickable prototypes
• Seniors (12), Managers (14), Vice President (1), Directors (3), Owners(1)
• All US based; half women and half men
High level findings
High level findings
• In both sidecar and on-canvas experience, users were delighted
with Copilot assisting them writing new questions or rewriting tone
• People found the prompts to be distinct and were great in guiding them
as they were just starting out and attempting to draft and edit questions
• There was usability confusion when opening sidecar and then trying
to write questions on-canvas
• Most people have very little exposure to Copilot and AI integration in
apps they use but quickly figured things out and were delighted.
• Users stated that if used in their workflow the benefits included:saving
time, writing more effective questions to their team, and encouraging
them to create surveys more often because of ease of use.
• In both sidecar and on-canvas experience, users were delighted with Copilot assisting them writing new questions or rewriting tone
• People found the prompts to be distinct and were great in guiding them as they were just starting out and attempting to draft and edit questions
• There was usability confusion when opening sidecar and then trying to write questions on-canvas
• Most people have very little exposure to Copilot and AI integration in apps they use but quickly figured things out and were delighted.
• Users stated that if used in their workflow the benefits included:saving time, writing more effective questions to their team, and encouraging them to create surveys more often because of ease of use.
• In both sidecar and on-canvas experience, users were delighted with Copilot assisting them writing new questions or rewriting tone
• People found the prompts to be distinct and were great in guiding them as they were just starting out and attempting to draft and edit questions
• There was usability confusion when opening sidecar and then trying to write questions on-canvas
• Most people have very little exposure to Copilot and AI integration in apps they use but quickly figured things out and were delighted.
• Users stated that if used in their workflow the benefits included:saving time, writing more effective questions to their team, and encouraging them to create surveys more often because of ease of use.
“Overall I really liked it. It made a ton of sense. You knew exactly which prompt to use, descriptions were really good. Very easy to make use of them from a process standpoint”
“Overall I really liked it. It made a ton of sense. You knew exactly which prompt to use, descriptions were really good. Very easy to make use of them from a process standpoint”
“It's pretty good. Each prompt is pretty distinct and you can understand what you can do in each area.”
“It's pretty good. Each prompt is pretty distinct and
you can understand what you can do in each area.”
“It doesn’t get any easier than this”
“It doesn’t get any easier than this”
“I use Copilot a lot to collect my thoughts so this is very streamline. Very easy.”
“I use Copilot a lot to collect my thoughts so this is
very streamline. Very easy.”
Developing a strategic approach: Sidecar or On- Canvas experience
Developing a strategic approach: Sidecar or On- Canvas experience
Developing a strategic approach: Sidecar or On- Canvas experience
Collaborate and gain insights cross-team
Collaborate and gain insights cross-team
Collaborate and gain insights cross-team
• We discussed with Microsoft Forms team, who were pioneers in hitting our scenarios of adding and rewriting questions within their framework on ChatGPT and Avalon, some of their challenges. We also had discussions around how they were implementing some common controls in the Copilot chat library and experiencing performance issues.
• We discussed with Microsoft Forms team, who were pioneers in hitting our scenarios of adding and rewriting questions within their framework on ChatGPT and Avalon, some of their challenges. We also had discussions around how they were implementing some common controls in the Copilot chat library and experiencing performance issues.
• We continued our discussions across Viva with Insights and Glint. They were both dealing with discoverability issues after rapidly building the sidecar experience to be in alignment across Viva. Their solution was adding skittles to the main page in order to encourage users to open sidecar to get users in faster and resolves issues of discoverability. Both teams main focus were scenarios only covering summarization.
• We continued our discussions across Viva with Insights and Glint. They were both dealing with discoverability issues after rapidly building the sidecar experience to be in alignment across Viva. Their solution was adding skittles to the main page in order to encourage users to open sidecar to get users in faster and resolves issues of discoverability. Both teams main focus were scenarios only covering summarization.
• We reviewed our work with Viva Copilot design team to gain any feedback for our distinct scenarios and approach since I decided to deviate from sidecar.
• We reviewed our work with Viva Copilot design team to gain any feedback for our distinct scenarios and approach since I decided to deviate from sidecar.
Workshop with Pulse engs
Workshop with Pulse engs
• We closely collaborated with Pulse engs to figure out how the product would scale.
• We closely collaborated with Pulse engs to figure out how the product would scale.
• Brainstorm to understand editing capabilities: what was manual vs edited by Copilot.
• Brainstorm to understand editing capabilities: what was manual vs edited by Copilot.
• Address any eng concerns, edge cases as they are integrating Forms framework and testing in prompt engineering.
• Address any eng concerns, edge cases as they are integrating Forms framework and testing in prompt engineering.
Outcome
Outcome
Early preview with internal cients
Early preview with internal cients
Daily engagement with the app - up 16%
Pulses sent increased by 28%
Pulses sent increased by 28%
A few things came up in the Responsible AI reviews since this was
the first AI integration. Pushed out GA release.
Copilot Authoring Future Vision (Desktop)
Copilot Authoring Future Vision (Desktop)
Copilot Authoring Future Vision ( Native mobile)
Copilot Authoring Future Vision ( Native mobile)