Deeper onboarding guidance

Last fall, I had the opportunity to run a content-driven A/B test on an onboarding surface known as the persistent guide (PG).

I hypothesized that by providing deeper, layered onboarding guidance for complex tasks, we can help merchants complete more setup tasks and start selling faster.

Before:

  • Guidance was minimal (1-2 sentences per task) and often repeated the same message from the Home guide

  • Some argued that the PG was only successful because it was being used primarily as a navigation tool and that the content inside the panel didn’t really matter

  • Recommended actions like “Add sample product” took up lots of real estate and only allowed for one suggestion per task

  • Help Center and support chat were still getting lots of requests around onboarding tasks

Step 1: Write up a project brief

As the project champion, I was responsible for leading and documenting the entire project from start to finish. The brief contains our hypothesis, plan of action, stakeholders, and outcome.

Previously we saw that merchants who completed more onboarding tasks were more likely to reach L2C (lead to conversion) and 2in14 (2 sales in 14 days). We set a goal to improve L2C, 2in14, and individual task completion rates.

Step 2: choose which tasks to include

There are over 100 onboarding tasks available for the Setup Guide at a given time. The average setup guide has 6 or fewer tasks, so the team decided to test 6 tasks for this experiment. I partnered with a data scientist to identify the 6 most commonly shown tasks according to our relevancy algorithm. They were:

  • Add products

  • Customize theme

  • Set shipping rates

  • Add domain

  • Set store name

  • Set up payments

Step 3: meet with subject matter experts

Once the tasks were chosen, I met with subject matter experts for each task. This included content designers, product designers, product managers, and product marketing. I asked questions about the most common user problems, frustrations, frictions, questions, concerns, knowledge/skill gaps, etc. Before meeting, I mapped out their surface areas to get a better understanding of all the possible actions a new merchant could potentially take when onboarding to their surface.

Step 4: start crafting

I took this new infomation from SMEs and started drafting updated tasks using our latest PG UI. As I explored, new patterns started to emerge and high-level principles started to solidify.

Some examples include:

  • The persistent guide is meant to complement the on-page guidance and should never explain how to use the UI – that’s the job of the page

  • Tone is a friendly, professional commerce coach

  • Help new merchants understand what “good enough for now” looks like

  • The complexity of the guidance should match the complexity of the task

Step 5: design, refine, repeat

I explored a version using our new Uplifted persistent guide UI, but it felt flat and didn’t accommodate for the level of detail we needed. So I partnered with my visual design partner to craft an entirely new PG IA and content structure from scratch. We tried several approaches such as:

  • Bulleted lists

  • Bolded headers + short paragraphs

  • Multi-step and multi-page tasks

  • Stepped subtasks

  • Nested content

  • “Smart brevity”

  • Inline hyperlinks

Step 6: test and learn

We launched the experiment and the results were very positive.

  • 3.85% relative lift in merchants reaching Shopify Payments set up milestone - this could signal an increase in trust and investment in launching their business with Shopify

  • 0.6% incremental lift on L2C in 5 days and an audience that includes desktop and mobile signups, annualized this looks like 21.6K new merchants and $6.5m 3 year GP

  • While other milestone completion rates are more neutral between control and exposure this doesn't capture an increase in quality of completion

  • This experiment also gives us reason to believe the PG is not just beneficial as a navigation tool, but as a qualitative tool to help merchants complete setup tasks.

  • Up next: I would love to user test the PG and get more thoughtful qualitative feedback on WHY this version works, how merchants use it, and what could be improved.

Step 7: document and scale

Once the A/B test wrapped and we determined it was a success, I worked with my design counterpart to craft a Figma UI build kit and a content design guidebook for other teams to use going forward in hopes of scaling the PG beyond the original 6 tasks and surfaces.

In addition to the primary guidance content, we built a suggestions system with recommended actions. One of which was a secondary page where merchants could open a summarized Help Center article to read about a relevant topic. The others triggered Shopify Magic, navigated internally, or navigated externally.