Beyond NPS - How to Build and Measure Trust with your Customers
Exploring the Trust Score we developed at Patreon
Hi there, it’s Adam. I started this newsletter to provide a no-bullshit, guided approach to solving some of the hardest problems for people and companies. That includes Growth, Product, company building and parenting while working. Subscribe and never miss an issue. If you’re a parent or parenting-curious I’ve got a podcast on fatherhood and startups - check out Startup Dad. Questions you’d like to see me answer? Ask them here.
Vanessa Van Schyndel is a User Experience Researcher at Figma where she leads research for FigJam. Prior to that she was the Head of UX Research at Patreon and a UX Research Manager at Hired. She has also held research roles at Yammer and Mindbody. While at Patreon she led the creation of the Trust Score. This article would not have been possible without her extensive contributions.
Q: How do I build and measure trust with my customers?
It was early 2019 and Patreon had just come off of a third, self-imposed crisis. It had become seemingly so common that I started calling it our Sideshow Bob rake-stepping moment.
The first was in December of 2017 when we announced that we were going to pass credit card fees on to creators’ patrons and the world melted down. Here’s how Vanessa remembers it:
“In one particular fumble, now remembered in infamy in creator circles across the internet as “The Fee-asco”, we announced that we were going to charge patrons an additional service fee per pledge. The rationale was that by charging patrons a bit more per pledge, Patreon could cover the costs of running the business while also giving creators a larger cut. We had other reasons too, but what we failed to do here was talk to creators before making this change.
Overnight, the announcement caused enormous backlash. Twitter was a dumpster fire. The support queue was full of messages ranging from confusion to violent threats. Creators were deleting their accounts left and right. Creators were furious at Patreon for stepping in between them and their patrons. We quickly rolled back the changes after a number of public apologies but the damage had been done. Creators no longer trusted Patreon to manage the financial relationship between them and their audiences. We had to figure out how to earn it back.”
The second rake hit us in the face around the middle of 2018 – this time it was an outage in our payments service - we were processing and sending so many transactions to creators that it finally tipped over. One of the downsides of our rapid growth. For several days or maybe even a week or more we couldn’t process and increment creators’ account balances and they couldn’t get their money out. Creators who depended on us for their rent payments or to pay contractors and employees, couldn’t get the money to do it.
The third rake came soon after, in December of 2018, when we removed a few controversial creators on the platform for violation of our policies.
Were we justified? Absolutely.
Did we communicate it well? We did not.
The impact was swift and immediate: loss of loyalty, churn, reduced spend and slowed month-over-month growth. The work we had done to rebrand and reposition the company as a top-tier membership platform was at risk.
One of the core tenets of our independent platform was that we were trustworthy. Much more trustworthy than the FAANGs of the world. And yet, over the course of ~1 year we had eroded that trust significantly.
We knew that we would continue to make changes to our platform and with that would come mistakes. But how could we make mistakes while still being viewed as a trustworthy company by our customers? How could we make sure they would stick with us rather than flee at the first hint of an issue.
We had been tracking our NPS for years and saw that followed creator sentiment pretty closely (albeit a lagging indicator). The challenge was it didn’t help us get to the root of the issues and a lot of the verbatim was using words like “trust” or “trustworthiness.” We had to dig deeper.
We would need to both measure and re-build that trustworthiness. This was especially important because we knew that we needed to make some big changes to our pricing and packaging in the coming year. Those changes had the potential to make or break our business.
The Components of Trust
We knew we had a trust issue, but we didn’t know how to get to the root of the issue. The first step was to understand the components of trust.
As of 2022 there is a great HBR article about trust, written by the authors of a book called The Four Factors of Trust. But in 2019, both this newsletter and that book didn’t exist.
So what were we to do?
It turns out that in 2019 there were at least a few decades of academic research on this topic; including from a couple of scholars named Dr. Linda Childers Hon and Dr. James E. Grunig who studied the roles of communications in building trust.
Jackpot!
They defined three measurements of trust known as the Grunig Trust Dimensions.
Competence: do you have the ability to do what you say you’ll do?
Dependability: do you act consistently and reliably?
Integrity: are you fair and just?
Now that we had our agreed-upon measurements the next step was to track them.
Measuring Trust
We wanted to be able to create a score and track progress against it over time. The approach we settled on was a longitudinal survey (one of the quantitative measures discussed in Mastering Product Discovery).
This monthly survey contained trust questions derived from the three elements of trust, service category questions (communications, content policy, payments, support, etc.), and open text questions. It used a 1 to 5 “agreement” score for each question. If you answered a 4 or a 5 you agreed with the statement; below that you did not.
Here are some examples of statements about trust:
Patreon treats creators like me fairly and justly.
Whenever Patreon makes an important decision, I know they will be concerned about creators like me.
Patreon can be relied on to keep their promises.
I believe that Patreon takes the opinions of creators like me into account when making decisions.
I feel very confident in the Patreon team’s skills.
Patreon has the ability to accomplish what they say they will do.
Sound principles seem to guide Patreon’s behavior.
Patreon does not mislead creators like me.
I am very willing to let Patreon make decisions for creators like me.
I think it’s important to watch Patreon closely so that they don’t take advantage of creator’s like me.
Patreon is known to be successful at the things they try to do.
Patreon delivers on the expectations that they set.
It was not good enough to take responses in aggregate, we also wanted to segment the results by earnings. Creators felt very differently about our platform and communication across various monthly income levels.
The survey helped us create an ongoing “Trust Score” that we could track over time. It provided enough granularity so we could see what components of trust were causing an increase or decrease in the score.
We could further segment by service category to zero in on specific areas of opportunity, for example: communications, payments, support and content policy were all areas of opportunity.
Within a service category we could get further feedback on how we were doing with specific elements. In this particular breakdown we could see that “transparency” was holding us back.
This was all well and good, but understanding and measuring trust is just part of the puzzle. You also have to decide to do something about it.
Taking Action on Trust
We presented the data and findings along with qualitative responses in our Monthly Business Review.
Over the course of a few months several patterns emerged. Here were some of our observations.
Over-communication. When we re-did our pricing we prioritized early outreach to creators. We saw our communications score improve significantly.
We added more support team members which reduced the queue and response time. This translated to an immediate lift in our dimensions of “support.”
Launching new features had limited or no impact on trust.
When we had bad news or issues a focus on transparency created a lift in our communication scores, but our dependability score would decrease if it was a core area that creators needed to function well at all times.
With further segmentation we learned about the importance of other initiatives as it related to trust, for example those who participated in our online communities or had a named account rep reported much higher trust scores.
Success with Trust
Over the course of many months we were able to diagnose and repair our trustworthiness by proactively addressing issues across the three dimensions of trust. We still took actions that had the potential to damage trust, like raising prices and sunsetting features, but because of the work we did those changes landed quite well with our creator community.
The feedback from creators went from this:
“Patreon is a good way to ensure a steadier stream of income for creatives, and is on the whole a good option, alongside other revenue streams. However, you are also atrocious at communicating changes to your service ahead of time and you clearly do not give a shit about smaller creators. You need to do better.”
“Your newly announced and entirely subjective anti-free speech position is simply untenable and has soured me to the idea that you are in any sense whatever a trustworthy organization for me to recommend to anyone.”
To this:
“For a series of changes that seem extremely necessary, I think y'all did a great job explaining all things considered. It's a big complex thing! I also want to say that I appreciate how this update directly addresses some of the feedback I had for the Patreon team at PatreCon (in particular, the request for team accounts...) but in a more thorough way than I was expecting, and I'm pleased with this rollout.”
“If you know me, you know I’m a curmudgeon about these sorts of things. But Patreon consulted with tons of people. They listened to our feedback. They walked through complaints. I can’t say I agree with every bit of what’s coming but it really was a genuine and thoughtful process.”
Flaws
Trust score is not a perfect measure, although for Patreon it was significantly better and more actionable than NPS. Here are three flaws with the trust score:
Subjectivity
Trust is a subjective measure and can vary greatly among individuals. It’s subjective in both its measurement and its interpretation. Our Trust Score may not have fully captured all of the nuance of our audience – especially as we further segmented and reduced the sample size.
Heavily Quantitative
Our goal was to create a quantitative measure and that’s exactly what we did. But the overreliance on quant data can overlook the qualitative aspects that are harder to measure. We did capture verbatims and open-text field responses but we didn’t conduct in-depth sentiment analysis on those responses. Trust is built and eroded through complex interactions and experiences that might not be fully captured by a numerical score.
It also didn’t capture those who had already given up on us and churned. It was designed to understand trustworthiness amongst current creators and not a churn analysis. We knew that trustworthiness was correlated to churn, but didn’t have the statistical evidence to say that it was completely causal.
Potential for Gaming or Misinterpretation
This is true of any metric that becomes a key performance indicator. Teams will typically end up optimizing for what moves the score vs. truly solving underlying issues. You hope that solving the issues moves the score, but there is always doubt and uncertainty.
We guarded against this by not setting trust score goals and instead leveraging it as inputs into our workstreams on planning and releasing product improvements and changes.
Closing Thoughts
The introduction of the Trust Score at Patreon didn’t magically improve everything. It did shed light on some significant areas of opportunity for us and provide a level of granularity we had been sorely lacking with our other surveying mechanisms.
More importantly, it gave us the knowledge necessary to roll out our largest pricing change to enormous fanfare instead of vitriol.
For some brands, trustworthiness doesn’t matter. These brand tends to recede into the background as commodity products. However, if you manage money and are a platform that can influence the livelihoods of those who use it then it is definitely a measure to pay attention to.