10 Years of Experiments

It’s not everyday a business turns 10 years old. Only ⅓ of businesses even make it to 10 years. But on July 6, 2021, Paid Memberships Pro turned 10, and we couldn’t be more proud.

One of the hardest aspects of running a business is that there is no playbook. There are a billion business books that offer generic advice—but not for your business specifically. So there’s a lot of head scratching, asking peer groups, and taking risks.

In the spirit of taking risks safely, we have run many experiments over the past 10 years. Some of them worked and became part of our strategy while others “failed” and made us re-think our assumptions. 

Successful entrepreneurs can use experiments like these to run tests, gather data, and learn from the findings. With that in mind, here are the top 10 experiments we ran and what we learned.

1. Testing a Price Increase

When we first started Paid Memberships Pro, we only offered a few Add Ons. Today, our platform includes over 65 premium Add Ons and with this growth, we increased our price accordingly. But, before we increased the price of our Plus level from $197 to $297, we tested the price increase.

We ran an A/B test. Half of users saw the $297 price everywhere on the website except the checkout page. At checkout, we truly charged this half of users $197 and displayed a message about a secret “Fall sale”.

While this test wasn’t perfect, it did allow us to see how the price increase would affect conversion rates from the homepage to the pricing page and from the pricing page to the checkout page.

We expected sales to fall (pun intended), but we also were charging 50% more. The higher price would drive better overall revenue as long as orders fell less than 33%. In our tests, sales were nearly even. The experiment proved that we could increase the price with confidence. Jason talks more about this experiment and pricing in general in his Art of Value Podcast appearance.

Takeaway: A/B test major changes to your product or membership. Sometimes you spend too much time worrying about nonissues.

2. Happy Banner FOMO

You may notice some websites with a banner at the top mentioning a 5-10% deal that ends in exactly 3 hours and 53 minutes. If you’re an astute web user you probably noticed that these deals are fake. They create artificial fear of missing out — or FOMO.

We don’t like lying to our customers. It sets a bad precedent. In practice, everyone gets this offer every time they come to your site. And is it really a sale if it’s always available? Is this illegal in some countries? Probably. But we don’t know the laws in every country. 

We didn’t feel completely comfortable using the FOMO tactic, but… if you look at the case studies, the tactic works. After all, customers like purchasing stuff on sale, plus countdown timers add a sense of urgency that can nudge people into purchasing now.

Queue the experiment. Are we leaving money on the table by not using the artificial FOMO  tactic? We created a version of this strategy that we were comfortable running.

We called this experiment the “Happy Banner”. The banner read: “Buy now and save 10%. Studies show customers who get a deal are happier, and we want you to be happy.”

We ran an A/B test where 50% of visitors to the site got that message and deal, while 50% were shown the full price. We did make a bit more money while running the banner, but in the end it was not enough of a difference to enter this kind of shady territory. Running a constant sale like this is also likely to negatively impact the seasonal sales we do run.

We suspect a banner with more “urgency” would perform better. We also suspect that part of the difference in revenue from the test might be due to funneling more people into our pricing page vs. going directly to checkout. So we’re coming up with some experiments to try there.

Takeaway: If there are best practices that tell you to do something uncomfortable—give it a test. See how much money you’re leaving on the table. Sometimes when you have hard numbers it’s easier to say “Ah for $1,000 a week I’d rather not engage in that practice.”

3. Hire in Advance of the Need

As we get ready to hire our 15th employee, I still remember how hard it was to hire our first full time employee. It feels like such a commitment, and it is. One thing that helps us as we expand is to hire early and view new hires as an investment. 

We target 25% profit margins in the business and have generally hit that target. At the start of a new year, we will set aside an amount equal to the profit from last year and use that to hire new folks—even if we don’t immediately, urgently, need people.

The benefits are subtle but hugely important. By the time we realize we have a need (we need at least 5 people in customer service) we already have people on our staff, they’re fully trained, and fully integrated into the team. If we didn’t hire in advance of the need we’d spend a month looking for the right candidate, and 2-3 months just waiting for them to get up to speed. 

And since we’ve been hiring in advance of need we are profitable every single year which gives us more margin to invest in the business and keep growing.

Takeaway: Don’t be afraid to use profit to grow your company. Solving problems when they pop up is efficient for your money. Solving problems before they come up is effective for your business.

4. Monthly vs. Annual Pricing

Way back in 2012, when we first launched the site, our membership was $19 per month. Many folks liked this price, and we had a good number of happy members ready to pay that price every month for access to our support, documentation, and good will.

However, most members were canceling their membership within 6 months. The average member stuck around for 3 months, paying us only $57.

We realized there was a disconnect between when people paid and the value they received. With PMPro you get all of the value immediately when you set up the plugin. Then each subsequent month feels expensive and wasteful. Changing to annual pricing more closely aligns our goals with our customers’ goals. 

Takeaway: Make sure that the timing of your pricing matches the timing of the value.

5. “Buy Now” CTA on the Homepage vs. “View Pricing”

It’s a good idea to test your intuitions. One of our intuitions is that most people wouldn’t want to immediately purchase Paid Memberships Pro. They probably want to read about the features, then compare plans, and then purchase.

And if we trusted that intuition (or assumption) we would have left money on the table. That’s why we A/B tested our homepage CTA and found that linking directly to the checkout page for Plus converts better: more people complete checkout after clicking this CTA over those that click through from the Pricing page.

Here’s what we learned with this A/B test: your homepage should have one or more links that are just one click away from your most popular product.

We are due to retest this one. It’s possible our pricing page is better today, or at least better than that one we tested with. It’s possible that refund rates are higher for folks who are pushed directly to the checkout page, and we should make sure to include that in our tests.

But overall, make sure it’s easy for people to pay you. Don’t force people to read through everything before they get to the payment part. Remember that not everyone is coming to your homepage for the first time. Many of the people visiting your homepage and your website are already familiar with your membership, so helping them join and pay ASAP will boost your performance.

Takeaway: Test your intuitions. It’s worth doing A/B tests on things you’re pretty sure are working.

6. Writing Transparency Reports

One of the ways we give back to the community is by writing transparency reports. It helps new business owners understand cash flow, how to hire, how to manage software, and more. It also forces us to take a close look at our own numbers. And reviewing these numbers from time to time teaches you what is working and what needs to change.

We are not operating this business just to make money (shocking, I know). We are helping entrepreneurs large and small with a software platform that helps them get paid, find financial freedom, and achieve their goals. Transparency reports help those people get to know us as a company, and also as human beings. We’re committed to supporting and growing this plugin for many years to come (another few decades, even).  

Aside from the customer-facing aspect of these reports, they also support our recruiting efforts. When applicants ask why they should join our company, we point to our transparency reports as a way to “get an inside peek” at how we run and operate. The report helps to build the trust that we are growing, that we are here to stay.

Takeaway: You can help yourself and your community by sharing regularly. You don’t have to share everything but share some important details and you might be surprised how they improve over the next couple years.

7. PMPro Core vs. Plus vs. Unlimited Plans

When we started Paid Memberships Pro and were still figuring out a pricing model, we didn’t offer automatic installs and updates for our Add Ons. We started by just offering downloadable .zip files of all the open source plugins, and charged customers for technical support. This membership was called “PMPro Core”.

Three years later we launched our Plus plan which includes premium Add On installs, updates, and super in-depth technical support. People had the choice of Core (support for the core plugin only) or Plus (support for everything + all the code served to them). Unfortunately, this assortment created loads of confusion for users who wanted to upgrade. And some of the people on our support team noticed that users would download the Plus Add Ons on GitHub, sign up for the less expensive Core level, and receive support for products they didn’t technically qualify for.

We realized we made a lot of work for ourselves. We had to simplify pricing. So we said goodbye to PMPro Core and now we only offer the Plus plan (and our Unlimited Plan for people with more sites).

The Unlimited Plan is there for users who might want to run PMPro on dozens or hundreds of sites. But it’s also there as a price anchor. By putting such a large package next to our Plus product it makes the Plus product feel cheaper.

Takeaway: Products evolve. What made sense one year might not make sense the next. Go through your products at least once a year. Do they still make sense for your audience? If they don’t, update or remove them.

8. Only Offering Automatically Recurring Subscription Plans

The business community loves annual subscriptions. And contrary to popular belief, the business model here is NOT to trap people into paying you because they forgot to cancel.

We don’t want those customers. Send a reminder prior to renewal so members who no longer need your product have a chance to cancel. Honor their refund requests when they realize they accidentally renewed.

The real reason automatic subscriptions are valuable is that members who want your product and are happy to pay for your membership inevitably, occasionally, fail to renew. It might be that they’re busy. Maybe they tried to renew but forgot their password and got distracted. Maybe they didn’t get the reminder when they were ready and able to pay.

We’ve seen many businesses go through this process. In year one, they offer a one-time payment for membership that expires in one year. The next year, they make renewals optional, perhaps using our Auto-Renewal Checkbox Add On. This step increases renewals and total revenue by as much as 25%. Then finally in year three, they realize that all checkouts should default to recurring subscription, increasing renewals AND total revenue by another 25%.

Don’t take three years to figure this one out.

Takeaway: Consider starting with default or mandatory recurring subscriptions. Some folks will want one-time payments and you can have a special discount code for them or allow them to cancel and retain membership until the date their next payment would have been, but for many membership sites, it makes sense to stick to recurring subscriptions.

9. Charging for Support

Some open source software developers get frustrated by the free support they are expected to provide for their plugins and apps. They rightly realize the need to get paid for their work, but what complex service, premium offering, or sponsorship system is the right way to make money?

In reality, they might just be able to figure out a reasonable rate they’d like to get paid, and charge that for support. That’s what we did. We found that push back in the channels we used to give free support were minimal, and many people were happy to pay. We were able to start with a simple business model to continue development on our platform and business.

Takeaway: Just like products, business models can change. Something that works for the first couple years can give you the knowledge and experience to pivot into a more sustainable business model later.

10. Making All of Our Plugins Available on Github for Free

We believe in open source software. We believe this license is the best way to make the best software.

And, while it is legal and reasonable to charge for open source software, and we do so ourselves, we do not believe in making open source code inaccessible to everyone. We believe that publicly available software is the best way to fully embrace open source. We share our source code publicly so people are able to find and use it for free.

We’ve checked and this hasn’t had a material impact on our sales. And we have a few theories why:

• Many folks start using our software for free because they don’t have a budget. And they upgrade to a paid plan after they built their business around our software. These people are the best kind of customer because they have already validated product fit.
• More users means more referrals. This works particularly well with developers who often try tools and recommend them to others. They might not have been interested in trying a paid plugin, but they are happy to experiment with a free plugin.

And unrelated to revenue, having our repositories public means more code and more ideas get contributed by others. This improves both the quality and the features of our platform. And further still, free access to our code protects users from downloading a “nulled” version of the code base that could be hacked, insecure, and possibly riddled with malware.

Takeaway: Customers buy for a variety of reasons. Access to the codebase is just one of them. So making code repositories open to developers likely won’t have a large impact on your bottom line, and we don’t worry about it. The pros outweigh any cons.

11. Black Friday Sales

We held off on sales for a long time because we believed that running sales can make your product look cheap. And that running frequent sales trains your customer to expect (and wait) for a sale. 

This is true. 

At the same time, running a small number of sales, particularly one around Black Friday, can still be a good idea. Many downsides are minimized around Black Friday when every business is expected to run sales. Running a small number of sales throughout the year will make your product accessible to customers who normally would not buy. And you don’t need to lower your price for the rest of the year.

We believe in Black Friday sales so much that we built a plugin, Sitewide Sales, to make it as easy as possible to set one up for your PMPro or WooCommerce website.

Read more posts on why and how to run a good sale on our Stranger Studios blog.

Takeaway: There are ethical ways to run sales, help your customers, build your brand, and be profitable. Don’t be convinced that all sales are “too promotion-y” or “attract the wrong type of customers”.

Wrap Up

We’re grateful and fortunate to be here after 10 years. Another 3 years and PMPro can go see a PG-13 movie without a guardian 🙃.

We want to help you grow your nest egg. One of the best pieces of advice we can give you is to run your own experiments. Not all of our experiments were successful. Sometimes they gave us unexpected results and we had to rethink our business model. As long as you’re learning you have a chance to improve your business.