TLDR: MTG sent out a survey about possible future directions for Secret Lair subscriptions and pricing. I was curious who received the survey, so I sent out my own survey about their survey. Here is the raw data I collected as a CSV, minus contact info if it was provided. I didn’t do a great job following up on this one. I’ve also made a better version of the survey, trying to learn from my mistakes.
About three weeks back, a lot of people who had purchased Secret Lair products received an email asking them to take a survey about Secret Lair products. I did not receive this email, but I heard from friends who did.
I don’t have any screenshots of the survey (if you do, please send them over to me, I’d like to include them in this article), but I reached out to a few of the folks who received the it. The general gist I got from talking to folks was something like this:
- The survey contained a variety of offers, most related to different subscriptions/subscription offers.
- The specific pricing on these offers ranged price based on the survey, and some folks who opened the survey multiple times stated that they saw the amount change, which implies there was some A/B testing going on regarding price point.
Regarding the specific offers themselves, there were a few different types I saw mentioned. These included same day shipping on orders, a discount on purchases, and subscriptions to all Secret Lair products for a year.
Now, all of this is very interesting, and I’d write about it more if I actually had the link to said survey, and screenshots. But I don’t. What I’m interested in is who WoTC sent the survey to. It seems like they were primarily targeting individuals who spend large sums of money on Secret Lair with this survey.
So, I did what any bored motherfucker who thought they could get a clickbait article for their blog out of it reasonable person would: I put up a Google form, posted it to Reddit and Twitter, and waited for some responses.
Unfortunately, I don’t think I got enough responses to reasonably answer my question, but I did promise to release my data in those posts, so here it is in csv format. I’ve removed the column that included contact information, but left everything else.
I do want to quickly go over why I don’t believe this data can be used to support my hypothesis which went something like “WoTC was specifically targeting folks who had spent lots of money.”
Lets start with the big one: this is an incredibly small set of responses. I’m incredibly grateful to the folks who took the time to fill it out, but 38 data points is nowhere near enough to begin drawing meaningful conclusions, and not everyone who responded had even purchased Secret Lair products.
In addition to this, this survey was from a specific subset of the population of folks who might have even seen it, i.e., Twitter and Reddit users. So we have a fairly heavy sampling bias to add to that as well.
(I think it’s actually fairly easy to manipulate this data to make the argument you want. For example, if you look at the average spend of folks who received the survey, they spent more than folks who didn’t. But if you look at actual numbers of folks who bought Secret Lair products and received the survey versus those who didn’t, there’s no clear cutoff. This is also why I’m not comfortable drawing conclusions with this data.)
On the other hand, I think there are some valuable lessons to be learned here, at least from my side. Let’s go over them briefly, shall we?
- Have a testable hypothesis before you randomly ask strangers to give you info. Pretty sure any science or stats teacher would smack me for what I did here, at least in terms of going “Hey, lets just gather a bunch of data, and then think about it.”
- If I’m going to try to do something that is time sensitive, I need to actually move quickly. Plenty of people gave me contact info. I didn’t reach out to them, partly because I had other work stuff, mostly because the new Path of Exile league came out, and I started playing that non-stop. (It turns out that doing meaningful data analysis and journalism is hard, who would have thought.)
- Reach out to the company involved. I probably could have just emailed WoTC. I mean, I doubt they would have responded, but who knows. It couldn’t have hurt to have tried.
So, in the spirit of being curious if WoTC is about to lean hard into whale fracking learning from my own mistakes, I’ve created a brand new survey that attempts to fix the problems with the first one. This time, we’re only interested in two things: if you received the survey, and how much you spent on Secret Lair in 2020, and 2021.
Again, I greatly appreciate anyone who takes the time to do so, and while I doubt this second survey will hit the numbers required to do meaningful analysis, I’ve tried my best to fix some of the flaws I noticed with the first one. And just like the first one, I’ll release whatever data I get publicly.