First things first, and the only truly pertinent thing to say in this space: thank you EHG for continuing to value community input into your development of LE!
Now then. I have a bone to pick over polling systems like what we have in this case. I advocate for modern, more representative voting systems both online and otherwise, and I’m always surprised when companies put product decisions to their customers for vote using traditional plurality polling. Especially when using modern software on the modern internet, it’s trivial to employ a modern method of assessing the preferences of a population.
Yes, I do appreciate that a crowdsourced item design for a video game would be a small hill to die on. However, this series of community polling turned out a perfect example of why discussion about voting systems is important. And with the potential that lack of awareness is the underlying problem, here I go.
For those new to this topic: this kind of voting system where each voter makes a single selection from the list of options is called first-past-the-post voting. I will not get political here, I only want to focus on a couple non-political specifics. FPTP has problems, and here we got a clear depiction of two of them.
First and most importantly, FPTP is highly conducive to leaving an unrepresented majority. In other words, the winner gets selected by less than the majority, sometimes by a huge margin. Let’s look at the first poll results from the series:
| CLASS |
|---|
| 24 Mage |
| 21 Acolyte |
| 20 Rogue |
| 19 Sentinel |
| 16 Primalist |
We see here an easily understood winner, as the highest ranked option did indeed recieve more votes than any one other option. But consider the inverse: 76% of voters wanted something OTHER than Mage to win that poll. Since there is only one winner and all others are not, we can count the votes in just those two categories: won and lost. Mage won this poll with a mere 24% of the vote, even though in the total vote he was outranked by more than 3 to 1. This is incredibly unrepresentative of the overall preference of voters.
The second problem is called vote splitting, which further exacerbates the first problem. I find this concept is more common knowledge, so put simply: the more choices there are, the more likely an outlier wins. Most of the results of the first five polls show a remarkably linear progression from highest to lowest ranking. Here are poll results two through five:
| CATEGORY | BASE TYPE | THEME | EFFECT |
|---|---|---|---|
| 25 Two Handed | 28 Spear | 23 Lost Refuge | 32 Volcanic Orb |
| 23 One Handed | 26 Staff | 22 Tri-Elemental | 29 Meteor |
| 19 Armor | 20 Axe | 18 Amalgamation | 19 Firebrand + Fireball |
| 17 Jewelry | 14 Sword | 14 Void | 17 Nova |
| 16 Off-hand | 12 Mace | 08 Lagon | 03 Firebrand + Damage |
| 08 Apophis | |||
| 07 Osprix |
We see from those results that voters had a relatively even spread for their first (aka only) choice in each poll.
For easier vote splitting maths and explanation, imagine one of the poll results looked like this:
| IMAGINARY POLL |
|---|
| 46 First |
| 16 Second |
| 16 Third |
| 16 Fourth |
| 06 Fifth |
Even though 46% is still not very representative of the population, it won by a clear margin and almost reached an overall majority. As such, voters might feel more confident in results like this, compared to the others above. First place won by 30%, so who could argue, right?
However, look at the similar percentages for the middle three places. This indicates that those three options were likely similar enough that if one was excluded, those votes would instead have gone to one of the remaining two. Further, if those three options were instead only one, it is probable that all of those votes would have been consolidated entirely, and the winner would be different. Hence, vote splitting (and also why “two party” systems are a thing).
In our actual results, the votes were split more evenly, suggesting that voters considered all options with relatively similar preference in each poll. For the sake of a community video game poll, I somewhat consider this a good thing. EHG asked, “What do you want in the game?” and the community responded, “I want all of these!” What a happy agreeable bunch of gamers we LE players are!
So, again, I am happy for this series of polls and I’m perfectly content with the results in this context. But that brings me to my next point: this is a SERIES of graduated FPTP polls to reach a single final result. For sake of argument, I’m going to assume some things for trends between successive polls.
Keep in mind the overall goal of the first five polls: to invent the functional characteristics of a new unique item. Five choices, with each beyond the first two only being made after the previous poll closes. Now that we have all the closed results, consider the winning selection in each. How many voters do you think voted for all five of the winning options? How about three out of five? How about one or none out of five?
Perhaps there’s already some expert or app out there that can crunch the statistics, but I’m just running this roughly by hand to illustrate the point: with each successive poll, overall representation of voter preference decreases.
On to my for-argument’s-sake-assumptions. Let’s assume the voters who picked the winning option in each poll were spread across the options for the next according to that next poll’s results. This is just simple multiplication, so the order is unimportant to the math, but the order of the polls is very likely important to the voters.
We start off with 24% of voters selecting Mage. For the second poll, 25% overall went for Two Handed, which would include 6 of the 24% who voted Mage. Then 28% of those 6 chose Spear: 1.68. Then 23% for Lost Refuge: 0.39. And finally, 32% for Volcanic Orb: 0.12. So, using this very basic model, we’d have a miniscule 0.12% of all voters who, based on their selections, are 100% in agreement with the final results after five rounds of design choices.
Of course, the presentation of each later poll included the information from the previous, which would affect voter opinions for them. In isolation, the question “We’re making a two handed item for the Mage, what base type do you guys want?” is significantly different from “We’re making a new weapon, what base type should it be?” So, let’s skip some other talking points and just change the assumption to be VERY generous to the polls, in terms of representation. Let’s assume that the maximum possible number of voters picked five of five winning options. This is easy to figure out: whatever is the lowest value among the five poll winners would be that maximum. In this case: 23%. So, at best, only 23% of voters are 100% in agreement with the final results.
There are, of course, a lot of other points of analysis we could explore. For instance, maybe 70% of voters selected 4/5 or better, and 80% of voters say that’s good enough. Once again, I am perfectly happy with the item we’re getting. But, that is to say: I would be just as happy with any other combination of design selections, and it is the poll results themselves making me feel that way.
Imagine yet another alternate final result, in which the winning selection of each poll won with over 90% of the vote. That would indicate a (relatively) very good representation of the population’s preferences, and further suggest that polling in this way actually did value the input from the population. However, this method of successive FPTP polling with so many options will almost always result in what we instead got here: a series of vote-splitting that is not significantly different from just rolling dice.
TLDR: there is little value in community input collected using this method, which I believe does not come close to matching how much EHG values community input in principle.
So, how can this be done better? The easy answer is to just use a better voting system, such as ranked-choice. If for some reason another system is not feasible, FPTP can be used for a series of sub-rounds to each top-level round of voting. For example, Poll-1-Day-1 naturally includes all the options, Poll-1-Day-2 includes just the top popular options from Day-1 results, then repeat until something wins with majority. This is obviously tedious, and hinders participation, but guarantees the winner claims a majority in the final poll.
On a parting note: I don’t claim to be any kind of mathematician, statistician, or scientist. If anyone has such expertise, please enjoy any poking of holes in my arguments that you think will help the conversation. I only have two simple goals here: to obtain, if possible, a better method of polling the community, and to get people thinking about voting systems.
Thanks for reading, and may your Legendary Potential be high.
It’s certainly possible we could do something like this again in the future, but with improvements from lessons learned during this event.
That said, such an option is quite a big variable that merits consideration.