Are there really no better voting systems built into forums? An Analysis of the Community Designed Unique Poll Results

First things first, and the only truly pertinent thing to say in this space: thank you EHG for continuing to value community input into your development of LE!

Now then. I have a bone to pick over polling systems like what we have in this case. I advocate for modern, more representative voting systems both online and otherwise, and I’m always surprised when companies put product decisions to their customers for vote using traditional plurality polling. Especially when using modern software on the modern internet, it’s trivial to employ a modern method of assessing the preferences of a population.

Yes, I do appreciate that a crowdsourced item design for a video game would be a small hill to die on. However, this series of community polling turned out a perfect example of why discussion about voting systems is important. And with the potential that lack of awareness is the underlying problem, here I go.

For those new to this topic: this kind of voting system where each voter makes a single selection from the list of options is called first-past-the-post voting. I will not get political here, I only want to focus on a couple non-political specifics. FPTP has problems, and here we got a clear depiction of two of them.

First and most importantly, FPTP is highly conducive to leaving an unrepresented majority. In other words, the winner gets selected by less than the majority, sometimes by a huge margin. Let’s look at the first poll results from the series:

CLASS
24 Mage
21 Acolyte
20 Rogue
19 Sentinel
16 Primalist

We see here an easily understood winner, as the highest ranked option did indeed recieve more votes than any one other option. But consider the inverse: 76% of voters wanted something OTHER than Mage to win that poll. Since there is only one winner and all others are not, we can count the votes in just those two categories: won and lost. Mage won this poll with a mere 24% of the vote, even though in the total vote he was outranked by more than 3 to 1. This is incredibly unrepresentative of the overall preference of voters.

The second problem is called vote splitting, which further exacerbates the first problem. I find this concept is more common knowledge, so put simply: the more choices there are, the more likely an outlier wins. Most of the results of the first five polls show a remarkably linear progression from highest to lowest ranking. Here are poll results two through five:

CATEGORY BASE TYPE THEME EFFECT
25 Two Handed 28 Spear 23 Lost Refuge 32 Volcanic Orb
23 One Handed 26 Staff 22 Tri-Elemental 29 Meteor
19 Armor 20 Axe 18 Amalgamation 19 Firebrand + Fireball
17 Jewelry 14 Sword 14 Void 17 Nova
16 Off-hand 12 Mace 08 Lagon 03 Firebrand + Damage
08 Apophis
07 Osprix

We see from those results that voters had a relatively even spread for their first (aka only) choice in each poll.

For easier vote splitting maths and explanation, imagine one of the poll results looked like this:

IMAGINARY POLL
46 First
16 Second
16 Third
16 Fourth
06 Fifth

Even though 46% is still not very representative of the population, it won by a clear margin and almost reached an overall majority. As such, voters might feel more confident in results like this, compared to the others above. First place won by 30%, so who could argue, right?

However, look at the similar percentages for the middle three places. This indicates that those three options were likely similar enough that if one was excluded, those votes would instead have gone to one of the remaining two. Further, if those three options were instead only one, it is probable that all of those votes would have been consolidated entirely, and the winner would be different. Hence, vote splitting (and also why “two party” systems are a thing).

In our actual results, the votes were split more evenly, suggesting that voters considered all options with relatively similar preference in each poll. For the sake of a community video game poll, I somewhat consider this a good thing. EHG asked, “What do you want in the game?” and the community responded, “I want all of these!” What a happy agreeable bunch of gamers we LE players are!

So, again, I am happy for this series of polls and I’m perfectly content with the results in this context. But that brings me to my next point: this is a SERIES of graduated FPTP polls to reach a single final result. For sake of argument, I’m going to assume some things for trends between successive polls.

Keep in mind the overall goal of the first five polls: to invent the functional characteristics of a new unique item. Five choices, with each beyond the first two only being made after the previous poll closes. Now that we have all the closed results, consider the winning selection in each. How many voters do you think voted for all five of the winning options? How about three out of five? How about one or none out of five?

Perhaps there’s already some expert or app out there that can crunch the statistics, but I’m just running this roughly by hand to illustrate the point: with each successive poll, overall representation of voter preference decreases.

On to my for-argument’s-sake-assumptions. Let’s assume the voters who picked the winning option in each poll were spread across the options for the next according to that next poll’s results. This is just simple multiplication, so the order is unimportant to the math, but the order of the polls is very likely important to the voters.

We start off with 24% of voters selecting Mage. For the second poll, 25% overall went for Two Handed, which would include 6 of the 24% who voted Mage. Then 28% of those 6 chose Spear: 1.68. Then 23% for Lost Refuge: 0.39. And finally, 32% for Volcanic Orb: 0.12. So, using this very basic model, we’d have a miniscule 0.12% of all voters who, based on their selections, are 100% in agreement with the final results after five rounds of design choices.

Of course, the presentation of each later poll included the information from the previous, which would affect voter opinions for them. In isolation, the question “We’re making a two handed item for the Mage, what base type do you guys want?” is significantly different from “We’re making a new weapon, what base type should it be?” So, let’s skip some other talking points and just change the assumption to be VERY generous to the polls, in terms of representation. Let’s assume that the maximum possible number of voters picked five of five winning options. This is easy to figure out: whatever is the lowest value among the five poll winners would be that maximum. In this case: 23%. So, at best, only 23% of voters are 100% in agreement with the final results.

There are, of course, a lot of other points of analysis we could explore. For instance, maybe 70% of voters selected 4/5 or better, and 80% of voters say that’s good enough. Once again, I am perfectly happy with the item we’re getting. But, that is to say: I would be just as happy with any other combination of design selections, and it is the poll results themselves making me feel that way.

Imagine yet another alternate final result, in which the winning selection of each poll won with over 90% of the vote. That would indicate a (relatively) very good representation of the population’s preferences, and further suggest that polling in this way actually did value the input from the population. However, this method of successive FPTP polling with so many options will almost always result in what we instead got here: a series of vote-splitting that is not significantly different from just rolling dice.

TLDR: there is little value in community input collected using this method, which I believe does not come close to matching how much EHG values community input in principle.

So, how can this be done better? The easy answer is to just use a better voting system, such as ranked-choice. If for some reason another system is not feasible, FPTP can be used for a series of sub-rounds to each top-level round of voting. For example, Poll-1-Day-1 naturally includes all the options, Poll-1-Day-2 includes just the top popular options from Day-1 results, then repeat until something wins with majority. This is obviously tedious, and hinders participation, but guarantees the winner claims a majority in the final poll.

On a parting note: I don’t claim to be any kind of mathematician, statistician, or scientist. If anyone has such expertise, please enjoy any poking of holes in my arguments that you think will help the conversation. I only have two simple goals here: to obtain, if possible, a better method of polling the community, and to get people thinking about voting systems.

Thanks for reading, and may your Legendary Potential be high.

6 Likes

FanTASTIC post! Salute to you (is that what people do? idk haven’t seen the Sun in weeks)! I agree with your points and, as a mathematician, your reasoning seems sound (although a statistician may tear us both a new one, who knows?).

If I may add to this, you are right about the Poll-1-Day-1, Poll-1-Day-2, … option being tedious and also contains flaws because it took several days for each of the 800-1,200 votes to come in. Perhaps one way to tackle this is, after the first week, each week has two polls, one for the current week and one to narrow down the top options further.

Or maybe a weekday poll and a smaller weekend poll?

While in the final results there were several close outcomes that made it seem like the vote could sway either way, we can actually use some more statistics to turn these smaller secondary polls into an poll of 2 things.

This process is called Re-Randomisation (or Bootstrapping). It’s vaguely covered in the final year of high school, but is definitely gone into at university. We write down a dataset, then re-roll each piece of data (usually thousands of times but I have other stuff to do).

This is used to measure how reliable the outcomes of your data are (ie. if it was a fluke or if you would consistently find such results from the same dataset). It’s a way to check that if the experiment was repeated, what kind of results would you expect and how would they match to your original findings?

Simple example to understand this better

Let’s say I have the following dataset of 6 points:

Data
1
1
3
4
4
5

For simplicity for this example, let’s say we are looking for the median of this data. We can find other stuff too (my graphs later use mode, for instance), but median is the easy one.

The median of this dataset is 3.5.

Now let’s imagine I have a 1d6 (6-sided dice to non-DnD nerds). The numbers on the d6 represent the element in the dataset. So “5” on the dice would be the 5-th element, or “4”.

After rolling the 1d6 6 times, we get

Data Re-rolled Data
1 3
1 3
3 1
4 4
4 4
5 1

This re-roll gives us a median of 3, which is 0.5 less than the previous median.

We re-roll the 1d6 a further ~6,000 times to get ~1,000 re-rolls. And perhaps after this we expect an expected range for the median to be 3-4, which our initial 3.5 fits cleanly in.

Further explanation of rolling can be found here.


Similar to the example, we can also do this with adding percentage weights to the scores (that way I’m not generating millions of pieces of data, ffs even I am bored of that). Since we want to find the winning outcome, we can take the mode (most common number) of each of the 702 re-randomisations.

I then plotted them in Pie Charts because it’s an easy way to represent comparisons like this and also because I’m currently baking an apple and cinnamon pie for my gf and that’s all I’m smelling atm :yum:

Based on this data, we can see that by far and away there are two options significantly more likely to win, almost being a coinflip between the two. Even though the other options had decent standings in the polls, they weren’t statistically significant enough.

In fact, we can see for Base Type outcomes that Staff actually beat out Spear in this re-randomisation process, and the other runner-up options came very close each time too.

So we really only have two statistically significant outcomes for each poll:

Category Base Type Theme Effect
Two-Handed Spear RAT CITY Volcanic Orb
One-Handed Staff Tri-Elemental Meteor

Given how close together these results are, having a secondary poll to finalise these would be a good option (like maybe on the weekend after running the other poll for 5 days, or added onto the following week’s poll).

@woozel made a very good point in that those who selected other options may have had the winning option as their least favourite, and thus a more accurate representation of the community’s desire may have been the runner-up option.

I doubt there’d be any changes to this poll, but it would be nice if EHG took Woozel’s advice into thought for future community events.

Once again, great post :3

4 Likes

If I take a look at the steam charts calling this poll a community designed item is a tad bit funny. Even if I count every person individually without counting persons that made a poll on every category we only have a fraction of people voting here.

On top of it it seems you change your vote at any given time if you want to.

It’s a bit sad and I’m not realy a fan of the whole setup but it is what it is even when it leaves a bad taste in my mouth but a new unique is a new unique.

Great post, I really enjoyed reading it!
Thanks!
:slightly_smiling_face:

I don’t have the mathematical / statistical talent of the first two posters. But just a few layman thoughts after reading this:

  • No offense, but I find your post focuses a bit too much on the negative side. You have so much quality insight, I would love to read your suggestions for better systems. Yes, “First Past The Post” is usually bad (even if, sadly, some major so-called democracies are still using it). But what else could we do? Ranked choices, Scottish-style? A final round of vote between just the two leading options, French-style? Something else? How representative would these systems be when passed through a strong statistical filter like you did for FPTP? I don’t know, that would be interesting to study.

  • In these polls, we are able to check the ongoing results and change our vote at any time. This alone entirely breaks any system you can think of. Let’s say the loosing option of a poll ends up at 7%: this might have been 14%, half of whom decided to change when they saw they wouldn’t win. Or more, or less, we just don’t know.

  • You already kinda made this point, but I would like to emphasize: thanks to EHG for making these polls and involving the community. I personally don’t give a hoot about the unique (I didn’t even vote last week), it will soon be drowned in a sea of nearly 300 other uniques and new ones coming out all the time. But I love that it made the forums alive and got a lot of people posting for the first time. So when you say “there is little value in community input collected using this method”, I agree, but I would answer, “the input doesn’t matter in this case, it is about the involvement”.

Thanks again to Woozel and Queermathsgirl for some fascinating food-for-thoughts (namely, apple-and-cinnamon-pie-for-thoughts).

2 Likes

I think @queermathsgirl has found a kindred spirit in @woozel :wink:

A lot of info here that is way over my head and I would have to research voting systems to even begin to provide some productive comment to the topic… One thing that is definitely a problem is the ability to change votes - that isnt something I believe should happen, but then I dont think the forum voting system was designed for this kind of thing specifically.

What I did want to mention is that in instances like this “voting” for the creation of something like this is sometimes not the best way to design a product/feature… In my experience as a product developer, when you drill down to this level of designe detail and allow it to be dictated to by the whims of random people voting, the end result can and usually is, pretty crappy… or at least ends up not meeting the needs intended… Not always, but usually. There are so many things that influence individual choices that it becomes hard to steer a design in the best direction… Everything from someones general experience in playing LE and understanding the mechanics to someone just liking the colour purple can influence the voting in ways that are not intended…

The devs even admitted that just doing the creator uniques takes many weeks of work back and forth because of the complexity & testing involved… and I dont think that a voting system is likely to be a better solution.

From a marketing and community building perspective, allowing a community designed unique & letting people vote for it is great but even with the devs cherry picking ideas to try and control the options to vote for, this process isnt, imho, the best way to do it…

What would be a better way? Hard to say… But I would rather the experienced testers & devs highlight areas of the game where there are no matching uniques and then propose potential unique ideas for the community to prioritise… i.e. get the community to confirm/chose but not get everyone involved in the detail design…

Sure people will probably not like that but I have been burned too much by low level product design overtly influenced by consumers/clients to agree…

1 Like

I see nothing wrong with how the voting was done nor the ability to change your vote. I think it’s perfectly reasonable to change your mind and should be allowed up to a point when the polling is done. I also don’t care how low the percentage of votes the winner had, if it had the most it won. That’s just how it works. No long post talking about how it’s not a majority matters, it doesn’t have to be a majority. It’s a simple poll for a simple design. I think this is all overthinking it.

Thank you for the great feedback! We’re certainly looking at these very things behind the scenes ourselves and reviewing them, but having this kind of feedback from the community as well is fantastic! While others may not agree, I personally see this feedback as a win from the event perspective since it shows that at least part of the community (yourselves) did enjoy the core idea, and had investment into it to the extent of making this kind of feedback.

This community designed unique event has been a bit of an experiment in this kind of thing. We’re not aware of an event like this having happened before, particularly with this much creative control given in the event. Overall, we’re very happy with the level of community involvement, and the results of it, but like any time you do something for the first time, there’s certainly aspects that could be improved upon. Once the event concludes, I might write a post discussing this, however we’re certainly analyzing the same kinds of things on our end as well. There is other considerations we have in this kind of event beyond just the polls themselves, however one of the core intents of this event was to provide the community the opportunity to have creative freedom in creating their very own unique outside of our influence (as much as reasonably possible).

When we reviewed other polling options, we ran into few other considerations:
1.) Interest exhaustion. To provide the amount of opportunity to ensure full community backing to have majority winning on each option, we would need a much larger set of polls. A longer running event would more easily lose interest from those involved and while it may remain majority of those that vote, there would be fewer members voting the longer the event runs.

2.) To avoid interest exhaustion we condense the number of posts within the event. If taking the route of being more thorough with each aspect, it would need to mean fewer aspects would be up to the communities decision. And that’s certainly a perfectly viable route, however with this event we wanted to experiment with giving the community pretty much complete control.

3.) While I absolutely love what I do: my time, and that of other social team members also isn’t ‘free’ unfortunately, and we need to organize ideas, write posts, create the polls, and post across various platforms with each one. It would also mean more time from the social team spent away from other things, such as answering questions, potentially other events, and many other behind the scenes planning and structure we do. This of course isn’t the biggest consideration in this kind of event as these kinds of events, particularly as it is part of what we do as a social team (it’s not taking us away from “our work”), but as a company cost is still something we need to consider.

In summary, we’ve been very happy with this event so far, and are really excited to the outcome. We agree that there’s improvements we can make to the process, and this kind of feedback is greatly appreciated. Once we’ve concluded the event, we’ll be doing a full analysis of how the event went, and what we could improve on. This feedback on the specific polling process certainly helps us in that aspect of the analysis :slight_smile: It’s certainly possible we could do something like this again in the future, but with improvements from lessons learned during this event.

7 Likes

Ah, yes it is, but you are missing the point: the problem in this poll was being able to change your vote AFTER SEEING THE PARTIAL RESULTS.
I don’t think people were changing their mind, I think they were strategically moving to one of the two options in front. Meaning whatever was in front after the first few hours became impossible to catch up with.

Absolutely!
That’s all the fun of it!!!
:grin:

2 Likes

Sure… you can change your mind - but you shouldnt be able to change your mind after SEEING the existing vote totals… That allows for strategic voting and messes things up - people want to be part of the “winning” selection so they change their vote just to feel good - if they want to change their vote based on a valid argument that changes their mind, then thats fine… but they should not be able to see the results first… that is a recipe for disaster and breaks valid voting…

Anyway… My specific comment was more against the use of majority / democratic voting process wrt general product design because of the varying perspectives/experience/understanding of random users. While it has its place in helping broadly direct what people want, I dont believe it works well in ALL scenarios and I dont think that its appropriate for the nitty gritty of unique item design… but thats my opinion…

1 Like

As part of your debrief once this is all over… I would be very interested in the viability of the end result of all this effort from yourselves and the community - in this case the Unique itself. I.e. is the Unique actually going to be a good one?

Sure, the metrics to determine if its successful or not will be hard to define, but simply put… is it going to be a Bastion of Honour or Wings of Argentus … or is it going to be a Fighting Chance or Falcon?

1 Like

That would make every poll an absurdity. Once you voted you voted. Sure if you missclicked there should be an option to revisit the poll and change yyour vote to the desired option.

I voted for a twohanded axe and never changed it because I voted for it. The possibility to change your mind however you want to is turning a vote upseide down. That’s just my point of view and I’m a “stick to rules” kind of guy who didn’t cross a street if the light is red no matter if there are cars or not ^^. I even feel bad because this poll would’ve been a big enjoyment for Trump because everything was rigged and fraudulent :D.

The outcome is what it is but I realy hope the poll module of the forum will be worked at for future polls of this extend because it is something important from my point of view and should be done as “perfect” as possible.

On the other hand, being allowed to change your vote as results are accumulating visibly can effectively act like multiple rounds of voting. People keeping watch might realize they voted for a less popular option and recast to participate in the higher rankings.

On the OTHER other hand… If too many people are watching in the early days of voting, an option that might have been popular could lose out by the sheer luck of who was keeping track at the time.

Also, for the record, my analysis did not include the ability to change your vote, as I clearly already had enough to post about without it. :laughing: That said, such an option is quite a big variable that merits consideration.

And I think it fair to say this has only added to the excitement for the game and its future. Obviously, we’re still in beta and can only expect so much participation on the forums overall, let alone a series of polls. An average 1000ish voters per poll so far seems a good start.

I was a little bit hoping to bring out an actual professional. Mmm…pie charts.

2 Likes

If it matters at all. I would like to throw my voice behind the Ranked Choice system.

For those who don’t know what that is. In a poll with 5 choices, for example, you would rank them 1 to 5. 1 being your most desired and 5 being your least. The polls would count the #1’s and drop the last place choice. Anyone who had the last place choice as their #1 would then have their number 2’s counted. This continues until a choice has over 50% of the votes.

It isn’t a perfect system either, but it seems much more representative of what people want than first past the post. How many of that 76% of people who didn’t vote Mage had mage as the choice they least wanted?

1 Like

I was thinking exactly that at first.
But I came to the conclusion that it doesn’t really work, because of two factors:

First, being able to change your vote is extremely unusual in a poll. Therefore, a lot of people wouldn’t realise they can do that. I myself only found it out by reading the comments, someone just voting and not reading would never know.
I think it would have to be widely advertised to be representative.

Second, it brings in a huge “time” importance. To win, you don’t just need the votes, you need to get them very early (so your option looks like it is in the lead when people start re-voting).

Conclusion:
Awareness of the system + Speed of answer: this system probably gives a lot more weight to the regular users of the forums, who would spot the thread as soon as it comes out (= speed) and later read all the comments (= awareness), compared to the casual players just popping in to vote later in the week, without reading everything.

Whether it is a good or a bad thing is open to discussion (one could easily argue that more involved players also have deeper / better quality input).

Yeah I get your points :). it’s just my viewpoint that if you voted you voted. It’s just my random oppinion and I didn’t want to criticise your post :). After all everyone is happy there is a new unique but the way we got there wasn’t my cup of tea and that’s all i wanted to tell :).

1 Like

Well first, that’s a lot of assumptions you can’t validate. Also, there’s no good reason why I shouldn’t be able to see current results before casting my vote. You are not providing any ethical or logical reason why it should be done any other way. It still sounds like people are upset their vote didn’t win. My vote didn’t win and I couldn’t care less, it is what it is and you can just move on.

Neither did mine and I honestly dont care very much as one unique isnt going to make that much difference in the in the greater scheme of the game.

Challenge accepted…

Scenario… A vote is almost at the deadline… There are two selections tied for the win… Option A - 10 votes, Option B - 10 Votes… Someone favouring option A sees the result and offers to buy a car for the first person who will change their vote to Option A instead of their already chosen Option B… Someone who has already voted takes up the offer, thereby forcing a win for Option A at the last moment. The person doesnt convince people to change their vote based on the merit, they basically coerce a change / buy a vote to push the result in their favour… Result - Voting system rigged, result invalid (from the perspective of the actual topic voted on)…

Removing the option to SEE the result in a voting environment where you can CHANGE your vote doesnt stop the buying of votes, but it limits last minute CHANGING of votes because no-one actually knows who is going to win until the vote period is officially closed.

In a voting environment where you CANNOT change your vote, then SEEING the result isnt as much of a problem - sure people can STILL BUY votes, but only those that HAVE NOT VOTED YET… which limits the ability to mess with results at the last minute…

Imagine the chaos if voting for political parties, government allowed you to chop and change your voting during the vote…

Obviously in this particular instance - creating a community unique in a game - its not really that important and the devs are unlikely to let a crazy stupid voted design through the system as they can veto it, but the concept still applies imho.

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.