Implementing Ranking system on GOS

[Professional Killer]

I had a quick chat with PunkUser just now talking about getting some sort of ranking system here on GOS.

I would like to see that back in regards to watch stats and get rank icons etc.
As we all know ranking more or less doesnt mean much these days to playesr, but I feel that it is a part of Myth when it comes to the competetive part of the game.
Maby its most for old memories, but I said I would like it back and he asked me to put up a post to hear what you guys think aswell.

He was interested in what the community think about this and some agreeing on what ppl want regarding it.

[adrenaline]

I'd be cool with it... seeing that icon next to my name is in some way nostalgic.

[punkUser]

Yeah I think everyone agrees on wanting some sort of ranking system, the question really is just how to make something that is at all meaningful and not completely exploitable. Note that icons beside your name is largely orthogonal to the ranking system (i.e. we could have icons without a ranking system too...) but I'm assuming people actually want them to be related to some sort of underlying ladder.

The most reasonable systems are Elo and TruSkill. Elo is used by most game ranking systems (in addition to chess, and a variety of other leagues) that I know of, and TruSkill is used for a lot of Xbox leaderboard stuff. MariusNet's points system was roughly based on Elo as well, but apparently a lot of people really disliked it, which puts a bit of a damper on the idea... Honestly I would be surprised if Bungie's system wasn't also Elo-based, but we don't have the source code for that so it's impossible to know for sure.

There are three big issues in Myth as well: two few players, no (mandatory) matchmaking and ubiquitous dummy accounts. All of these contribute to making ranking systems unreliable and not converge nicely. Ranking systems that are easy to exploit tend to just get ignored and I definitely don't want to spend a ton of time doing something that no one will end up caring about.

So let's discuss possibilities here, but I'll lead off by saying I'm pretty wary of people proposing their own custom ranking systems that they invented on graph paper last night. Almost everyone one of those I've ever seen outside of math papers tend to be even more exploitable and terrible than just raw Elo and TruSkill. So we can have the discussion and see if people agree on stuff, but ultimately if you roll your own not based on established systems there's a large chance that it'll have major issues.

Another thing to throw out up front... the only valid results to use from a game are win/loss/placement. Introducing secondary stats (damage, ratios, etc) as actually affecting ranking just adds useless noise to the system and encourages people to optimize for something other than winning, which is really just a silly metagame. This is why none of the serious ranking systems in games use anything other than the match result in the ranking system.

Don't mean to sour the waters too much, but as a math major and someone who has read a lot of these papers in the past, I'm well aware of the strengths and weaknesses of these systems. My hesitation to implement something early on has been largely due to the negative reaction I've heard from folks about MariusNet's system. If people want something like Elo/TS/Mnet ranking, that's definitely doable.

One other related topic: some people have mentioned that one of the main complaints about Mnet was plugins... people who played WW2 a lot, etc. would tend to get higher rankings since they ended up playing lots of quick games, etc. A discussion around how people want to handle plugins would be useful here too... it's probably possible to basically maintain a white-list of allowed plugins for a game to count as ranked if desirable, but not sure what people prefer.

[adrenaline]

First off, the thing that was completely fucked about mnets ranking system is that the less points you have, the more points you gain for a win... which is why we constantly saw people with ~20 games with comet or celest ranks, while better players with hundreds of games struggled for that many points. That should not be allowed to happen. It just encourages the idiots to create a new account, newb bash 20 games, and park that account at comet.

The amount of points you gain for a win should somehow be related to how many players were in the game, i.e. more pts for winning a 6-person FFA than a 1v1.

Win:loss ratio should count for something. I actually liked how mnet tracked 1st place, 2nd place, 3rd place, low, too. Tells a lot about a players FFA prowess.

An interesting way to do it would be 2 different systems for FFA and 2-team play, where FFA points are acquired something like this: 3 for win, 2 for 2nd, 1 for 3rd, -1 for 4th, -2 for 5th, -3 for 6th.
For 2 team, something like # of players/4... i.e. a win in an 8v8=4 pts, 6v6=3 pts, 4v4=2pts, 2v2=1 pt, 1v1=0.5 pt... or even something simpler, like 2 pts for win, -2 pts for loss, though I think 1v1 should be less rewarded to discourage dummy bashing.

Perhaps implementing a system where you have to register (independent of initial account registration) to partake in the ranking system... and perhaps somehow disallow multiple accts from a single IP address or some other way of discouraging dummy accounts... I dunno... just spitballin' here.

[punkUser]

First off, the thing that was completely fucked about mnets ranking system is that the less points you have, the more points you gain for a win... which is why we constantly saw people with ~20 games with comet or celest ranks, while better players with hundreds of games struggled for that many points. That should not be allowed to happen. It just encourages the idiots to create a new account, newb bash 20 games, and park that account at comet.

Without going too much into the theory of ranking systems though, the modification of rank after playing has to be based on the delta of ranks between the players. Otherwise it's just a glorified win count... which is fine, if that's what people want. But then the ranks don't reflect who is the "best", just who plays the most.

Note that TruSkill is slightly better than Elo in its handling of "unknown"/new accounts and players, but ultimately ranking system must take their predictions into account when adjusting rankings after games.

The amount of points you gain for a win should somehow be related to how many players were in the game, i.e. more pts for winning a 6-person FFA than a 1v1.

Well it shouldn't just be points for a "win". Try not to think of it in terms of "points" per se. Really what you want to do is given the current rankings, the system can "predict" what it thinks a game result is. Based on how different the real result of that game is, it adjusts its rankings/predictions for next time. And yes, the entirely placings can be taken into account there.

Win:loss ratio should count for something.

It shouldn't be used directly, no, because that just encourages people to play weaker players and avoid games against stronger ones. Games where a good player plays against a bad one and wins provide almost zero useful ranking information to the system. As with MFC, the games you need are the ones that are expected to be close. (Which is incidentally why mandatory matchmaking helps enormously as well.)

Perhaps implementing a system where you have to register (independent of initial account registration) to partake in the ranking system... and perhaps somehow disallow multiple accts from a single IP address or some other way of discouraging dummy accounts...

There's no reasonable way to prevent multiple accounts these days, period. The best we can do is have some sort of time/games requirement in unranked before you can play ranked, but that's exploitable by other dummies too... it just wastes a bit more of people's time.

Note however that Milk Man's FFAclub league starting in September will have scoring similar to what you describe. Since it has somewhat more "managed" games, it can pull that off a bit better.

[adrenaline]

valid points

[pallidice]

im guessing you guys have no access to the way bungie.net did it back in the day?

[punkUser]

im guessing you guys have no access to the way bungie.net did it back in the day?

Unfortunately no, that was not included in the source code release they made. But as I mentioned, I believe it was also mostly Elo-based. The main difference is just that there were a lot more players and it was more difficult to have dummy accounts, which as I noted does help a lot regardless of the ranking system.

I'm considering just implementing something like TruSkill. It's probably the best we can do for now and although I'm sure some people (or even most people) will have issues with it, at least it's one that I might actually enjoy implementing Plus it can deal with dummies fairly well compared to other systems. Note that we'll have to come up with a better approximation of "team skill" than just summing all the individual player skills, but clearly there should also be some penalty for having more players on a team (i.e. just using the average or something).

But let's keep the discussion going in case people have other ideas.

[zagon]

I agree with alot Adren said. Having points for different places and negative points for certain places was always nice. If you do poorly you should be penalized. If you do better you should be rewarded. I also liked bungies method of more points for beating higher ranked players. I remember some games going up 30-40 pts for beating celests.

Maybe we can message up bungies forums and see if anyone would be willing to share some tips/ enlighten us on how they did things. Im sure someone from those days is still around

The one thing I did like about mariusnet was not needing 4 people for the games to count. Sometimes its hard to get a game, I know I log on sometimes at 9-10 in my timezone and there is 12 people on and maybe 1-2 not afk.

[dac]

I'm pretty wary of people proposing their own custom ranking systems that they invented on graph paper last night

First off, the thing that was completely fucked about mnets ranking system is that the less points you have, the more points you gain for a win... which is why we constantly saw people with ~20 games with comet or celest ranks, while better players with hundreds of games struggled for that many points. That should not be allowed to happen. It just encourages the idiots to create a new account, newb bash 20 games, and park that account at comet.

The amount of points you gain for a win should somehow be related to how many players were in the game, i.e. more pts for winning a 6-person FFA than a 1v1.

Win:loss ratio should count for something. I actually liked how mnet tracked 1st place, 2nd place, 3rd place, low, too. Tells a lot about a players FFA prowess.

An interesting way to do it would be 2 different systems for FFA and 2-team play, where FFA points are acquired something like this: 3 for win, 2 for 2nd, 1 for 3rd, -1 for 4th, -2 for 5th, -3 for 6th.
For 2 team, something like # of players/4... i.e. a win in an 8v8=4 pts, 6v6=3 pts, 4v4=2pts, 2v2=1 pt, 1v1=0.5 pt... or even something simpler, like 2 pts for win, -2 pts for loss, though I think 1v1 should be less rewarded to discourage dummy bashing.

Perhaps implementing a system where you have to register (independent of initial account registration) to partake in the ranking system... and perhaps somehow disallow multiple accts from a single IP address or some other way of discouraging dummy accounts... I dunno... just spitballin' here.

bro...

in addition to adrens spitballing, you should separate into the following categories:

1x1
2t
ffa
third party (where plugin includes tagset/custom units - sorry lichen!)
POSSIBLY vtfl (will probably never have enough games to count)


give me 4 icons! i look forward to wielding 4 sword and daggers.

[punkUser]

I also liked bungies method of more points for beating higher ranked players.

Right, that's not "Bungie's method" though - that's all ranking systems.

Maybe we can message up bungies forums and see if anyone would be willing to share some tips/ enlighten us on how they did things. Im sure someone from those days is still around

You can ask, but honestly I think it was just pretty basic Elo. Like I said, it's secondary factors that are making people remember it as some great system. MariusNet's is probably objectively better TBH...

The one thing I did like about mariusnet was not needing 4 people for the games to count.

Right that was actually a restriction in the Myth client itself. I removed it in the latest hotfix so we're good to go on that.

[dac]

mnet would count against you if you teamed with people and gave way more points for ffa. thats why the higher ranks are the ones that do well in ffa, and people who do 2t exclusively woudlnt really do well.

[punkUser]

mnet would count against you if you teamed with people and gave way more points for ffa.

Well beating unbalanced teams has to not count for as much (i.e. 8v3, yay!), so averaging ratings is not good, but as I mentioned, adding ratings to get a team rating is clearly not reasonable in Myth either, so something in between.

Categories of different ladders is getting a bit much for the current community size I think. Plus there's no way to differentiate icons and stats for that ingame anyways. To some extent tournaments really are always going to be more relevant.

[Giant Killer General]

not too picky about the system ultimately chosen, but like you said you need ways to thwart the exploiting.

If you are going to use a point system, look at bungie's model as an example:

Bungie had a max point win or loss of 32 points per game. 0 was the minimum, but there should be at least 1 point minimum. Basically what I am saying is put reasonable boundaries on everything to help fight exploiting, such as a min/max on points lost/won per game. I don't think removing exploiting entirely is possible, but some basic mechanics combined with some human moderation as a backup would go a long way.

Regardless of what kind of system you use:

Put a minimum number of games played on each rank so you have to meet a certain minimum activity threshold just to climb the ladder, again so that if someone does dummy they are going to spend a lot of time just getting the games in on that dummy, or group of dummies that they may want to try and use to propel their own rank. So it should discourage dummying. This also encourages people to play more which is ultimately what we all want. Also don't only count the most recent 90 days of games like mnet had it, have it reset every 6 months or so like bungie had it (coinciding with MWC and TWS). People were always super excited about the resets and it would spark activity again.

You should get points for 1v1. FFA should definitely be in a separate category, 1v1 possibly too. Non-reg-myth definitely also needs to be a separate category (or disallowed completely). Don't have it based on gametype like bungie had it.

Also, ranked or not, I still think FFA would be vastly improved with an option for BC-weighting in there to remove the politics from it. That is probably a side-topic though.

[tirri]

i had completely forgotten about the resets. resets were freaking awesome as people really cared more for a while after them

[spongefile]

TrueSkill looks good! Matchmaking would be an interesting feature too. MythCupid!

[pallidice]

ok didnt read much of this thread but some sujestions are. Yes use the rest system or make it so you go down a certain number of points a week. so that if u are active ur gaining points most of the time but if u arnt u losing points per week or day or some such so a dummy account cant just sit it out

[punkUser]

Good feedback, especially that people like resets vs. rank decay. That's definitely doable.

You should get points for 1v1. FFA should definitely be in a separate category, 1v1 possibly too. Non-reg-myth definitely also needs to be a separate category (or disallowed completely). Don't have it based on gametype like bungie had it.

Unfortunately this is not something that can be changed - it's built in to Myth. Sadly, a lot of the ranking stuff is sort of mixed between client and server, so there's only a certain amount of flexibility we have on the metaserver side.

Given that, the degrees of freedom as I understand them from browsing the code basically are:
1) An arbitrary ranking system can be used to update "points" based on game results.
2) A fairly arbitrary mapping from "points" or whatever to icon. Icons can be hidden or shown in different rooms, but ultimately you have one icon/rank.
3) We can choose to exclude games that meet certain criteria from ranking entirely.
4) Can show overall stats "Total" and for each game mode.

i.e. the "Stats" screen is not something we can change the layout/information displayed in - just fill it in with numbers we derive. So there's no easy way to do something like have separate ranking for 1v1, FFA, 2-team or similar, at least not within the game. It's possible we could have such a ranking entirely outside the game on the web site (like Mnet's coop ranking for instance), but I'm not sure that's what people really want.

Alternatively we can completely omit certain results... it could only consider 2-team games for instance with a certain minimum number of players (like MWS). But I imagine since everyone likes different things that would not have as wide appeal. Thus we may have to compromise on some sort of unified system with all the different types of games included. We can definitely do stuff like weight games with more players higher or something, but it might be hard to get agreement on those sorts of policies. Also the more complex the system is with special cases and such, the less intuitive it is which frustrates players.

[dac]

4 different ranks: 1x1, ffa, 2t, thirdparty. average of the 4 gives you the total rank.

mnet/bungie had it split by gametype (Fr, lmoth, etc) and you could instead split it into those 4 categories and rank people individually in that, then combined score gives the icon. that would also compel both people who care about their rank icon to broaden their horizons a little.

[punkUser]

mnet/bungie had it split by gametype (Fr, lmoth, etc)

That's what I'm saying - that's built in to myth. Any other split would be website-only and would not be reflected in the ingame stats.

If people don't care about the ingame stats panel then there's more flexibility of course. I had the impression that they do though as I still get a lot of requests for "stats" even though they are already available - for instance - at http://games.gateofstorms.net.

Note as well that there's no really good way to differentiate "regular Myth" from other plugins if you want to include stuff like MWC map plugins. The metaserver has no idea what's in a plugin, so while I can almost certainly tell "uses plugins" from "doesn't use any", eliminating all of those map plugins may not be entirely desirable either. There's potential for using some sort of white-list, but I need to investigate exactly what plugin info is in the game structure that the metaserver gets.

[dac]

dont the tagsets have special c hars like tilde and delta? also more than one plugin might be an obvious thing too.

[zagon]

3) We can choose to exclude games that meet certain criteria from ranking entirely.


What do you mean by that? What games would be excluded?

[HeadHunterKC]

My question is along the line of zagons' above.

Could we as a community agree on maps that get voted into the rank system? For instance, many of the MWC maps would be worthy candidates. Perhaps with every reset (I love the idea of resets, superior to the mnet way) there could be a new plugin called Myth Ranked or something that would include all of the maps that are allowed that reset. This also gives new maps a gestation period to get community to support to be added to the list.

Rank, believe it or not is a desirable aspect of Myth online. I know people who would return if they had a chance to become a celest again on a server that had a capable ranking system.

Hell, I would play more.

Can you make it so the same IP can't share a game? Or is that how it works already? I'm thinking of dummies (won't say anybody's name!) who have multiple accounts in a single game.

[vigor]

I don't care about icons or stats in the lobby.

My suggestion is to add some lists over here --> http://games.gateofstorms.net/users

Could be most wins, best win ratio, placements overall, kills, damage, and any of those combined with game type or map.

So if I want to see who had the most damage on koth games on gimble, I could just check some boxes and filter that out.

[vigor]

Based on the information you could get from that, let the forum be the arena for deciding who is the best player. Then we won't have the "unfair ranking system" complaints any more.

[pallidice]

i dont know how the ip thing works... i know i cant join my own host but i think u can have 2 accounts in the same game

[punkUser]

dont the tagsets have special c hars like tilde and delta? also more than one plugin might be an obvious thing too.

That's not required, and unfortunately unit changes can be done in any plugin outside of tagsets too. I mean, look at the Zak FFA pack with all of its different units. It's not really possible to determine if something should qualify as "regular Myth" without someone looking at the units, trade, etc. Seems like a white-list is the most reasonable if people care a lot about this, as otherwise anyone could make stupid plugins to exploit the system.

The only negative bit about a white-list is that there's no really good feedback about if a game is going to end up not counting due to settings... seems undesirable to just find out later on the web site.

Can you make it so the same IP can't share a game?

Like I said, anything IP-based is just not going to work. It is trivial bypassed for anyone that cares to do it. Slightly better is do something mobile-phone based (i.e. require a unique phone #, send a text message to activate ranked on that account), but that both excludes people without mobile phones and there's no way to prevent people from using their friends' phones to get dummy accounts either.

There's really just no good solution here. You just have to accept that there will be dummies and minimize the impact in the ranking system.

My suggestion is to add some lists over here --> http://games.gateofstorms.net/users

Could be most wins, best win ratio, placements overall, kills, damage, and any of those combined with game type or map.

So you can already sort by a lot of those things, but yes the plan is to add filters and such there as well, similar to the game list site.