‘Adding to the conversation’ of ESL’s new Dota 2 rating system.

A snippet of some per-rating comparisons from a 2017+ dataset I prepared for a colleague’s PyData presentation (correct / incorrect | brier score)
I cannot think of a worse input for a rating system than social media
via https://liquipedia.net/counterstrike/ESL/One/2019/New_York
  1. Quantifying events by size (the huge/large/medium/tiny criteria). No matter how many teams go to the LAN finals, this can be irrelevant if the tournament format is poor — winning a 32 team single elimination bracket shouldn’t mean more than winning an 18 team round-robin groups into 16 team double elimination event (like TI is).
  2. Quantifying events by skill. Using the skill of a team within the model to classify future events within the model is introducing an unnecessary feedback loop. This means a relatively small error can be compounded over time.
  3. Evaluating tournament performance by just final placement. Imagine you had a 256-team single elimination bracket where one team defeated 5 of the top teams in the world before being eliminated and another team defeated 5 random pub stacks. Despite finishing in the same position, would you say their performance is the same? If you had to start a tournament the next day, would you realistically offer the same odds on both these teams since they placed the same? Obviously not — individual matches describe the actual experiences and performances of individual teams. If there was an accurate “margin-of-victory” metric, that would also be a great addition.
  4. TI being weighted 50% more. After TI we have a huge break, a moderate patch, and the biggest shuffle of the year — creating the largest amount of uncertainty within any system. TI being weighted heavily just serves one purpose: the rankings look “right” after TI for 5–6 weeks just because there’s no other information. Given how the decay works, this indirectly means that TI results against completely different teams are used as a justification for up-seeding teams which did well at TI (generally big names) over potentially up-and-coming newer teams with much better recent performances (within the current set of teams, not a previous one).

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Ben Steenhuisen

Ben Steenhuisen

Dota 2 statsman and occasional caster | runs @datdota