Player Behavior Knowledge Base > Instant feedback system

Instant feedback system



1. About

The Instant Feedback System, as the name implies, is the system that analyzes player reports and chat logs and issues punishments when due; all of this happens within 15 minutes from the report that triggers the review [417, 574].

The punishments being issued vary from Chat Restrictions, to bans according to the afflicted account’s behaviour history.

 

2. Implementation

The system initiates analysis on a reported player, and checks the in-game data (such as statistics, chats, item builds) [227]. The reports, however, are not used indiscriminately, but are first filtered to ignore the false (i.e. not justified) reports, which lead the reporter to lose some of their “report value” [179, 227]. Correct reports that do not lead to punishments are “stored” and can be used for the next analysis on the same player account [137, 227].

Behavioural standards that influence report validity are not dictated by Riot, but by the players themselves: this is accomplished through data gathering from old Tribunal, reports, honours and social media [80, 389]. For example, “gg ez” has become a reportable option fairly early in the system’s lifespan specifically thanks to this unique method [80, 144]. These data points also help Riot refine the Instant Feedback System, in order to catch and punish players that otherwise would stay unpunished for too long [117]. Speed of the response is the key here: feedback being delivered practically instantly allows players to reform at a much higher rate, than punishing them months after the facts [454,87].

This method also has the added value of updating the system very quickly whenever behavioural trends shift [389]; however, while the system adapts itself using reports, any newly found (and reported) bad behaviour gets manually checked, to ensure that it has been correctly perceived as toxic [495].

 

Machine learning is used to feed the Instant Feedback System with data on the context of phrases [149, 175], but even while taking context into account, players in a premade aren’t safe from punishments, if they use offensive language in a setting where other players aren’t aware that they’re talking to friends [171, 580]. Full premades don’t get automatically checked [546].

It’s a fairly common misconception that the chat log analysis only involves the in-game chat, but pre- and post-game chats are also recorded, but rarely used to review a case [540].

 

Usually, offenses are punished with an escalating system that starts from Chat Restrictions and goes up to 14 days and permanent bans [344], with the added ability to override punishments with heavier penalties if necessary [591]; homophobia, sexism and racism are considered particularly severe offenses [203], despite being rare (only 2% of matches globally) [221], and can get detected and punished with a 14 days or permanent ban [32], even without prior warning or punishment [254, 255, 344]. The reason being offered for such severity is that such remarks are “overwhelmingly rejected by the community, and hard to mistake” [224].

While it’s true that the system punishes toxicity, it actually doesn’t punish profanity -- not even vulgar words that belong to the in-game culture [305, 548] -- but it concentrates on both common and culture-related malicious or insulting phrases [259, 305, 353]. According to Riot, thanks to it there was a drastic reduction of racist, homophobic and sexist remarks, accompanied by a rise of common profanity, such as “fuck” or “shit” [304].

The reason for choosing only four steps of punishments was stated to be reform rates, which plummet after simple warnings [351]. Each punishment is one chance less for the affected player to reform, because permanent bans due to behaviour are never lifted, unless an error occurred [557].

 

A pop up informing a player of a punishment being dealt to a reported offender might appear. It was initially set to be seen only by the player whose report triggered the punishment [46, 68, 127], but it has been changed since then, and doesn’t even appear every time action has been taken [618]. Riot stated that tuning it to pop more often would have been seen akin to spamming, since about 20% of the community wasn’t interested in those pop ups [279], and the would have been worse after the inclusion of Chat and Ranked Restrictions [127], but this last part of the plan was never implemented.

 

The system is tuned fairly conservatively to avoid false positives as much as possible [65, 126]. It doesn’t scan private conversations by design; the reason given being that context there is much more difficult to discern [65].

 

3. Successes

The system was intended to have a 0,1% error rate since launch [260, 472], to minimize false positives, and accuracy is ensured through case sampling, performed by Player Support employees [175, 472, 602]. One of these quality checks in the very beginning of this system showed that 1 out of 6000 punishments is undeserved [250].

 

Since the introduction of the Instant Feedback System, verbal abuse has dropped by more than 40%, and 91,6% of negative players change their attitude after the first punishment, thanks to fast feedback and Reform Cards, therefore avoiding subsequent penalties [20, 67, 589, 601]. According to Riot, warnings are enough to convince 75% of the players to improve their behaviour, thus avoiding the penalty system altogether [351]. Ranked games experienced a 40% drop in toxicity after the launch of the system [67].

This is important because the reform rate is very high, compared to traditional systems of manual reviews (which scored a 50% reform rate). Adding evidence, such as chat logs, to the punishment increases the reform rate to 75% [376], thanks to players getting a clear picture of what got them punished.

 

Another accomplishment is that it managed to basically equalize player behaviour standards across all servers, even those that never ran a Tribunal to begin with. European servers are considered to be more difficult to keep in check, due to the huge variety of languages spoken across the region [125, 475].

 

An interesting phenomenon about players not reporting behaviour that the Instant Feedback System believes to be toxic, has put onto the table the problem of understanding how the community thinks on certain issues [330].

 

4. Miscellaneous

  • While the current Instant Feedback System works exclusively as a punishment system, at some point there were plans to include positive reinforcement, in the form of making the system learn and identify positive behaviour through reports and honours [187, 207, 208, 229] and send prizes to players [122, 208, 209]. Furthermore, with the new Tribunal voting system players would have been able to review both positive and negative behaviour [208, 209], and even receive a monthly “report card” showing the voters’ contributions for the betterment of the community [278]. It is currently unknown whether these plans are still in development, or have been iceboxed.
  • Another, less known, function of the system is that the data it collects is used by player support specialists, scientists, designers to review dubious cases, or analyze behavioural trends that can help designing new social systems [227].
  • Ranked Restrictions never impacted the overall toxicity in Normal Draft Pick [594].
  • One way to avoid restrictions would be not using the chat altogether, since smart pings are enough to communicate most information in game when used intelligently, and most players don’t use the chat anyway [182, 336, 584].
  • PBE used to be devoid of behaviour systems, and players exploited this loophole by performing all kinds of bad behaviours that are normally punished on the regular servers, which would in turn show how servers would look like if they didn’t have any punishment system implemented [460].
  • Asking other people to report someone is useless, since just one report already triggers an investigation of that players behavior and more reports don't change anything [650]

 

5. Sources

20 http://www.gamesindustry.biz/articles/2015-07-08-homophobia-sexism-racism-reduced-to-2-percent-of-league-of-legends-matches
32 http://hwcdn.libsyn.com/p/d/e/d/dedcb49677a8dcd1/PoGCast_Episode_4_-_Toxic_Behavior.mp3?c_id=9204818&expiration=1451594700&hwt=4085b42190c19a0a1b41cdca4124fca5
46 https://www.reddit.com/r/leagueoflegends/comments/3jkcg6/am_i_the_only_one_who_has_reported_like_extremely/cuqjx0f
65 https://www.reddit.com/r/leagueoflegends/comments/3dq6gw/theres_a_man_whos_ruining_plat_mmr_draft_games/ct7saev
67 http://gdcvault.com/play/1022160/More-Science-Behind-Shaping-Player [min 19:05 - 20:30]
68 https://www.reddit.com/r/leagueoflegends/comments/3cf5ky/i_have_never_seen_the_successful_player/csvhgtq
80 https://ask.fm/RiotLyte/answers/132069455298 ; Page 140
87 http://hwcdn.libsyn.com/p/d/e/d/dedcb49677a8dcd1/PoGCast_Episode_4_-_Toxic_Behavior.mp3?c_id=9204818&expiration=1452162393&hwt=7007297a4a1f6e78a32a038fbbd96486
117 https://ask.fm/RiotLyte/answers/130575469250 ; Page 326
122 https://ask.fm/RiotLyte/answers/130576421570 ; Page 324
125 https://ask.fm/RiotLyte/answers/130577031106 ; Page 324
126 https://ask.fm/RiotLyte/answers/130577014978 ; Page 324
127 https://ask.fm/RiotLyte/answers/130577014978 ; Page 324
137 http://boards.na.leagueoflegends.com/en/c/riot-official/BvErb2td-new-player-reform-system-heads-into-testing?comment=00120000
144 https://www.reddit.com/r/leagueoflegends/comments/36sgex/soo_anybody_gotten_banned_yet_willing_to_share/crh7gnt
149 https://www.reddit.com/r/leagueoflegends/comments/36ow3s/new_player_reform_system_heads_into_testing/crh580c
171 https://ask.fm/RiotLyte/answers/130581249730 ; Page 324
175 https://ask.fm/RiotLyte/answers/130588741826 ; Page 316
179 https://ask.fm/RiotLyte/answers/130598831042 ; Page 315
182 https://ask.fm/RiotLyte/answers/130601189570 ; Page 314
187 http://na.leagueoflegends.com/en/news/game-updates/player-behavior/upgrading-tribunal
203 http://boards.na.leagueoflegends.com/en/c/miscellaneous/kUgY8BYE-regarding-voice-chat-leauge?comment=0005
207 https://www.reddit.com/r/leagueoflegends/comments/36ow3s/new_player_reform_system_heads_into_testing/crh56sq
208 https://www.reddit.com/r/leagueoflegends/comments/36ow3s/new_player_reform_system_heads_into_testing/crglm6d
209 https://www.reddit.com/r/leagueoflegends/comments/36ow3s/new_player_reform_system_heads_into_testing/crglm6d
221 http://na.leagueoflegends.com/en/news/game-updates/player-behavior/player-behavior-design-values-punishment
224 http://na.leagueoflegends.com/en/news/game-updates/player-behavior/instant-feedback-powers
227 http://na.leagueoflegends.com/en/news/game-updates/player-behavior/reports-how-do-they-work
229 http://na.leagueoflegends.com/en/news/game-updates/player-behavior/reports-how-do-they-work
250 https://www.reddit.com/r/leagueoflegends/comments/36ow3s/new_player_reform_system_heads_into_testing/crgerx5
254 https://www.reddit.com/r/leagueoflegends/comments/36ow3s/new_player_reform_system_heads_into_testing/crg2ulr
255 https://www.reddit.com/r/leagueoflegends/comments/36ow3s/new_player_reform_system_heads_into_testing/crg2tym
259 https://www.reddit.com/r/leagueoflegends/comments/36ow3s/new_player_reform_system_heads_into_testing/crfuqva
260 https://www.reddit.com/r/leagueoflegends/comments/36ow3s/new_player_reform_system_heads_into_testing/crfulfc
278 https://ask.fm/RiotLyte/answers/134527663810 ; Page 6
279 https://ask.fm/RiotLyte/answers/134527205570 ; Page 7
304 https://ask.fm/RiotLyte/answers/130594064578 ; Page 312
305 https://ask.fm/RiotLyte/answers/130601915586 ; Page 311
330 https://ask.fm/RiotLyte/answers/134185966530 ; Page 28
336 https://ask.fm/RiotLyte/answers/132195767490 ; Page 121
344 https://www.reddit.com/r/leagueoflegends/comments/369a4k/reporting_feels_so_useless/crc0gy8
351 https://www.reddit.com/r/leagueoflegends/comments/369a4k/reporting_feels_so_useless/crbz8bh
353 https://www.reddit.com/r/leagueoflegends/comments/34rghy/why_does_it_say_the_tribunal_system_has_improved/cqxo5wx
376 https://ask.fm/RiotLyte/answers/130998053570 ; Page 210, 211
389 https://ask.fm/RiotLyte/answers/130717609922 ; Page 283
417 https://ask.fm/RiotLyte/answers/130722330562 ; Page 277
454 https://www.reddit.com/r/leagueoflegends/comments/34rghy/why_does_it_say_the_tribunal_system_has_improved/cqxiw6t
460 https://ask.fm/RiotLyte/answers/132338559426 ; Page 107
472 https://www.reddit.com/r/leagueoflegends/comments/34rghy/why_does_it_say_the_tribunal_system_has_improved/cqxhupk
475 https://www.reddit.com/r/leagueoflegends/comments/32tq7r/eune_toxicity_report_79_of_games_are_toxic/cqepf4n
495 https://ask.fm/RiotLyte/answers/131165741506 ; Page 140
540 https://ask.fm/RiotLyte/answers/133520179906 ; Page 61
546 https://ask.fm/RiotLyte/answers/133489071298 ; Page 64
548 https://ask.fm/RiotLyte/answers/133478793154 ; Page 64, 65, 230, 24
557 https://ask.fm/RiotLyte/answers/132976907714 ; Page 77
574 https://ask.fm/RiotLyte/answers/132567836098 ; Page 133
580 https://ask.fm/RiotLyte/answers/132066956738 ; Page 243
584 https://ask.fm/RiotLyte/answers/132037756866 ; Page 152
589 https://ask.fm/RiotLyte/answers/131818129602 ; Page 163
591 https://ask.fm/RiotLyte/answers/131632237250 ; Page 167
594 https://ask.fm/RiotLyte/answers/131560051650 ; Page 118, 171
601 https://ask.fm/RiotLyte/answers/131185401794 ; Page 189
602 https://ask.fm/RiotLyte/answers/131188058306 ; Page 189
618 https://twitter.com/RiotSupport/status/77656037665787904
650 https://boards.na.leagueoflegends.com/en/c/player-behavior-moderation/9VrUtrJo-being-solo-and-having-a-team-report?comment=000000000002

Created by Coxis on 16-01-2017. Last edited: 28-06-2017