About Stack Overflow’s screwed moderator system

Hi ladies and gentlemen,

Welcome back to our weekly blog posts that intend to constructively criticize Stack Overflow’s function and the Stack Exchange site overall. It has been a pleasure to see so many views that these blog posts get.

We have not even dreamed about it, but it is really cool to see it happening. We would like to express my greatest gratitude for that to you guys even if you disagree with what we write over here! Enough of that said, let us get back to
our main topic today.

As you may probably already well know, the Stack Exchange sites are community moderated. Just in case, so that we know that we are on the same page, the moderation happens in different layers:

Moderation by regular users via so called “privileges”

The higher reputation you gain, the more power you have on the site, including meta, i.e. not just the main site. The highest privilege is becoming a Trusted User at 20K reputation as of today. Typically, this is not particularly difficult to achieve if you are an expert of a technical area, especially if that is a very common mainstream thing, e.g. Java or C#.

Some people have achieved this in a few months, including me, even though these volunteers are employed, have families, other hobbies and all that. At this privilege level, you can see deleted posts, vote for closure, reopen, deletion undeletion (also on answers) with certain limitation and all that.

Elected moderator team

There are annual elections held and the team members are elected by the community, who are not necessarily affiliated with Stack Exchange by any means and they usually are not. There are currently 17 moderators on Stack Overflow as of writing this. The last election raised three new moderators on board. I must confess that the election process is relatively good, although there are some quirks in there, too, including severe ones.

For instance, last time in the beginning of this year a 100K+ expert was led to committing account suicide after getting many childish and revenge downvotes on his answers all around. His honest mistake was to be culturally different than some other “valuable” contributors who happened to be childish with committing to revenge activities. Either way, they called the nominee snarky and that is why he “deserved” many downvotes on technically correct and valuable answers.

Anyway, this team has much more power than the Trusted Users in the previous group. They have binding close votes, they can delete posts, even selected answers or upvoted and positively scored questions; they can also undelete posts and once they delete posts, other community members cannot undelete them unless they also happen to be at least elected  moderators. They can also suspend users, modify spam flags on posts, investigate more about puppet accounts, remove comments and all that jazz. They cannot, however, disassociate users from the their posts as per the Creative license!

Community managers

These are the Stack Exchange employees that even have more power than the community elected moderators. They can talk to the site developers for deeper analysis about pretty much anything that is happening all over the place, but they can also have access to tools for detecting serial down/up vote patterns, etc. They can also disassociate contributors from their posts as per the Creative license. You could call these the kings and queens in kingdom.

Since many of us already have a great job, enjoying and living in harmony with it, it is unlikely that the vast majority of the community will ever apply for this category. As I already wrote above, the first category is relatively easy to achieve if you are really good at what you are doing, and willing to help newcomers with your skills and knowledge. I am hereby willing to write more about the “middlemen”. To be honest, it is not fair to call them middlemen since that usually raises negative feeling among technical people (or even outside) in the UK, so I will call them their name, diamond moderators.

As you may have hopefully noticed above, and I hope you indeed did since that means you read the post carefully in every important detail, this team consists of 17 members as of today. You may be pondering why I am emphasizing this fact again. Alright, let us give some food for thought in here: Stack Overflow has gazillion posts even per day and that is just posts! How about the noisy comments that, I believe, happen in large amount, too?

To give some exact, concrete and real numbers without gut feeling, I looked up the meta site for further statistics. There was a meta post last October that says that there are about 7.2K questions and 15.84K answers per day. That sums up to 23 K posts.

I am fairly certain there is at least that amount of comment taking place, too, if not much more! It is probably needless to repeat, but these numbers are just raising up over time as the tendency shows that more and more people come to Stack Overflow to ask (mostly silly) questions since this is the first google hit for them when looking for programming answers, sometimes coming up even overtaking the official documentation in the given field.

Right, I am hearing you…. moderators ain’t so overloaded because there are a large number of Trusted Users and otherwise useful contributors who aid their job. There is probably some agreement there for most of the parts, but are they really not overloaded for the worse of the site? Can we revisit that question for a moment, please? Alright, you are claiming that most of the moderation queues are well handled, the site is getting more and more popular, so everything has been working fine. Why are you even criticizing then?! OK, fair enough, but please bear with me for a moment; I will throw some observation in.

* Post per diamond moderator for one day

First and foremost, let me repeat, there were about 23K+ posts per day around last year and very likely much more comments. Considering 17 moderators, that is probably about 1400-1500 posts per day for a moderator with the help of the community. That is a very high number, let me repeat, that is a lot, especially since most of the questions are somewhat rubbish on Stack Overflow, and that is not what I am saying. That is what the community has been saying at large, including old-time and freshly joined experts.

* tag per diamond moderator

This is also a quite interesting and important metric here in my opinion, namely: there are at least about 100 tags which need experts on daily basis, and most of them more than just one to answer questions, let alone moderating. I may actually be underestimating the important tags in here since I have just taken the first 3-4 tag pages into account based on all-time traffic, and there are always raising tags, like the new shiny Swift language tag after Apple’s announcement, that is getting very huge traffic. Anyway, this means about six extremely popular and common tags to moderate. This does not require a genius to claim that due to this, moderators usually need to act out of their expertise when judging which inherently leads to mistaken actions.

* meta questions per diamond moderator

There are usually lots of questions on meta sites addressed to diamond moderators that probably they can answer the best way without other community members writing gut feelings down. This also consumes their resource relatively heftily. This is not to underestimate either since one of the diamond moderation nomination principles is that you ought to be hanging on Meta a lot to know the community and its rules well.

* meta post moderation per diamond moderator

This is another thing to consider, respectively. Although with the Stack Exchange and Stack Overflow meta split, things got a bit better since the reputation from the main site entitles you to moderate with the same privileges as on the main site. Still, this puts further burden on the moderators’ shoulder. I would say the discussions are usually even more heated on Meta than Stack Overflow, so this is not a negligible amount of work here, either.

* chat and other communication channels

Not rarely, moderators are “pinged” in chats, email and via other means to discuss issues about the operation of the site. This is again further time taken from them. I am sure there are other issues, too, that I have forgotten to mention or just not been aware of!

* technical participation to provide answers and comments

Yes, this is not necessarily unusual either. Some moderators may still prefer to spend their free time with providing good answers to interesting questions that newcomers may have. This could also consume a considerable amount of time from them depending on the involvement proportion.

So, as you can see, there is a lot of work to be done from their side based on the responsibilities they accepted before nominating themselves. Still, I think they cannot fulfill the responsibilities that they have. I see that regularly that valid comment flags are declined, when for instance they are marked as obsolete, just because they do not understand the context or they do not even have time to do so.

This is not only sad in that particular scenario that the mess is left there intact, and hence the noise, but what is more unfortunate is that the experts may completely lose the motivation to flag such content anymore and will not be part of this workflow. In other words, it is demoralizing which could lead to all kinds of bad avalanche if that expert expresses the demotivation in public without suggesting a cure that is followed by the moderator team, or even considered.

This is again a real world example. One of us was lurking in the “Tavern” chat on Meta Stack Exchange when one of the diamond moderators proudly wrote that he had handled about 1K flags in one day. This sounded truly impressive at first followed by huge gratitude from the participants. But let us step back for a moment: how many flags really does that include? It does not require a mathematician to realize that it is more than 33 flags per hour, which in turn means one flag per every two minutes.

While, it is possible to burn several comments in one go after understanding the context, I find this very inconvenient and uncomfortable because I am almost certain this had not included the real quality job needed for each individual action. That is, and do not forget that I have calculated this number with 24 hours per day, and this was a week day if I remember  correctly. What about family, school/work, eating, having shower, girlfriend time or just sleeping? This number will be much higher when those are rightfully taken into account.

Unfortunately, I also notice that there several questions brought up on Meta, almost on a daily, but at least weekly basis about diamond moderator mistakes or not completely careful actions from them. What is even worse than that is that they seem to oftentimes take defensive stance rather than admitting mistakes upfront. I completely understand where they are coming from with a lot of work to be accomplished, but frankly, I think this pressure is turning them into what they are.

I do not think being defensive is the right action to call for in this scenario. I also have several inappropriately declined flags and some still pending after a couple of weeks that I have not brought up for remedy, and I will likely simply not do so to avoid the conflicts. But then again, this is kinda defeating the original purpose of the community moderation if we are let down by bad reactions. Having said that, you reach the point at times after which you just give up, and that is relatively understandable.

You may be claiming at this point that yeah, you agree with us that there is a lot on the moderators’ plate, but what could be done here other than doing our Trusted User and other roles extraordinarily well? They are trying to improve the web UI for moderators at Stack Exchange you are saying… Oh yeah?

Well, the solution is really relatively easy and simple in our humble opinion, but of course this means there is more trust to be given Trusted Users and experts from Stack Exchange’s side. They are called Trusted Users for a reason after all! I would give them much more power than they have, either through a selection process or just by badges (gold, silver, etc.).  We have been speaking about this for a long while now.

Luckily, after a lot of pushing, Stack Exchange seems to have started doing something about it in this regard as experts can now mark questions duplicate or even unduplicate them with one single action. However, this is still very far from what the normal situation would be in our opinion. The noisy comment walls are still there uncleaned oftentimes, some poor questions  and answers are still unhandled, and do not forget about the important fact that it is just raising over time, sadly. I will not reiterate all the issues that I have written above. I think you got the point.

I will finish it here for today. Hope that you have enjoyed this blog post, too, and as usual, we will be back next week. We will write a bit more about the expert role in the Stack Overflow community, but let us leave the long details with the upcoming blog post.

You will be more than welcome to come back, dear reader. We wish you an awesome week ahead; be happy and smile a lot!