Yomu Investigates… MyAnimeList Reviews

Welcome back everyone. I’ve decided to start a new series of posts I call “Yomu Investigates…” where I look into certain theories I may have in regards to anime. Considering I get all of my general anime statistics from MyAnimeList (MAL), such as ratings and popularity, I’ll be using MAL for a lot of these.

Although it wasn’t titled such, I’d like to consider my post on sequels the spiritual predecessor to this series.

And with that, the first topic I’d like to investigate is reviews on MAL!

Figure I should try to look smart every once in a while. In reality I have no clue what I’m doing.

I have two theories in regards to MAL reviews:

1. People are more likely to leave a review that is overly positive, or overly negative. In other words, people are more likely to leave a polarized review.
2. People are more likely to find polarized reviews “helpful” due to confirmation biases.

So basically, the plan is to look at reviews on MAL and see if my theories are correct. One of the aspects of the “rule of 7“, which I wrote about in this post, is that deviation from the standard (which is a 7 rating), will cause polarization of opinions by the consumer. This polarization is what we’ll look at here.

The idea behind this is pretty simple – people are more likely to leave a polarized (overly negative or positive) review because they are more invested in the product, either through a negative or positive experience.

For example, if I bought a product that just did what was advertised, and nothing more, I probably wouldn’t bother leaving a review just to say “works as intended”. But if that product broke down, or didn’t work as advertised, I might be angry enough to leave a negative review warning others about my experience. The flip side of course would be if the product worked better than what was advertised, I might be so pleased with it that I want to leave a positive review to encourage others in regards to my experience.

So in theory, it makes sense that this would also apply to anime reviews right?

Lucky for us, MAL has a tab specifically for reviews, so I can scroll through and get a general idea of the ratings people are providing for anime of all kinds.

Here is how I will consider ratings in terms of “negative” or “positive”:

• 1-3: Overly Negative
• 4-5: Negative
• 6: Below Average
• 7: Average
• 8: Above Average
• 9-10: Overly Positive

For this investigation, I’ll be looking for mostly 1-3 and 9-10 ratings, which I consider polarized. 4-5 I consider negative, but not necessarily polarized – I think that anime in this range might be bad enough to motivate people to leave a review though. 6-8 I consider in the “average” range.

It’s funny how the rating scale works out, top heavy as it is. You’d think that 5 would be the average rating, but in practice that’s just not the case. I believe the reason for this is because 7/10 means that the anime managed to be 70% of what one would consider a “perfect anime”. 5/10 would mean an anime only managed to be half as good as a “perfect anime”, which is pretty bad when you think about it.

So let’s take a look at theory 1.

1. People are more likely to write a polarized review

I’ve looked at 5 random pages of anime reviews on MAL, and here are the numbers I have come up with:

• 118 polarized reviews (1-3 + 9-10 rating)
• 103 average reviews (6-8 rating)
• 29 negative reviews (4-5 rating)
• (147 polarized + negative reviews (1-5 + 9-10 rating))

And when you look at my sample size of 250 reviews, it’s actually fairly close. If you look at strictly polarized and average reviews, it’s pretty even.

If one was to consider the 4-5 “negative” range as polarized, with the argument that 4-5 ratings are negative enough to serve as motivation to leave a review – we then get approximately 50% more polarized reviews than average. It just depends on how you view it here.

Either way, I was surprised at the number of average reviews. Overall, in aggregate, people leave reviews of all ratings. Slightly more polarized than average, but even enough that I’d call it balanced, considering the small sample size.

So theory #1 has been debunked. While there are slightly more polarized reviews, it’s not enough to definitively confirm the theory. People overall write reviews of all ratings pretty evenly.

Onto theory #2!

2. People are more likely to find polarized reviews “helpful” due to confirmation biases

So people may not entirely be more likely to WRITE a polarized review, but what about their likelihood of upvoting / finding polarized reviews helpful?

This is the data from the top 5 pages of “Most Helpful” reviews on MAL:

• 173 polarized reviews
• 44 average reviews
• 33 negative reviews
• (203 polarized + negative reviews)

Crazy right? In aggregate people put reviews out there of all ratings, but to make it to the top of the “Most Helpful” list, you have a MUCH better chance with a polarized review. Last theory we could call close, but this time we are very far from that.

People are MUCH more likely to upvote a polarized review on MAL. Almost 5 times more likely, from what we see here at least.

Is this due to confirmation biases? While I can’t state this without a doubt, I believe that it is the case. If I hated an anime, and see a review from someone else who hated it, I’ll be more likely to find the review helpful because I agree with it. Same goes for anime we love.

The other point to possibly consider is that the “Most Helpful” reviews get more exposure than regular reviews, and so they are more likely to get upvotes in the first place. But even then the reviews would have had to get some upvotes to get into that position initially, and polarized reviews seem more likely to get there.

So what does this mean?

When we look at the synopsis page for any anime on MAL, the reviews that we are provided with are the “Most Helpful” reviews for that anime. My question then is this: are they REALLY the most helpful reviews? Or is the review upvoting system just a way for polarized opinions to make it to the top, through confirmation biases?

If polarized reviews are approximately 5 times more likely to receive a “helpful” upvote, then the system is a little skewed isn’t it? After all, we now know that people tend to write reviews of ALL ratings in aggregate.

From the samples I took today, there are almost an even number of polarized and average reviews being written, but polarized reviews are simply more likely to get upvoted.

So I think the takeaway from this investigation is this: the reviews that you see on synopsis pages for anime are skewed towards polarized opinions. Keep this in mind if you are looking at an anime and decide to skim the reviews that are there for a second opinion. An anime that many people may have found average might only consist of polarized reviews on the synopsis page as a result of the “helpful” upvote system.

Here is a good example of this in practice: Sword Art Online. The anime has a 7.61 rating, which is slightly above average. But the top 4 reviews shown give the anime a 4, 7, 3 and 3 rating – which would average out to only 4.25 rating. Huge difference between the aggregate opinion of viewers and the “Most Helpful” reviews.

I sense it too. It’s coming from the MAL “helpful” reviews…

I suppose the final question here is this: are the reviews on MAL intended for people who HAVEN’T yet watched the anime? Or are they intended for people who HAVE already watched the anime?

If they are for those who have not yet seen the anime, then I believe that the “Most Helpful” reviews are biased towards polarized opinions, which is not representative of the aggregate opinion. This could affect someone’s decision as to whether or not they want to watch a certain anime, and could also affect their viewing experience due to polarized expectations.

If they are for those who have already watched the anime, then I think the polarization isn’t as bad, seeing as people have already had the opportunity to form their own opinions on the anime.

The case remains though that the “Most Helpful” reviews are 5 times more likely to be polarized, and overly negative or overly positive.

And that’s all I’ve got for this post. It ended up being pretty interesting! While my first theory was incorrect, my second theory ended up providing us with some useful information in regards to MyAnimeList “Most Helpful” reviews.

What do you all think about this?

At the very least, I think it’s something to keep in mind next time you decide to skim the reviews on the synopsis page when deciding to watch an anime!

Until next time.
Thanks for reading.