Welcome back everyone. I’ve decided to start a new series of posts I call “Yomu Investigates…” where I look into certain theories I may have in regards to anime. Considering I get all of my general anime statistics from MyAnimeList (MAL), such as ratings and popularity, I’ll be using MAL for a lot of these.
Although it wasn’t titled such, I’d like to consider my post on sequels the spiritual predecessor to this series.
And with that, the first topic I’d like to investigate is reviews on MAL!
I have two theories in regards to MAL reviews:
- People are more likely to leave a review that is overly positive, or overly negative. In other words, people are more likely to leave a polarized review.
- People are more likely to find polarized reviews “helpful” due to confirmation biases.
So basically, the plan is to look at reviews on MAL and see if my theories are correct. One of the aspects of the “rule of 7“, which I wrote about in this post, is that deviation from the standard (which is a 7 rating), will cause polarization of opinions by the consumer. This polarization is what we’ll look at here.
The idea behind this is pretty simple – people are more likely to leave a polarized (overly negative or positive) review because they are more invested in the product, either through a negative or positive experience.
For example, if I bought a product that just did what was advertised, and nothing more, I probably wouldn’t bother leaving a review just to say “works as intended”. But if that product broke down, or didn’t work as advertised, I might be angry enough to leave a negative review warning others about my experience. The flip side of course would be if the product worked better than what was advertised, I might be so pleased with it that I want to leave a positive review to encourage others in regards to my experience.
So in theory, it makes sense that this would also apply to anime reviews right?
Lucky for us, MAL has a tab specifically for reviews, so I can scroll through and get a general idea of the ratings people are providing for anime of all kinds.
Here is how I will consider ratings in terms of “negative” or “positive”:
- 1-3: Overly Negative
- 4-5: Negative
- 6: Below Average
- 7: Average
- 8: Above Average
- 9-10: Overly Positive
For this investigation, I’ll be looking for mostly 1-3 and 9-10 ratings, which I consider polarized. 4-5 I consider negative, but not necessarily polarized – I think that anime in this range might be bad enough to motivate people to leave a review though. 6-8 I consider in the “average” range.
It’s funny how the rating scale works out, top heavy as it is. You’d think that 5 would be the average rating, but in practice that’s just not the case. I believe the reason for this is because 7/10 means that the anime managed to be 70% of what one would consider a “perfect anime”. 5/10 would mean an anime only managed to be half as good as a “perfect anime”, which is pretty bad when you think about it.
So let’s take a look at theory 1.
1. People are more likely to write a polarized review
I’ve looked at 5 random pages of anime reviews on MAL, and here are the numbers I have come up with:
- 118 polarized reviews (1-3 + 9-10 rating)
- 103 average reviews (6-8 rating)
- 29 negative reviews (4-5 rating)
- (147 polarized + negative reviews (1-5 + 9-10 rating))
And when you look at my sample size of 250 reviews, it’s actually fairly close. If you look at strictly polarized and average reviews, it’s pretty even.
If one was to consider the 4-5 “negative” range as polarized, with the argument that 4-5 ratings are negative enough to serve as motivation to leave a review – we then get approximately 50% more polarized reviews than average. It just depends on how you view it here.
Either way, I was surprised at the number of average reviews. Overall, in aggregate, people leave reviews of all ratings. Slightly more polarized than average, but even enough that I’d call it balanced, considering the small sample size.
So theory #1 has been debunked. While there are slightly more polarized reviews, it’s not enough to definitively confirm the theory. People overall write reviews of all ratings pretty evenly.
Onto theory #2!
2. People are more likely to find polarized reviews “helpful” due to confirmation biases
So people may not entirely be more likely to WRITE a polarized review, but what about their likelihood of upvoting / finding polarized reviews helpful?
This is the data from the top 5 pages of “Most Helpful” reviews on MAL:
- 173 polarized reviews
- 44 average reviews
- 33 negative reviews
- (203 polarized + negative reviews)
Crazy right? In aggregate people put reviews out there of all ratings, but to make it to the top of the “Most Helpful” list, you have a MUCH better chance with a polarized review. Last theory we could call close, but this time we are very far from that.
People are MUCH more likely to upvote a polarized review on MAL. Almost 5 times more likely, from what we see here at least.
Is this due to confirmation biases? While I can’t state this without a doubt, I believe that it is the case. If I hated an anime, and see a review from someone else who hated it, I’ll be more likely to find the review helpful because I agree with it. Same goes for anime we love.
The other point to possibly consider is that the “Most Helpful” reviews get more exposure than regular reviews, and so they are more likely to get upvotes in the first place. But even then the reviews would have had to get some upvotes to get into that position initially, and polarized reviews seem more likely to get there.
So what does this mean?
When we look at the synopsis page for any anime on MAL, the reviews that we are provided with are the “Most Helpful” reviews for that anime. My question then is this: are they REALLY the most helpful reviews? Or is the review upvoting system just a way for polarized opinions to make it to the top, through confirmation biases?
If polarized reviews are approximately 5 times more likely to receive a “helpful” upvote, then the system is a little skewed isn’t it? After all, we now know that people tend to write reviews of ALL ratings in aggregate.
From the samples I took today, there are almost an even number of polarized and average reviews being written, but polarized reviews are simply more likely to get upvoted.
So I think the takeaway from this investigation is this: the reviews that you see on synopsis pages for anime are skewed towards polarized opinions. Keep this in mind if you are looking at an anime and decide to skim the reviews that are there for a second opinion. An anime that many people may have found average might only consist of polarized reviews on the synopsis page as a result of the “helpful” upvote system.
Here is a good example of this in practice: Sword Art Online. The anime has a 7.61 rating, which is slightly above average. But the top 4 reviews shown give the anime a 4, 7, 3 and 3 rating – which would average out to only 4.25 rating. Huge difference between the aggregate opinion of viewers and the “Most Helpful” reviews.
I suppose the final question here is this: are the reviews on MAL intended for people who HAVEN’T yet watched the anime? Or are they intended for people who HAVE already watched the anime?
If they are for those who have not yet seen the anime, then I believe that the “Most Helpful” reviews are biased towards polarized opinions, which is not representative of the aggregate opinion. This could affect someone’s decision as to whether or not they want to watch a certain anime, and could also affect their viewing experience due to polarized expectations.
If they are for those who have already watched the anime, then I think the polarization isn’t as bad, seeing as people have already had the opportunity to form their own opinions on the anime.
The case remains though that the “Most Helpful” reviews are 5 times more likely to be polarized, and overly negative or overly positive.
And that’s all I’ve got for this post. It ended up being pretty interesting! While my first theory was incorrect, my second theory ended up providing us with some useful information in regards to MyAnimeList “Most Helpful” reviews.
What do you all think about this?
At the very least, I think it’s something to keep in mind next time you decide to skim the reviews on the synopsis page when deciding to watch an anime!
Until next time.
Thanks for reading.
13 thoughts on “Yomu Investigates… MyAnimeList Reviews”
Great post. Although Im not sure about the confirmation bias. I’m sure it plays a role but i find reviews that clearly state It’s the best or It’s the worst more helpful at a quick glance. MAL doesn’t necessarily attract people interested in reading long essays of a show so a clear unambiguous love it/hate it review is probably helpful in an uncomplicated sense.
I could be wrong but this may also factor in there
LikeLiked by 1 person
Yeah there’s no doubt much more to it. That’s a good point though, polarized reviews are more likely to get their point across for sure. Seems like a very good reason for why they may get “upvoted” more often.
It’s just situations where an anime has an average rating, and all of the top reviews are negative (or vice versa) – can that be considered actually helpful? Seeing as on average people enjoyed the anime MORE than the top reviews…
LikeLiked by 1 person
I read some MAL reviews whenever I go to an anime page. More than half the time they’re pretty unhelpful. Confirmation biases I feel plays a huge part in the voting for some of these series. On Aldnoah.Zero MAL page I have one of top four reviews. I got comments when I posted that of many people basically agreeing with me. I did get one fanboy who called me a moron for over-analyzing it. I mean hey, if I feel like something has horrifically bad writing I’ll tear it apart.
From my experience, whenever I post a negative review on MAL I get more comments of people saying they’re glad to find someone who agrees with them than what I’ve actually written. My favorite one is one I posted for Death Parade (not in the top 4), and this user response to everything I wrote was “You missed the point”.
Reviews on MAL usually try to ride some kind of train. I’ve seen dozen of reviews on anime pages where the top reviews didn’t even finish watching the anime. Almost all of them try to keep what they like vague, though don’t do a good job explaining their position. Either being too brief to be of any help, or write a lot without a whole lot of substance. On rare occasions, there are ones that have a lot of effort put into them that I like reading. Almost all of them so far have been of differing opinions to mine.
Also one thing about MAL, if a popular series ends, like it did with Your Life In April, you can expect the review feed to be absolutely flooded with several pages of reviews of a single anime that can last for days. MAL reviews for me are generally badly written (including my early ones, yikes), but hey, if someone wants me to help them with theirs because they like mine (which happens sometimes) it’s posting them to help “improve” them with my help haha
LikeLiked by 1 person
Hey that’s pretty neat! I’ve never really looked into the MAL community so to speak, so it’s pretty cool that you are in it like that. Do you find it rewarding to write on MAL? Just curious.
Also that all makes a lot of sense. So even if people aren’t more likely to write a polarized review, they ARE more likely to leave polarized feedback, both in the form of upvotes on polarized reviews AND comments to the review author.
To be honest, this is all more interesting than I had expected it to be when I started this post!
LikeLiked by 1 person
I did, but once they got rid off their “Not Helpful” button it didn’t feel quite as rewarding. Before they took it out, I recall my Aldnoah.Zero review having like 147 helpful vote out of 170 total votes. That felt like an accomplishment to me because before that it was either split, or largely negative. It also gave me small feedback in figuring out what type of review people who read reviews on MAL wanted from me.
It worked better with lesser known anime titles like J.C. Staff Cat Soup. There I felt the total votes was a better indication of feedback. I still get a rare comment from random users telling me my review of it either helped them understand the movie a little better, or made them have a greater appreciation of it. I feel like my reviews made me come across as approachable compare to the average reviewer on MAL. From the impressions I got the users end up valuing our casual conversations more since it helps them better think about how they view anime, and how to express themselves on it.
I know several people who are also in the Top 50 reviewers on MAL. 4 out of the 5 I’ve talked to don’t really consider it a badge of honor. Telling me they simply got it because they would occasionally ride a hype train for votes. One stands out to me because his One Punch Man got over 1000 help votes, and that was all he needed to crack the top 50. Once they got into the Top 50, they just went on to cover whatever they really wanted.
The one friend I know who had goals to reach it is upset about it since he hardly gets any votes. He expressed a few times how he’s envious that I write significantly less than he does, and typically get more votes in one review than he does in five put together. I’m pretty sure he’s happy I haven’t posted a review on MAL for over a year. He might not like my return if I post them (at least the ones I don’t consider impressions) on there again.
If you investigate suggestion, I recommend you go to the Pingu anime page by Polygon Pictures. It’s a great source of comedy.
I didn’t realize it all worked that way, although I guess I am not entirely surprised. Do you get more votes than that one friend because you write better reviews? Or is it because you are more recognizable or something like that? Idk how MAL works with this, can you follow reviewers like I can follow blogs here on WordPress? This is all very interesting to me, thanks for sharing.
Haha I’ll keep Pingu in mind.
My friend, who I’ll refer to as Pryx (his username is something like that), has actually been on MAL a lot longer than I have. I feel like I made smarter choices when interacting with the community on MAL. Whenever I could, I welcome people into the community, participated in silly threads that would make people laugh, and commented on some blogs I read on MAL whenever I did. I had more engagement in my interactions in the community because even if I didn’t talk about anime I could speak about games, music, and a number of other stuff with good results. Pryx, just mostly about anime.
I would say I wrote better reviews than he did, but I’ll grow an ego if I do XD. For him, his thing was writing about harems, and airing anime. That’s about it. He didn’t try to go out of his comfort zone so if someone did follow him they might have developed a fatigue from what he wrote about. He didn’t use the blog feature to showcase what else he’s capable of writing so that might added to it. I regularly go out of my comfort zone, and tried to have a nice balance of things. One moment I would review Berserk from the 90s, and the next it’s an obscure Spanish dub of a Korean anime called Golden Batman. Unlike him, I didn’t take myself seriously in everything I wrote. I would crack jokes whenever needed to keep readers engaged in some reviews I wrote. That pays off well since some comedic offshoot makes random users check your other stuff regardless.
I also think it was through small word of mouth from some of the Top reviewers on MAL that got me more votes than he did. I know I got ZelphSilver who commented to me he liked my review of Aldnoah.Zero a lot, and like how I don’t take an opposing viewpoint personally. My reviews seemed to be better received by more popular reviewers on MAL, and so they would casually recommend my stuff to other users. My review of the SAO recap movie was very briefly mention in a video by RogerSmith2004 so that boosted me a little bit. He’s not a big YouTuber, but it helped all the same. Here’s that very, very brief mentioned.
So while I consider myself a tiny spec in that whole MAL reviewing community I made good impressions where it counted unlike Pryx where reviewers I asked privately in DM didn’t know who he was. In one case, someone asked if I was playing a joke Pryx went into the Top 50. Pryx not a bad dude, but he has a very limited range of what type of anime he watches.
No you can’t follow reviewers on MyAnimeList. Best thing they have for that is simply friending someone, and looking through their reviews that way. They won’t even notify you if a friend on your friend list on MAL posted anything or not. The most it’ll remind you of is either what they watched, or when a birthday is coming up. If they had a feature, it would make sense, but they don’t for some reason.
LikeLiked by 1 person
Thanks for sharing all of this. I found it quite interesting, MAL is a very different animal from our blogging community here it seems.
LikeLiked by 1 person
I’m going with confirmation bias…
Yup. And that’s all I have to say.
LikeLiked by 1 person
I feel like part of it is that people also don’t know how to review on MAL. Half the time they rank it and go ‘OMG XXX CHARACTER WAS SO HOT. ANIMATION GOOD. YAY!!!’, or maybe that’s just what pops up into my feed. Regardless if it’s positive or negative, that type of review doesn’t “help me” in anyway. Same with series suggestions (maybe something for you to investigate later?).
The few times I’ve actually marked a review as helpful or not, regardless if it was positive/negative is if it was actually articulate in WHY they thought that way. I disagree with a lot of people when it comes to something like ‘the art style is crap’. If they explain that the art style being so poor in animated movement actually deterred them during their watch experience… that’s different. I feel like people are more articulate when their experiences are more positive or negative (it’s certainly true for me). But I do think confirmation bias is a factor in there too.
Certainly food for thought. I’m looking forward to your other future investigations!
LikeLiked by 1 person
Good point, it definitely seems easier to write a negative or positive review than try and write about something that is average. Makes sense that these reviews may be a bit more appealing to readers.
A lot of the generic reviews I read seemed to be just as you say, people listing things they liked or didn’t like, and providing a rating. No explanation as to why.
I’ll take a look at series suggestions and see if I can come up with something to test there. Thanks for the idea.
LikeLiked by 1 person
Pingback: Late Appreciation Post but still Thankful! – Sakuras Galleria – An Anime Blog
Pingback: Handa-kun & Barakamon – Prequel / Sequel Disconnect | Umai Yomu Anime Blog