Unfair Overall Rating Calculations

Unfair Overall Rating Calculations

Mean rating status calculations are exactly as the name implies, mean to hosts! One bad review can easily destroy a host's efforts to get or maintain star status and there is nothing we can do about it.

 

If a host gets several 5-star reviews and only one 1-star review, their ratings status calculated currently by mean (average) is pulled right down. A median rating is  more reflective of the most common guest experience and would remain 5 or nearer to it...

 

For an understanding of the difference or to see what it would mean for you, check out: https://www.calculators.org/math/mean-median-mode.php

 

I believe Airbnb should show more fairness to hosts and change to median calculations of overall ratings of both guests and hosts. If we are going to be ruled by algorithms and metrics, we should at least have fair ones... After all, it is we the little hosts that bring profits to the platform and it's billionaire owner.

46 Replies 46

@Patricia1556  I don't think there exists a fair way to reduce something as nuanced and varied as hospitality to arbitrary star ratings. But devil's advocate here, if you're a guest who actually has faith in those star ratings, you'd probably prefer a system that didn't sweep 1-star ratings under the rug.

 

Imagine, hypothetically, a new listing that's not quite ready for prime time. The first 3 guests have a disappointing but acceptable experience, and charitably rate it 5 stars to be nice (you'll find some hosts regard even 4 stars as an act of violence, so that happens all the time). Maybe they each tell the host privately that some things are seriously wrong with the house, assuming it's not worth mentioning in the review because surely they'll be fixed soon anyway. But instead of taking that feedback on board, the host takes those 5's to mean everything is hunky dory as the home falls further into disrepair.  Guest 4 arrives to a home with a "perfect" 5 stars and is shocked at the awful condition it's in. Host points to their perfect record as evidence that the guest is the one with the problem. Guest leaves 1 star review.  Next guest turns up, and by now the house is just a rotten box of rats and snakes. Another 1 star.

 

So here's the data set:  5, 5, 5, 1, 1.

 

Mean:  3.4*

Median:  5*

 

Which do you think would be more reflective of the most common guest experience? 

 

Of course this story is an exaggeration, but actually I did once give a new listing 5 stars for potential and alerted the host to the issues privately. Sadly, I saw later that subsequent guests had the same problems over a year later. The listing's current mean is much closer to my experience than I rated it.

 

Anyway, I can tell you're feeling burned by that review from P*******a, and maybe it was a pure, unmerited revenge review. But it would be worth posting a response owning up to whatever went wrong with that stay, and assuring prospective guests that it was an anomaly. As a guest, when I look at reviews, I value consistency over perfection.

 

 

 

 

 

I gather from the 6 likes that people prefer mean over median? I agree it's good to look at this from the perspective of both consumer and host... I'm not convinced mean is a more accurate reflection of any "truth". That is not my experience... If 7 people out of ten say they had a 5 star experience.... then that is likely closer to the truth of what I'll experience than the 3 out of 10 that nitpicked or purposely slammed over some minor issue...

 

Maybe we should put it to a vote...

I get both perspectives... I think the real issue is the different messaging Airbnb gives hosts and guests.  On most platforms a 4 our of 5 is considered good and Airbnb for guests selecting what to rate Aibnb calls 4/5  "good" too. Buy for hosts, sink to an average 4.5 rating and the company starts threatening suspension of your listing! This is the real problem and has been raised by hosts several times (I found a post below from 2017!!!!) So why the disconnect? If I wasn't threatened with losing my "superhost" status or even having my account suspended because of 1 bad experience in 10, I wouldn't mind an mean over a median. But Airbnb policy is unfair and mean to hosts.

 

Read this from @Rachel26  from 2017:

"Is it fair for Airbnb to have such a high average standard (above 4.5) which basically makes anything lower than an overall 5 Star 'Bad'....... but the email they send to guests after a stay doesn't point this out. It actually reads like this (see below) and I believe this is sending one message to the guest - while sending another message to the Host.

'How was your stay at Fred's place?

 

Terrible

 

Bad

 

Okay

 

Good

 

Great

Share your experience while it’s still fresh. Your review will help Fred improve and tells future guests what to expect.'

 
SO IF FOUR STARS ARE CONSIDERED GOOD - and Airbnb sends this explanation out to the guest with the review request - then why such an unsupportive reaction from Airbnb when Hosts get a 4 star review?
Wouldn't most people booking a place to stay to save money and somewhere cheaper than a hotel - wouldn't they be happy with a 'Good' place? And delighted, but not expected, if a place turned out to be 'Great'? So why is it so bad from Airbnb's point of view to get a mix of 4 star and 5 star reviews?
I also agree with the suggestion that Airbnb should wait to send the warning emails until a host receives an overall 4 star review three times in a row. That would be more indicative of below par performance than the occasional 4 star overall that some guests just seem determined to give to be difficult or whatever reason they have in their heads.
 
This email message from Airbnb to the guests (showing 4 stars to mean 'Good' and 5 stars 'Great') is not helpful if Airbnb continue to use a different measuring stick for Hosts (meaning 4 stars is not 'Good' but you are failing as a Host).

@Patricia1556  This is quite a different argument from mean vs median, but I agree - communicating 4* to hosts as a total failure is Crazypants. It's also by design:  from the perspective of a tech firm, when they see their users obsessing over banal triggers such as stars and likes, it only proves that their behavioral modification strategy is working. 

 

It's operant conditioning, just like training a dog:  create a system of rewards (e.g. Superhost badge) and  punishments (intimidating threats) to trigger the desired behaviors. They don't want you to ever feel confident that your well-earned standing is secure enough to weather even one disgruntled guest - it's in their interest for you to be scared of every single review, even after you have hundreds.

 

But that's not in your best interests as a host. The more you can train yourself to resist the manipulation and ignore the stars and badges and hollow threats, the happier you'll be with your job. 

LOL operant indeed... we are well trained dogs (woof!) because our visibility on the site depends on it.

🙂

basing a rating system in lone with the hotel rating system in the UK of 1 to 5 stars is short sighted - is it not.  guests who have paid £400 a night at a 5 star hotel who then stay in a clean/good/well equipped place may well judge this based on the 5star hotel level when in fact this is never to be confused - said guest then leaves a 3 star review NOT knowing this is possibly the worst thing they could do - AB do not look at anything else lower than a 5star because they want us all to be superhosts (or do they - this is questionable). After 7 years or so of hosting my standards have not dropped. for the 1st 3 years we were superhosts but now find that the several places we host have NO CHANCE of getting a 80% plus hit rate of 5 stars. Let us face it many a fair minded guest might love the place and just decide that a 4 star is more than adequate or just never give top marks for anything. SO the rating systems is flawed. I have asked AB to look into this but simply get ignored.

Pat271
Level 10
Greenville, SC

@Anonymous  @Patricia1556 

 

Andrew, you beat me to it…and I was going to use the exact examples 1,1,1,5,5 and 5,5,5,1,1. You have an annoying time zone advantage. 🙂

 

I think that the primary reason that the ratings are unfair is the difference between the precision calculated for hosts, versus the precision entered by guests. The mean is taken by adding all the values and dividing by the number of values, and then rounding to 2 decimal digits of precision, e.g. 4.89. However, when a guest rates us, they can only enter the whole numbers 1-5.  That is what is creating the frustration of needing a large number of 5’s to recover from one 4.

 

Is the answer to change the software to allow the guest more precision in setting the ratings? That’s probably not a practical solution either, in that guests probably won’t set ratings such as 4.89, or even 4.8.

 

For fairness, Airbnb could drop the decimal and simply round to the nearest whole number for the host rating calculation. In that implementation 20 5’s and one 4 would still be a 5. Although this provides equity between guest ratings and host ratings in that they would both be represented as whole numbers, Airbnb probably thinks that this would not provide enough distinction between hosts.  We’d all be divided into only 5 groups.  However, I think it’s the best solution. I’m finding that guests look a lot more at reviews than the difference between a 4 and a 5 anyway.

 

Airbnb should be helping hosts to expend their energies improving their hosting skills, rather than encouraging them to be preoccupied with and to stress out over fractional ratings.

 

 

 

@Pat271   "rather than encouraging them to be preoccupied with and to stress out over fractional ratings."

 

see above.

@Pat271 and @Anonymous

As it's our business that puts millions into the pocket of Mr Chesky, I feel it should be company policy not to try to stress us out. And it's not about being vulnerable to stressers intended to keep us in line, it's about a very real threat of losing business. It may be 2021 and tech sites may rule the world but"I am not a number," even if I must be Prisoner to the system.

@Patricia1556 “….it’s our business that puts millions into the pocket of Mr Chesky…”

 

Actually it’s guests who bring in the money. They pay the high service fee. There is no shortage of hosts/listings on the platform, and not likely to be any time soon. So, Airbnb will continue to appease the guest population. Keep ‘em coming, at any cost to the host.

@Colleen253 You're right guests pay higher fees but the owners bring the platform it's business, right? No places to stay, no guests.

@Patricia1556 As I said, no shortage of listings for guests to book, and not likely to be, ever. It is what it is. We can all try other platforms, sure, but none have the market reach Airbnb does. 

Your last statement @Pat271  sums up the problem: "Airbnb should be helping hosts to expend their energies improving their hosting skills, rather than encouraging them to be preoccupied with and to stress out over fractional ratings."

 

And I also agree that if Airbnb ratings must be "mean", then at least lose the decimals in the calculations. Yet another way the site treats guests and hosts differently. In rating, we can only opt for full numbers but in being rated Airbnb standards for "super" host &  "threatened with suspension" host are a mere 3 decimals apart.

@Patricia1556 

 

As it's our business that puts millions into the pocket of Mr Chesky, I feel it should be company policy not to try to stress us out. And it's not about being vulnerable to stressers intended to keep us in line, it's about a very real threat of losing business.

 

The problem is, this is all about supply and demand, and from Airbnb’s perspective there really doesn’t seem to be, as of yet, a very real threat of losing host business. I have read hundreds upon hundreds of complaints from hosts (including myself) over the years, but the vast majority of us are still here and still using the platform. The bottom line is that we get enough benefit from the Airbnb platform to keep us here, otherwise we would abandon it in droves. Airbnb knows that their platform is worth it to us, so they have no incentive to cease their prejudicial treatment.

 

You may be right that at some point there may develop a “very real threat of losing business”, as other companies may spring up with a more amenable host culture. At that point, it will be a priority for Airbnb to adjust their attitudes and change their approach towards hosts. But that point hasn’t arrived yet.