Dear CarExpert readers and friends,
A few months ago, many of you sent us a link to this Reddit thread about our review ratings. You can go and have a read of it yourself, but the essence is that our rating system for reviewing cars sucked.
The author, who supposedly has a background in data science, observed that most cars reviewed by CarExpert receive scores clustered narrowly between 7.8 and 8.4, regardless of their actual performance or quality, meaning that essentially our ratings were not very useful in helping you buy a new car.
Given that our entire company mission is to help make buying the right new car quicker and easier, that was feedback we took seriously.
Hundreds of new car deals are available through CarExpert right now. Get the experts on your side and score a great deal. Browse now.
There is no denying the facts around the data. Our ratings are indeed clustered in that spectrum because, ultimately, we are allowing each individual CarExpert reviewer to come up with his or her rating for the categories in which we had previously used. This was not by design, but simply because of the human element involved.
Our previous rating system was a great idea in theory, but it is limited because one expert’s rating is not always identical to someone else’s. Plus, what really is the difference between 7.5 and 7.6 when it comes to ride and handling, especially from one author to another?
So, in the spirit of transparency, we acknowledged internally that the current process was not working (in fact, it sucked) and we started work on our new rating system.
The reality is that no matter what we do, or how we do it, there will be individual bias from our writers around cars, and it is difficult to ask a person to judge a car on a numerical score and then have that score compared to another author judging another car. It will never be apples versus apples.
As such, we have come up with two separate rating systems, both of which will evolve over time. The first iteration of our new ratings sees our human subjective rating remove any numerical reference. There is no longer a 7.4 or 8.5. We are now using simple words to convey the author’s overall thoughts on the car.
The human rating will be on a scale of excellent, very good, good, average or below average. Simple and effective language to describe how our car experts feel about the vehicle, and probably a little bit more useful than ‘7.5’, which can mean average to some and amazing to others.
The second part of our new rating system is entirely data-driven. We are now going to take certain comparative rating categories using cold, hard data for each vehicle segment to give readers an idea of where the car they are interested in sits among its peers. Look at the data table below for an example of what our reviews will soon include.
You will notice there are no numbers on this scale (yet). This is by design because we will first launch the ratings to give a visual indication of where the car sits among its category competitors before then looking to add a numerical score (or percentage) based on your feedback.
You will notice that the blue bar can change size. That is due to the example listed, which takes in the full make/model range for our ratings (if there are different engines, variants etc). For individual make/model/variant reviews the blue bar will be far smaller and more representative of where the car actually sits.
To better understand how this works, say we have a Toyota RAV4 Hybrid AWD that consumes 4.8L/100km. That fuel efficiency number will now be put on a scale that shows the best option in the mid-size SUV segment at one end and the worst option in that category at the other end. Readers can see for themselves what the figures show across many other data points to decide what is important to them and what is not.
Again, this is not a perfect rating system because many cars – even in the same category – are different and have different purposes, so it’s not a fair comparison. Another obvious potential issue is that we have to take the claimed data from automakers when it comes to things like fuel economy, towing capacity, ground clearance, etc.
In our reviews, we will continue to always mention discrepancies between claimed fuel consumption figures and what our experts experience in the real world, but the data rating will use actual OEM-supplied data, which as we know isn’t always be 100 per cent accurate.
Some things to note here: all our previous reviews will have their numerical rating converted to a word rating. Some of this may cause some initial confusion or misunderstanding. We will fix that as we go.
Also, as we launch this new rating system we’re seeking initial feedback for improvement. This improvement and iterative updates will take time. Please note that we do listen and we will get better, but please be realistic about the time it takes for iterative updates.
With all that in mind, firstly a big thanks to our avid readers and Reddit users who sparked this change and, secondly, please use this article as a reference for collating all the feedback around our data ratings. And know that even if we don’t respond, we do read everything and take it on board.
