CarAdvice Ratings V3 Explained
For those of you that read CarAdvice regularly, you’d know that our current ‘version 2’ rating system dates back to late 2013. Now, after nearly four years, and with the benefit of user feedback, it’s time to evolve our strategy once more.
Times have changed, and certain criteria for vehicle assessment have changed, too. And, just as the buyer demands and driving technologies have evolved, we felt our ratings needed some additional diversification of categories. As such, we’ve expanded the number of scoring categories – and thus the level of detail – from five to ten.
We have also changed the overall score to reflect the ten categories mathematically, which now have a range spanning from 0.0 to 10.0, and every decimal in between.
If the individual category scores for a given review are 7.6, 7.8, 8.1, 6.8, 8.2, 9.1, 7.6, 7.4, 7.7, and 6.5 – the overall score will be 7.68, which we round up to 7.7.
Now, before we go through the categories, it’s important we offer you some insight as to how we arrived here.
Our old ratings system wasn’t perfect. Indeed, our new ratings may not prove perfect, either. This is a system we will work on over time to continuously improve, but by no means do we claim these to be the perfect judging criteria – because we do not believe a perfect ratings system exists.
To develop our new system, we took the whole editorial and senior management team away for a few days, and locked them in a room until there was agreement.
It won’t surprise our regular readers to learn there was a lot of debate – and a good amount of yelling. Mike Costello may have even left the room in disgust at some point (only later we realised he had, in fact, had a hit on tinder).
Crucially, we all eventually agreed we had found a solution that fits best with what we do here at CarAdvice.
Many of you have let us know, in no uncertain terms, that our old ratings could be confusing. We have always emphasised that a rating is only relevant for a car in its own segment. So if a Toyota is scored 9.5, it is not therefore better than a Lamborghini that scored 9. A vehicle’s score is relative to its own segment, judged against its price and class rivals. This hasn’t changed.
What has changed, is that we now have a benchmark for average: 7 out of 10.
This applies to both the overall score and its subcategories. The view is that our ratings now reflect a feeling we have for the cars. As such, if we give a car:
9.5 to 10: This is one of, or indeed the best in its class. A car you can buy just about sight-unseen, without regret.
9 to 9.5: An excellent vehicle for its category, one we highly recommend.
8 to 8.9: Well above average in its class. A car you should absolutely have on your consideration list of three-or-so cars in this segment.
7 to 7.9: Average to above average, still worth considering in a purchasing group of five. We recommend haggling for a good price and a test drive for personal confirmation.
6 to 6.9: Below average in its class. Buy only on price or if a great deal presents itself.
5 to 5.9: Well below average in its class. We would recommend something else entirely, unless there is a bargain too good to ignore – and even then, you should think twice.
0 to 4.9: Avoid like the plague.
These new rating scores will see our general overall rating come down.
We have been ‘shadow rating’ the last three months worth of reviews with both our old and new ratings systems, and the general deviance is around 0.5 in the negative.
It’s important to note, the new rating criteria cannot be put up against existing reviews.
We will also produce these ratings based on year, so as we now start in 2017, all ratings are reflective of this year (much in the same way that ANCAP ratings are now rated by year). Once we run a large number of cars through the new rating system, we will be able to produce much better analysis for each car against its competitors.
Now, you may be asking why we have set the average at 7, rather than 5.
Apart from the mathematical application of normal distribution (bell curve), which would see the gap between a car scoring 1 and 10 far too big to fill if the average was 5, the other argument for this is that a 5 out of 10 carries the connotation that the car is only barely a ‘pass’.
This is simply not the case for average cars, because an average car is still a very good proposition for the right person.
For example, a current-generation 2017 Toyota Yaris scores a solid 7 out of 10. That is not a barely passable car; it is average in its class, but under the right circumstances, it’s an excellent choice for certain buyers. A 5/10 does not reflect that thought.
Now, to the categories, which are all weighted equally.
SAFETY
Safety is a very challenging category to score, considering there are whole organisations dedicated to evaluating safety in new cars and establishing ratings for those vehicles.
Our system is intended to benefit cars that have been crash tested, but also affect those that haven’t. Because, frankly, how can we rate a Porsche 911 high on safety when the current generation has never been crash tested by an independent body? We believe it is safe (we even own one), however, it would be unfair for us to say we know more than the European, American and Australian authorities when it comes to handing out safety ratings.
With that in mind, a vehicle that scores 5/5 in the latest iteration of ANCAP/ENCAP safety ratings will get seven points out of ten, by default. A vehicle that scores 4/5 in the current iteration of safety ratings will get six point, 3/5 is five points, and so forth.
This allows us three points to hand out for safety. Cars that have not been tested will get the default rating of a three-star car. This means that the maximum an untested car – such as a 911 – can score in safety is 8/10.
The three points that we can use to rate safety, measure active and passive safety features against the segment. We have decided to include all available options for active safety as part of the score, but penalise manufacturers that choose to offer them as options in the value for money category (below) rather than here.
Why? Some buyer types may not necessarily want the standard-equipment list to include features like blind-spot monitoring, and some manufacturers have therefore decided to bundle them up separately for those that do.
It would be unfair to punish those manufacturers in the safety category for missing features that are indeed available for extra cost.
These features include rear and front cameras and sensors, lane-departure and blind-spot monitoring, rear cross-traffic alert and other forms of active safety that are there to stop you from doing something that may lead to an accident. We will also take into consideration cars that are missing curtain airbags in the second or third row.
FUEL EFFICIENCY
This is mainly self-explanatory. We compare the claimed fuel economy figure of the vehicle against the class, and rate it accordingly. So if it uses similar fuel to its segment rivals, it will score average, the lower the better and vice versa.
Unless stated otherwise, we use the ADR combined city/highway fuel economy figure for this measurement for it gives the most comparative analysis. We also take into consideration its fuel requirement (e.g. 98 RON) and if its claimed figure differed significantly in the real world when tested.
PERFORMANCE
The assessment of performance will depend very much on the car and its intended purpose. It is basically an inclusion of all linear motion – acceleration and braking. However, a vehicle with less than 100kW may score really well, because it may offer excellent performance both in its category and for its intended purpose.
Everything from drivability to low-speed urban assessment, plus highway assessment, will be included in this category. This will include the whole drivetrain, from the engine to gearbox, to their coupling and the relative performance they provide for the vehicle.
RIDE QUALITY
Here we put vehicles through a series of tests to understand and rate their ability to cope with a variety of roads both in terms of comfort for occupants, and the levels of noise vibration and harshness produced for its intended purpose.
We will include small and large bump absorption (where appropriate), body control, plushness and adaptability of suspension (either manual or continuously adaptive). We will also measure the cabin noise where appropriate and compare it against the segment.
HANDLING & DYNAMICS
This is where we judge a car’s dynamic capability. This is kept separate from ride comfort, but includes steering feel, grip on multiple surfaces, traction, suitability of the supplied tyres, feedback, driver friendliness and neutrality (understeer/oversteer), predictability/progressiveness, playfulness (where appropriate), fun factor (where appropriate), surefootedness (AWD/4WD), ESP calibration, ESP switching ability appropriate for segment, and more.
DRIVER TECHNOLOGY
This category is focused primarily on technologies that do not include infotainment or safety.
This score will reflect the vehicle’s capabilities (in segment) around autonomous driving, adaptive headlights, adaptive cruise control, torque vectoring, drive mode selection and suitability for segment, drivetrain enhancements (i.e electric turbocharger), head-up display, and even slip mitigation and traction enhancement technologies in so far as they don’t involve safety.
It will evolve over time, as relevant new innovations and technologies make their way onto the market.
INFOTAINMENT & CONNECTIVITY
This category focuses on the infotainment and connectivity systems of cars. From design and ease of use, to the feature list (e.g. Smartphone mirroring technologies such as Apple CarPlay and Android Auto) to common features accessible via steering wheel, limited use of sub menus, legibility, audio fidelity, ease of Bluetooth pairing, its operation and sound quality both for telephony and audio streaming.
Further, points will be rewarded for special features such as wifi hotspots, inductive phone charging, the number of USB ports and 12V outlets, availability of telemetry data, app support and also the manufacturer’s willingness to future-proof its software via over-the-air updates.
INTERIOR COMFORT & PACKAGING
Here we assess a vehicle’s interior in so far as it applies to the ambience, build quality, practicality and versatility.
Factors such as cabin design and the fit and finish, seat comfort and features (seat memory, remote dropping second row etc) are considered, as is cargo space relative to category, cargo aperture size and intrusions, cabin space, overall packaging, ergonomics, entry and egress angles, ambient lighting and overall suitability of interior for intended purpose.
VALUE FOR MONEY
Here we look at the feature list of the vehicle against its segment contenders. What comes standard, and what doesn’t. How the vehicle’s list price compares. Most importantly, the number of options required and whether they are excessive, average or otherwise, in its competitive segment.
It’s important to note that an expensive car may score very well here, for it offers excellent value for money against its competitors. This is not a score about actual price, but relative value for the money spent.
When we review a car from its international launch, generally many months before its local introduction, we do not have all the details such as price and local fuel consumption figures. If we do not know pricing, that category may receive an N/A. However, if more than one category has an N/A, there will be no overall score.
It’s also hard to judge a vehicle’s performance, dynamics and ride compliance for Australia on super smooth European or American roads. In such cases we will score the car as we see it, based on the best available information to us. We will mention that these scores are based on our international drive and are subject to change when the price, features and specification of the car is locked in for our market, and after we’ve driven it on our very own test courses locally.
FIT FOR PURPOSE
As it says on the tin, this is a category that reflects the vehicle’s suitability for its intended purpose.
As an example, a Lamborghini Aventador may not be great value for money (even in its own segment), but it is definitely fit for purpose in its segment. Or, a Toyota LandCruiser, which may not be segment leading when it comes to technology in general, is very much fit for its intended purpose.
As for comparisons, they will live in their own little universe.
When conducting a large comparison test of multiple vehicles, the scores will reflect the same ‘7/10 as the average’ rule, but within the realm of that comparison only. So, a vehicle may score higher or lower in that particular comparison than it would on an individual test. The reason for this is that we want our comparisons to be a comparison between only the cars in that test, and not the overall segment.
These new ratings are live from today. We are keen to hear your feedback and incorporate changes as they come with each iteration.
From all of us here at the CarAdvice team, we thank you for your patience as we went about creating this new system.
Alborz Fallah, Founder, CarAdvice.com
Podcast
Listen to CEO Andrew Beecher discuss the the CarAdvice review ratings below, and catch more like this at http://ift.tt/23kpPZz.