Survey

Thursday, October 3, 2013

Are Benchmark scores for mobiles fake?? LG,Samsung , HTC all found to boost scores

Samsung has made the news a couple times because the company has been caught artificially boosting benchmark scores on various devices, most recently with the Galaxy Note 3 which saw 20-50% increases in benchmark scores depending on the tests run. And now, more research from AnandTech is showing that almost no manufacturers are innocent in this practice, which really shouldn't come as a surprise.

AnandTech has been looking through the data since July, and has concluded that benchmark gaming is a pretty common occurrence. In fact, the only devices that were found to not change CPU behavior when a benchmarking app is detected were Nexus devices and newer Motorola devices (most pre-Google Moto devices weren't tested). The testing showed that at least one device from major manufacturers like HTC, LG, and Asus all push higher CPU frequencies in order to score better on benchmark tests. 

So far, Samsung seems to be the only manufacturer that has also changed GPU frequencies in order to game benchmark tests, and not all tests are being gamed. AnandTech talks about the need for manufacturers to simply stop the entire practice of altering test results, but that idea seems unlikely to us. Manufacturers all want to gain advantages, even if the number of users who would care about said advantages is relatively small. 

In reality, the most likely scenario is that benchmark makers and OEMs will continue to play cat-and-mouse. Benchmark makers will try to stay ahead of the optimizations, and OEMs will keep trying to gain artificial advantages. We can always try to rename every benchmark test, and get around the optimization whitelists. But really, the best option all around is just to stop putting all that much faith in the raw numbers (which were never that useful to begin with), and just look at real world performance. 

source: AnandTech

No comments:

Post a Comment