Apple working with Consumer Reports to pin down inconsistent MacBook Pro battery test results

On December 22nd, Consumer Reports said it could not recommend the MacBook Pro — a first for a Mac laptop. It caused a stir! Consumer Reports’ reasoning was simple: it got wildly inconsistent battery test results every time it tried to test multiple versions of the laptop. Its test is one that’s pretty common across the industry, loading up web pages one after another:

For the battery test, we download a series of 10 web pages sequentially, starting with the battery fully charged, and ending when the laptop shuts down. The web pages are stored on a server in our lab, and transmitted over a WiFi network set up specifically for this purpose. We conduct our battery tests using the computer’s default browser—Safari, in the case of the MacBook Pro laptops.

In that test, CR saw the 13-inch MacBook Pro (with the Touch Bar) get results from 16 hours all the way down to less than 4 hours — again, wildly inconsistent. Similarly, the 15-inch version saw results range from a “that can’t be right” 18.5 hours down to 8 hours. Because the results ranged so widely, CR concluded that it’s not a good idea to buy this laptop until we know more about why these battery results are so inconsistent.

Yesterday, Apple’s Phil Schiller tweeted that the company itself doesn’t see anything like these results and is “Working with CR to understand their battery tests.”

That stir I referred to? It’s very real, and it happened mainly because the MacBook Pro was already at the center of very strong feelings amongst the Apple faithful. The laptops have not been as well-received as Apple might have hoped — for reasons ranging from dongles to processor power to price. You can add battery life to the list of things people don’t fully trust about the MacBook Pro now — especially after Mark Gurman reported in Bloomberg that Apple originally wanted to use some more advanced battery technology but had to fall back to something more traditional. Oh, and Apple decided that giving you an estimated “time remaining” time in the menubar was a bad idea and so pulled the feature from MacOS.

In short, this particular Consumer Reports battery test drama isn’t happening in a vacuum. Now, the going argument right now is whether or not its test is trustworthy. CR says it “tests hundreds of laptops each year, using identical procedures in highly controlled conditions,” presumably without varying results on those other machines. After Apple and CR hash out what happened with these tests, we will hopefully get more clarity about what is going on here.

Battery tests should be a guide

The Verge also uses a looping website battery test, though unlike CR we use live data on the web. Because battery tests have a lot of variables, we don’t assume ours is objectively accurate in all cases — it’s better as a guide. In fact, every battery test should be treated more as a guide for what you might expect in real world use rather than an objective measure. The most diligent publications limit every possible variable to come to objective, comparable measures — work that is useful and laudable. But in the real world, the variables are infinite, so you want to do your best to describe real world results whenever possible.

In our review of the MacBook Pro with Touch Bar, we did both. Our looping battery test clocked in at 10.5 hours for the 13-inch model, but in real world (and not very intensive) usage, we got less than 6 hours of life on two different machines.

It’s too much to ask for objective, universally applicable battery life estimates. But it’s not too much to ask that these machines at least be a little predicable. Perusing 9to5Mac’s roundup of reviews, I’m sad to say you’ll find an equal number of reviewers suggesting battery life is an inconsistent question mark and others saying it’s in line with Apple’s 10 hour estimate.

Here’s one more result: in the three hours I’ve been awake today, the MacBook Pro (sans Touch Bar) has gone from 100 percent to 25 percent. But I have consistently had 15+ Chrome (a known battery hog, despite Google’s protestations) tabs open alongside Slack and Tweetdeck. In short: battery life varies and your results won’t line up with objective tests.

Whether or not the truth of the inconsistent battery results some are seeing on the MacBook Pro comes down to flawed tests, flawed software, or flawed hardware is unfortunately not something we can definitively say right now. What we can say is that it’s worth looking into. These are Apple’s flagship Macs, and the Mac’s reputation is built as much on trust as it is on finely milled aluminum.