Popular Mechanics on Macs and PCs
Oftentimes when people try to make a decision about whether or not something is better than something else, they’ll perform a study comparing the two “somethings,” whatever those happen to be.
For example, if you’re got to type a paper but don’t have any word processing software installed, you might try NeoOffice and comparing that to Microsoft Office. This could be considered a study, because you’re trying to figure out which one best suits your needs. If you’re reading an article in the Journal of Bacteriology, the authors might be examining the growth rate of a particular bacterium in several different types of media; they’re trying to figure out which media best supports the growth of that bacterium. If you’re a technology geek (and I can’t deny that I’m a technology geek, for better or for worse) you might try to find out which computers, Macs or PCs, people like better.
The point is, regardless of the question proposed, the researcher needs to think critically about how the study should be conducted and what, exactly, it is trying to answer. Some sample points to consider: Can I collect the data in an unbiased fashion? How precise/accurate are the data collecting tools? Can the results be reproduced under slightly different conditions? What are the defining characteristics of comparison, and can they be extrapolated beyond the scope of the data being collecting? Can we draw broader conclusions? If so, why?
Failure to ask these questions can lead to the spread of misinformation, as conclusions are drawn beyond what the data is able to suggest.
The people who read studies need to ask these questions, too. Read any study with these questions in mind. It’s not a skeptical frame of reference; it’s only thinking critically. Often, there are many correct ways to go about doing things. There are also many wrong ways to go about doing things.
And there certainly is a right way and a wrong way to go about comparing technology products. This lead to myself, and more than a few others, calling into question the results of a probably-not-so-scientific study on the “usability” of Macs versus the “usability” of PCs performed by the guys and gals over at Popular Mechanics.
The article is headlined Mac vs. PC: The Ultimate Lab Test for New Desktops & Laptops. Yet it is not the ultimate lab test; to call it even a good lab test would be a stretch of blind optimism.
The Popular Mechanics crew recently compared an iMac and a Macbook to a Gateway One and an Asus M51SR notebook, respectively. Then, they went on to compare hardware specifications, user ratings, and application performance across the four machines. The full article can be found online here, and I encourage you to read it before continuing, although I will attempt to make things clear enough so that you’ll understand my main points of argument even if you haven’t read the article yet.
The main issue with the comparison between the iMac and the Gateway One involves the dissimilar hardware in each. The iMac comes packing a 2.4 GHz Core 2 Duo, while the Gateway One sports a 2.0 GHz Core 2 Duo. RAM also varies widely, with the iMac containing 1 GB and the Gateway 3 GB. Then, application performance was measured.
Looking at the raw data, you can see that the iMac posted a faster time than the Gateway One in nearly every test. Does that mean that Leopard runs and loads applications faster than Vista?
No, of course not. In this study, how can you separate the variance due to hardware differences from the variance due to the operating systems? You can’t, even though we see this assertion on page 4 of the Popular Mechanics article:
“Leopard OS trounced Vista in all-important tasks such as boot-up, shutdown and program-launch times.”
They then make the following boneheaded statement:
“We even tested Vista on the Macs using Apple’s platform-switching Boot Camp software—and found that both Apple computers ran Vista faster than our PCs did.”
Of course Vista will run faster on the iMac than on the Gateway One; The iMac has a faster central processing unit! If Popular Mechanics is interested in testing the speed of the operating system, then they’ve got to run each OS on the same hardware.
While the application performance section is by far the most quantifiable data obtained in this study, the computers compared were of unequal hardware specifications, and this makes it difficult to draw conclusions. Is the iMac’s faster (by one second) Microsoft Office load time due to its 400MHz processor advantage? Maybe. What about its 1.5-second advantage in Photoshop loading time? Perhaps. Or does OSX Leopard just load applications faster than Vista? Who knows. Never mind the vast differences in Mac and PC versions of programs similar in name. Microsoft Office 2007 might sound a whole lot like Mac Office 2008, but in reality they are two different programs with different sets of features, despite producing the same types of document files.
The author attempts to rationalize the hardware differences between the machines as “not a heck of a difference in this age of dual-core chips.” How much is a “heck” of a difference?
Apparently, it’s about 1.5 seconds.
To come up with a decent comparison of which machine really is faster, Popular Mechanics needs to compare application load times and scripted processes across Macs and PC with identical hardware specifications, manufacturers, and brands. Only then can it be determined if any performance differences are due to the Leopard operating system or the Vista operating system. Still, it’s impossible to determine if the performance differences are due to the operating system alone or the efficiency with which that particular program was coded and compiled.
The usability tests were even more ridiculous:
“Our testers were instructed to divorce themselves as much as possible from their previous technological preferences and rate their experiences with each computer’s software and hardware.”
Humans are notoriously bad at being completely unbiased, random, or objective. This fact is well-known but not well understood. We’re even worse at attempting to distance ourselves from prior experiences. So why should we expect that the panelists in this study were able to wipe their memories of previous computing experiences?
Never mind the number of testers they used. The article doesn’t mention this information. “A panel” could be three, or ten, or one hundred. Can you generate statistically significant data with only three values? Probably not. With ten? Yes, but those 10 panelists better be representative of the population. With 100? Yes, it becomes much easier to generate reproducible data with that kind of n.
But I digress, and this rant is leaning towards the long side of things. Popular Mechanics is not the battlefield where Macs and PCs wage war to the bitter end. The conflict is best left to the users themselves, who will vote with their hearts and with their wallets.
Tags: Asus M51SR, comparison, computer, Gateway One, GHz, iMac, Intel Core 2 Duo, Leopard, Mac, Macbook, notebook, OSX, PC, Popular Mechanics, RAM, research, study, Technology, Vista
You can comment below, or link to this permanent URL from your own site.
May 13, 2008 at 10:13 pm
I agree. This seems to be a common practice on most tech websites.
How hard is it to find a pc running osx and one running vista with the same specs?
I’m sure leopard is faster at booting up and down, but that doesn’t really matter, it’s the speed the apps open, and how fast the application run that matters.
One thing I dislike about this article is the usage of Mac vs PC.
The “mac” is a normal pc (nowadays), why do you think it can run windows and tons of others OS?
Apple brags that it can run vista and xp and only “mac” pc’s can run OSX.
This is nothing to brag about, it only says things about Apple and OSX.
1. Apple won’t let non-mac pc’s run Leopard.
It’s perfectly possible to run osx on other pc’s.
2. Leopards bad hardware support.
There is a reason they only let it ship on mac pc’s, well there are loads of other reasons, but one is it’s bad hardware support.
A system will be a lot more stable if you can control the hardware it runs on. While all other operating systems must support all possible hardware created by mankind.