From a Sprint Guy: Why stuff gets missed in testing

[midmofan]

Admittedly this applies somewhat to all phones and is from just one source, but the discussion I had with my Sprint friend was about the EVO LTE 4g and the issues that I thought should have been caught before shipped. The guy I was talking to is a pretty high up guy with Sprint in Kansas City. He is on the legal/regulatory side of the business but he is a big time phone geek. Always wants the latest thing. Used to be a big Palm-nut, so you know he has shared a lot of our pain.

His beef with both how Manufacturers and Sprint test phones is they don't require the testers to use that phone EXCLUSIVELY for a long enough time. Also he does not think its given often enough to what he calls "regular users."

What he pointed out was that the people that test these phones already have two or even three phones that they carry around at all times -- a personal phone, a work phone, and sometimes their latest toy. As a result, they are not using the "test phone" they are given the same way a person who will be relying on this phone for everything every day would.Same thing he says it usually true even with outside testers under a NDA, they are guys that are testing and playing with phones all the time so they are not using ONLY the test phone for an extended period.

He compared this to they way cars are tested. Cars, he says, are given out to people who use them every day for months as their only transportation and then report what they have found both good and bad.

With phones, they are given to people that are not using it as their only device -- their "daily driver" to abuse the metaphor some more, so while they play with it and try a lot of different things, a lot gets missed. Specifically with the LTE, he says if testing had been like is done with cars, the Google wallet problem would have been discovered, the issue with the camera saturation would have known, the wi-fi dropping in and out could have been addressed and things like the multitasking, even though that was a design decision, there would have been more real-world feedback to HTC about it.

He also said manufacturers and Sprint are getting so paranoid about leaks that they don't let enough people use them who will be using them in different types of ways and situations.

My "source" is not an Apple fan, but he says they generally do testing better than anyone else since it is his impression, they make testers "live" with the phones for a while. He said the antenna mess from last year was probably the result of a similar situation, though, since probably everybody testing the thing had it in a case (to disguise it) where the "grip" was not a problem, while lots of normal users go without a case.


He still likes his LTE overall and has heard rumors of an OTA update soon but really doesn't know.

 

[mazzmoney95]

You would think using a phone as a daily driver would be a pretty obvious thing to do in testing.

 

[Darth Mo]

I don't buy that reasoning. Most of these issues with the LTE were exposed within hours of regular users that purchased the phone. So it's hard to believe that someone testing the phone would completely miss most of these.

As an engineer, I can tell you that sometimes someone over your head will decide that a product has to go onto the market and that's final. You are fully aware of any outstanding issues and they are noted to (usually) be fixed after the first production run. The money guys have far more decision making power than the engineering team.

This is why so often new products can't meet the initial demand: there are flaws with the design that aren't necessarily crippling, but they will be fleshed out at some point and you want to keep as few of those initial designs off the market.

With all these dealings with Apple it's likely certain aspects of the 4G LTE's software had to be compromised in order to address that patent infringement claims. Makes sense that Google Wallet seems to have taken a back seat because there are a small portion of customers that even know what it is, and even less that will actually use it.

It's just the price you pay when you buy a device immediately when it's released. The people that purchase one six months down the road will never hear of these issues.

 

[midmofan]

You would think using a phone as a daily driver would be a pretty obvious thing to do in testing.

Yup, I agree, but according to him that just isn't happening. Of course there is a hassle element to it. If it is your real everyday phone, it has to have your everyday phone numbers on it, you have to have all your contacts, media and accounts sync'ed with it and ALL of the apps you use put on it. I have abut 150 apps on my phone, I doubt many testers and reviewers have gone through the trouble of putting all of those on their test-phone.

And you need to have it long enough to do everything you do with it, using it while on the move, in the car, walking down the street, in the airport, streaming and texting, trying to update all your apps while taking a phone call or on navigation and so on.

Now, here's a question I would love to see answered: When Phil did his review, was the LTE the ONLY phone he used for the several weeks between when he got it and the article, or did he have other phones he used as well. Did he have all contacts and apps on it? Was it his "daily driver?" I doubt it. That doesn't make his review a bad one -- indeed its a lot better than most I have seen and he at least waited a bit before posting in order to work with it longer, but that still will result in missing a lot of things that a real user/tester would find.

 

[IAmSixNine]

Give me an NDA to sign and ill be more then happy to make it my daily driver phone.

 

[thomasr1950]

I work in mfg. so i agree at times to get a new product on the market to meet the demand quality can be affected a little. Not enough to prevent the product from being released but higher ups will give us the go ahead to make the product, at that time we have to mfg. to their standards even though we don't want to we do what we are told. Then the obvious don't ask just do it

 

[droidmyme]

Interesting stuff. Thank you, OP

 

[crxssi]

Another factor to consider is that many companies rely mostly, primarily, or largely on PAID EMPLOYEES or consultants to do most of their evaluations. As such, those "evaluators" cannot be impartial and won't necessarily argue about design decisions because there is a conflict of interest. A perfect example might be this whole "multitasking" fiasco.

 

[midmofan]

two other things he got into that I didn't mention since OP was getting long and I thought the main point was more important, was what that the first let testers at the manufacturer are the same people that worked on the thing in the first place and that, just like it's hard to catch your own typos, you don't catch "mistakes" that you have been a part of creating. Another closely related item is what he calls the WordPerfect shortcut syndrome (yup that dates him AND me for getting the reference*) where the geeks that design and work with phones all day (he considers himself a semi-geek) as part of their job don't get put off things that a "normal" user might not think intuitive or might even hate.


*you have to go back to the original DOS WordPerfect to get this....

 

[midmofan]

I don't buy that reasoning. Most of these issues with the LTE were exposed within hours of regular users that purchased the phone. So it's hard to believe that someone testing the phone would completely miss most of these.

Well, none of the reviewers seemed to catch most of the issues that we are complaining about, including Phil, who played with it for a couple weeks before posting his review.

I suspect a lot of the issues we see are resource, memory and conflict problems which don't necessarily show up if you have a totally "clean" phone that you just pick up and use to see how this feature works or that feature works. Once you load it up with 150+ apps and widgets and are trying to do three things at once all day long on the go, THEN the stuff starts going wrong.

Now the HTC employee testers might feel pressure not to dump on a device, but most of what we are seeing are not "this device is worthless, don't sell it," its just things that can/should/will be fixed. Further, going back to the Palm days, I know Sprint did, in fact, ask for changes to some of the Treo phone software when they discovered issues after testing (ah, the good old days, when we used to email directly to Palm techs and they would reply....) so at least SOME of these things I think could/would have been fixed with "real world" testing.

Now, if anybody takes the position that HTC just doesn't care, I can't argue with you since I don't have anybody I know that works for them!

 

[Darth Mo]

Well, none of the reviewers seemed to catch most of the issues that we are complaining about, including Phil, who played with it for a couple weeks before posting his review.

I suspect a lot of the issues we see are resource, memory and conflict problems which don't necessarily show up if you have a totally "clean" phone that you just pick up and use to see how this feature works or that feature works. Once you load it up with 150+ apps and widgets and are trying to do three things at once all day long on the go, THEN the stuff starts going wrong.

But the thing is that there is no way to design a single phone to work the exact same way with 150+ apps. You could spend years getting the phone with those particular 150 apps working perfectly. Now take the next phone and put 150 different apps on it, now the phone is "broken" again.

This is why the reviewers review the phones at pretty much stock. At least the reader will have a baseline to start with when choosing between phones. Two identical phones loaded with 50% different apps can possibly behave very differently over the course of time. It's impossible to design a phone to combat that.

I believe a lot of the blame is some of the apps themselves. Some don't clean up properly when they close, some continue to accumulate memory when they should be dormant. Android tries to handle this on it's own, but as we see we get mixed results.

As for employees not wanting to "dump on" their company's phone or that they simply don't pay enough attention, don't believe that. Most products, in any category, have issues that aren't changed/fixed generally for three reasons: it's a design choice therefore is not broken, changing one thing compromises several others, or admittedly it's flawed but we don't know how to fix it or we'll just fix it later. Think about it: Do you really thing the "testing" process for a consumer device valued at $600 which you hope to sell millions of is really tested by handing out a few to employees for a couple weeks?

Nothing as complicated as a cell phone has ever been perfect nor will there ever be one. The manufacturers just release them knowing they have flaws otherwise they would never get released at all.

 

[midmofan]

The issue is not whether they can make a perfect phone -- no they can't. And its true that with so many apps that it is impossible to test every possible issue. The point my Sprint guy was making is that they (both HTC and Sprint) don't require the testers to use the phone as their "one and only" device for a month or so. If you did that with even a relatively small sample of people, you would see many of the problems that popped up here from people in the very first few days after release. That is simply -- according to him -- not done. If that happened would everything get caught? No it wouldn't, but more things would.

As for reviewers, yes its good to have a baseline/stock review that applies to everyone before adding things. But its also good to have a real-world "daily driver" review that explores what happens with real use. That is what you see with car review at places like consumer reports - both an initial review and a long-term report after driving it for real for a number of months under different conditions. Very often you see things that did not seem a big deal initially become irritating after living with it for a while.

So, yes there will always be flaws, but if the testers (both internal and external) would be required to use ONLY the test phone for a decent period of time (whether they have 10 apps or 1000) there will be better evaluation and more problems will be caught before it is released. Apple, although certainly not perfect. apparently does this and so could others.

But the thing is that there is no way to design a single phone to work the exact same way with 150+ apps. You could spend years getting the phone with those particular 150 apps working perfectly. Now take the next phone and put 150 different apps on it, now the phone is "broken" again.

This is why the reviewers review the phones at pretty much stock. At least the reader will have a baseline to start with when choosing between phones. Two identical phones loaded with 50% different apps can possibly behave very differently over the course of time. It's impossible to design a phone to combat that.

I believe a lot of the blame is some of the apps themselves. Some don't clean up properly when they close, some continue to accumulate memory when they should be dormant. Android tries to handle this on it's own, but as we see we get mixed results.

As for employees not wanting to "dump on" their company's phone or that they simply don't pay enough attention, don't believe that. Most products, in any category, have issues that aren't changed/fixed generally for three reasons: it's a design choice therefore is not broken, changing one thing compromises several others, or admittedly it's flawed but we don't know how to fix it or we'll just fix it later. Think about it: Do you really thing the "testing" process for a consumer device valued at $600 which you hope to sell millions of is really tested by handing out a few to employees for a couple weeks?

Nothing as complicated as a cell phone has ever been perfect nor will there ever be one. The manufacturers just release them knowing they have flaws otherwise they would never get released at all.

 

[Darth Mo]

So, yes there will always be flaws, but if the testers (both internal and external) would be required to use ONLY the test phone for a decent period of time (whether they have 10 apps or 1000) there will be better evaluation and more problems will be caught before it is released. Apple, although certainly not perfect. apparently does this and so could others.

You're misunderstanding what I'm saying.

Just because the problems exist when released doesn't mean they weren't discovered in texting.

If an issue exists at release, it doesn't automatically mean it was missed in testing. Sure tech support my say they've never heard of the issue, but that doesn't mean the engineering team doesn't know about it. That was the initial point I disagreed with your friend.

 

[midmofan]

You're misunderstanding what I'm saying.

Just because the problems exist when released doesn't mean they weren't discovered in texting.

If an issue exists at release, it doesn't automatically mean it was missed in testing. Sure tech support my say they've never heard of the issue, but that doesn't mean the engineering team doesn't know about it. That was the initial point I disagreed with your friend.

Nope, I get it, but I also note that 1) previous experience shows that changes have been made before release when fixable issues found and 2) these issues were also not discovered by any of the reviewers, leading credence to the idea that you need to use the device "for real" to catch everything.

It may be that some things were found and HTC or Sprint didnt care. That's possible, but given the reviews that missed this stuff too, its also likely that many of the items were really and truly missed.

 

[crxssi]

But the thing is that there is no way to design a single phone to work the exact same way with 150+ apps. .

The thing is, you don't have to load ANY additional apps on this phone to trigger issues with crappy memory management and multitasking. I noticed it almost immediately, using nothing but the stock stuff included on the phone.

 

[kidstechno]

It basically comes down to weighing up-front profits vs. customer satisfaction. Obviously products take time to engineer/build so if the phone isn't on the market, you're losing money. So you have to decide at what point can you eat x amount of disgruntled people who don't like it and return it versus the product being stable "enough" to allow for consecutive updates to fix any problems there may be, while releasing a decent enough product to garnish good reviews and build hype.

I've learned to accept buying anything tech related when it first comes out is bound to be met with bugs and setbacks.

Now is this a sad statement that reflects the state of electronic consumer products? Very much so.

 

[Darth Mo]

The thing is, you don't have to load ANY additional apps on this phone to trigger issues with crappy memory management and multitasking. I noticed it almost immediately, using nothing but the stock stuff included on the phone.

I gave the OP the benefit of the doubt he was speaking generally and not specifically the 4G LTE.

But I said in my first post how the most prominent issues (multi-tasking, Wallet) for the 4G LTE were fleshed out within hours. There's no way they slipped by HTC or Sprint.