A phone with a non-removable battery has to go to the shop to have the battery changed, you can't do it yourself, because the phone has to be taken apart. So aside from costing more (the batteries have cables on them so they're more expensive, and you have to pay for labor), you can't carry a spare battery to swap in the middle of the day if the battery in the phone gets down to 40% (which is as far as you should discharge a lithium battery - see
The Care and Feeding of Lithium Polymer Batteries), you can't swap batteries every month (which tends to prolong the life of both batteries).
Also, it's possible (VERY rare, but possible) for a lithium battery to start overheating and keep getting hotter until it bursts into flames (or even explodes - see
Lithium Battery Fire Crashes). If it's removable, when you feel it getting hot you pop the back off and remove the battery. At worst, the battery burns or explodes (with no one near it) and you lose $10. If the battery isn't removable, you drop the phone and watch it melt - then you buy a new phone, because melting isn't covered by warranties.