I too have many older chargers laying around. Really, who doesn't? Just because the plug fits into your S4 or because the charger says Samsung on it, doesn't necessarily mean it is compatible. You have to look at the label on the charger & read the output. Most smartphones now a days require a 1A (or 1amp or 1000mA) charger. Many of the older chargers that I have at home (Samsung, Motorola, Palm) have an output of 700mA or 800mA. Some of the dumbphone chargers may even have the correct plug but will only output 500mA. Sure, they will probably all work, but they will be slow.
To try to give a really simple analogy... Let's say you want to fill a big fish tank by pouring a pitcher of tap water. You have a 1000 oz, 800 oz or 700 oz pitcher. Which one would fill the fish tank the fastest?