App caches are all just temporary and non-vital data that an app relies upon when running, nothing as far as your data files (photos or music) are kept in the app's cache. But your apps do need their caches so when you delete them, the next time you use an app it will need to rebuild its missing cache. As an example, if you have the Google Photos app on your device once you wipe its cache, your photos are not affected, they'll all still be on your device. But the next time you use Google Photos it will need to rebuild its cache, so while its re-indexing your photo library it might lag a bit as far as displaying everything (those little thumbnail images). The same goes for whatever media player you use, your music files remain as is but if there's a lot of album art involved the app's cache needs to be re-indexed and restored.
Generally you don't need to wipe all the app caches, but if you're having a specific problem with an app, wiping its cache is often one the first things to try and fix things. But if you do opt to wipe them all at once, this won't affect your saved files. Just keep in mind that once you do, with at least some apps there will be a initial pause as they rebuild their cache.