...if you think that only your contacts sync is affected - try to switch-off power saving mode on your phone.
It may work well on a Galaxy 7 etc. This may be easier than looking for the setting that prevents loading data in background. Sync of mail works fast and is finished before power-save...