Well... If you are downloading from your browser then you can simply go to your file manager and open the folder where it is being downloading.... Simple remove that file from there... Bingoo!! You stopped downloads... Cheers!
Simply go to download manager and click on force stop, then click on clear cache and clear data.
In Android 6, you may don't see clear cache and clear data, don't worry, simply click on Storage text and then clear cache and clear data.
App-->Settings-->Applications-->Application Manager-->More-->Show System Apps-->Download Manager