EXE files are not executable on Android, so at least on that side you shouldn't be worried. As to not being able to delete it, it depends on where/how it was downloaded, but you may need to use the stock File Manager (and make sure it has the right permissions enabled to access Storage) to modify/delete the file. Just to make sure no other app is 'using' the file, reboot your phone or boot up into Safe Mode to delete the file.