Yes, if you go to Settings and navigate to "Apps" scroll right to "All" and locate the "Gallery" app down the list - select the "Gallery" app and select "Disable" you will be presented with a warning message, select "Ok"
This will remove the Gallery app and any default settings it will have saved to it. You will notice that the app has also disappeared from apps. This wont impact any of your pics, they will all still be saved in "Photos"
The above method isnt removing the app, its simply disabling it. If you want to enable it further down the line then you can repeat the above process, only select "Enable" and the Gallery app will re-appear.