When you tap the photo to make it fullscreen, tap the garbage can icon at the lower right, instead of tapping the Menu button (which is where the Delete Device Copy option is). This will move the photo to the Trash folder, which will be emptied after about 30 days, I think.