That would normally be an issue with phone. Do you have another app that uses gyro or accelerometer at the same time as VR? Try download an app like Google sky map from the play store. See if it does hat the same thing. If it doesnt, I'd delete everything oculus and reinstall. If that dont work, then you have a messed up directional pad on the goggles.