Welcome to Android Central! It also sounds like you have no screen lock whatsoever. You should at least have the Swipe Only option turned on -- that way, if the screen turns on by accident in your pocket, you're much less likely to rearrange the homescreen (or inadvertently open apps or call anyone) because the screen would have to be swiped all the way up first before the homescreen appeared.
Also, it's possible that the phone remains unlocked because you have some Smart Lock option enabled, for example trusted places or on-body detection. If you want to keep this functionality and avoid accidental changes in your home screen setup your best option is using a custom launcher that includes a feature to lock it, such as Nova. This is what I use.