You didn't say what email client you're using. Knowing that would help. I don't know of any significant changes in the default Gmail app from 5.0 to 5.1. From your post, I can't tell if your problems have anything to do with Android itself or not, but it's doubtful.
Depending on what email client you:re using an option to "Delete from server when deleted on device" may or may not be available when using IMAP service. Some have it, some don't.
The "Leave message on server" option is for POP3 service. Again, it's not always there and is kinda iffy, it doesn't always work
When you set "Leave message on server" and it was gone a couple of weeks later, it might be because the account is setup to clear deleted mail after a certain time, say 7 days or 10 days so limited storage on the server is not filled and mail does not get rejected. POP3 accounts often have very little storage. IMAP usually has large, sometimes unlimited storage.
You should definitely go with IMAP if at all possible. It's capable of doing much more than POP3. It will sync your email across all devices and computers, keep everything on the server, push notifications and more than POP3 just can't.
I don't know what your problem was with it, but it's normally very easy to setup. Initial setup should be automatic.
You didn't exactly say, but I think you might be using an outlook.com email address. If you want to use IMAP with Outlook.com, here are the settlings:
Incoming IMAP
Server: imap-mail.outlook.com
Server port: 993
Encryption: SSL
Outgoing SMTP
Server: smtp-mail.outlook.com
Server port: 587
Encryption: TLS
Android since v1.0. Linux since 2001.