Getting that kind of info is extremely rare on Android. About the only way your friend could know all of your passwords is you having an extreme lack of security practices. As in almost intentionally giving it away. So I'd like to first know what you have in that respect.
Do you have a secured lock screen set up to require pin/password/biometrics on startup and turning the screen on?
Do you allow anyone to have unfettered access to your phone?
Are your passwords all easily guessed?
Are you using a password manager that has a strong master password?
Did you turn on the option to install apps from unknown sources?
Have you installed anything outside the Play store (from him, random website pop-up, etc)?
Did he prove he has your passwords by logging into an account while you watched?
What do you mean by "more and more?"
Most of these hacking threads turn in too nothing. In cases like this, it's usually someone trying to trick you into thinking they have all this info, when they really don't.