Can an app install malware on my phone?

I downloaded a photo editing app and it was working fine, but after I downloaded it my phone started getting really hot and rebooting by itself, also the camera got really slow when saving the pictures.

So anyways, I decided to delete the app (It was the only thing I had installed in the past 6 months) but after deleting the app the problem was still present. I guess the app wasn't the malware, but maybe it installed something in the background without my knowledge.

I fixed it by doing a Factory Reset, but I want to know if the app caused it or it was just a coincidence

It's possible, sometimes apps are disguised as the real app or similar to a real app but are ultimately malware, Google does a pretty good job of removing those and of course Apple does an even better job by blocking content they simply don't like at all or find some arbitrary rule to block at some later date.

That said its possible, it could also be the app was not written well, was doing something to the processor, forcing it to run too hot, etc. I would keep an eye on your phone and perhaps alert the developer if you suspect the app was behaving oddly or alert Google.