Not unusual. It'll vary by phone, OS, launcher, what's happening in the background, maybe the phase of the moon. Nothing to worry about, unless the lag is long enough to hamper performance.
I used to frequently clear recent apps, ram, caches, everything that could be cleared. But I've come around to the view that Android is designed to run the way it runs, and that memory is there to be used, not held in reserve. Since I started leaving things alone instead of clearing them out all the time, my phones run better and batteries last longer.