I'm a former loyal Samsung owner.
Speed:
In comparison, you'll notice immediately that the phone runs much faster; Samsung TouchWiz seems to always lag a bit. The LG OS is optimized and uses less RAM, whereas Samsung hogs a lot of RAM, probably due to the TouchWiz overlay and Samsung apps/services. Don't even think about updating the OS on the Note 4, it'll slow down your phone like crazy (maybe they do this on purpose to encourage new phone purchase?)
Display:
The AMOLED display is always beautiful but after using it for many years, I got bored looking at the oversaturated colors. Sure, blacks looks deep and true, though white colors always seem off, almost kinda beige like color. Grey colors also look ugly and there's a lot of flaws visible over a grey background. So the IPS display wins when it comes to white colors, you'll get used to it as well. During the day, the IPS looks perfectly fine, but at night, it doesn't look as nice as AMOLED. I'm actually used to it now and like it much more. Also, I'm not paranoid about screen burn in anymore, I could leave the display on for hours on end.
Battery life:
I don't know why everyone's complaining about battery life and screen on time. Coming from a Samsung Galaxy S6 and S7 active (3500 mAh and 4000 mAh battery), the V20 outperforms those two in any given day. It's much more reliable and also efficient. I get between 4-6.5 hours screen on time on my V20, and at least 14+ hours or more battery life (pretty much all day). And plus, it still has a removable battery and supports Quick Charge 3.0, much faster than 2.0. I've never had a phone battery that's reliable as this one. Maybe you might have to do some tweaking to get better results (disable useless apps and services). Also the phone runs much cooler, and stays cool when you charge it thanks to USB C and Quick Charge 3.0 technology. My S7 gets pretty darn hot when charging. (Pro tip if you own a Samsung: don't charge your phone wirelessly, it heats up your battery and your battery will get worn out faster. I think my S6 Active battery got damaged overtime, stopped holding a good charge like it used to).
Camera:
Obviously the camera is much better. Dual cameras on the front and rear, much better quality. I enjoy this camera over the S7 & S7 Edge. Much more details and options, though I love the simplicity of the Samsung camera app. For those complaining about color reproduction in the photos, just download a photo editing app and adjust it to your liking.
Size and design:
The V20 is much thinner and lighter than the Note 4 & 5. Bezels are slimmer and comfortable to hold (the Note 5 feels bulky). Also the V20 has a second screen with always on display, way better than LED notification. The all metal build feels premium and less fragile compared to the plastic backing of Note 4 and glass back of the Note 5. What turned me off from the Samsung series is the all glass build, why buy an expensive phone that's fragile? I used to have the Note 3 and I hated the size, it was too uncomfortable to use one handed so I switched back to a 5 inch screen. The V20 feels nice to use with one hand.
Honestly the Note 4 is a great phone but clearly it's aging. A lot of Note 4 owners don't want to give it up because of the removable battery and great screen size. The V20 is definitely next gen and you can be assured it'll meet your needs for a few years until you upgrade.