I use my N7 charger to charge my N4 all the time. Since it is more powerful it will charge it faster, but I worry that it will damage the battery in the long run. The N7 won't charge on the N4 charger, to weak I guess.
They are more or less one way interchangeable in that the N7 one should have no problem charging your N4 (not the other way around). The N7 charger delivers 2.1A and seems to really need that kind of draw to charge quickly. The actual USB piece on the N7 charger is slightly longer than typical phone USB chargers, so there is a risk to damaging that end when using it with other devices (the stock charger).
Thanks for the replies. Just tried the N4 charger on the N7. No sparks or smoke and successfully put 20% into the N7 in 50 minutes so I'm happy with that.