Best phone I ever owned was a Samsung Galaxy Note 3 rooted on stock JellyBean. It was huge but the battery lasted DAYS and to be honest I never once encountered lag or even was annoyed by Touchwiz and I used to be a pure Google guy.
I'm probably under the impression that you don't use your phone nearly as much as some people so maybe you don't encounter these issues. Also running roms that have been detoxed, debloated will always run better than an OTA android from your carrier.
My phone has pretty much been my PC replacement for the past few years. I finally just picked up a new laptop last week. Before, I used my phone for EVERYTHING so I spent a lot of time on it. I can assure you, Toucwiz/Samsung phones do experience quite a bit of performance issues. The fact that it took FOREVER just to get a root on the S5, I didn't even bother with it once it happened. Then not having a unlockable boot loader made it worse. I think they have safe strap working on it but it just seems like too much work for a halfway of doing it. Quite a few people who had Verizon S5 got hit with performance/lag issues when they finally pushed out the update. I was stoked for the S5 to get Lollipop until they actually released it. My phone ran worse than it did in the past year of owning it. Did a complete wipe/cache wipe, etc. I almost started to feel as if it was intentional to get people to go out and upgrade to the new S6.
I loved my Note II as you though. Had that unlocked, rooted, ROM'd. Jellybeanz was the shit.