I would tend to disagree with this. the hardest thing about any relationship is for person A to realize that it is not working and pack it in instead of continuing to beat their head against the brick wall that is person B.
How about "the absolute hardest thing about keeping any relationship functioning....".
I agree, sometimes the hardest is knowing when to throw in the towel.