You’re right. I’ve seen people who use this term eventually have an affair more than not. I think people taking offense have a work spouse and know they have these feelings.

Spouse implies sex. We aren’t being a way to assume these things.