You could still keep it simple - by applying the bonuses when equipping the item, and de-applying them when you unequip it. Then, "wielding the sword" would grant you the sword bonus, and "unwielding" would undo it. That way, you only have the bonus when you are using the sword. This of course would still affect climbing, if carrying a sword in your main hand doesn''t stop you from climbing the ladder ~ but that can be solved in other ways.
People might not remember what you said, or what you did, but they will always remember how you made them feel.
~ (V)^|) |<é!t|-| ~