Incorrect.
R’ Abadi in the Ohr Yitzchak is very clear that one should wear a talit katan all day except during those times when it is TRULY bothering you during exercise or causing issues in your work. The Rav goes further and says that if you just don’t wear a talit katan “the door is open” as the Gemorah says if someone wants to sin the door is open.
