I stayed 2 additional days after surgery. And to be honest, "wearing" the hat made it more obvious in my opinion than not wearing it. I mean you literally have the hat just resting on top of your head and not pulled down snug like usual.
This was a huge concern of mine pre-HT and to be honest, after it was all said and done, I just went to the airport hat off and didn't even care. No one even paid attention to me because they were more worried about getting to their flights. And again, the hat made me stand out more than not wearing it. So, I wouldn't worry about it too much. But good luck with it all!