Maybe my head is lopsided

I have noticed something at practice lately. It's been cold in the mornings, so I wear my green fleece hat to keep my head toasty while I'm rowing. Despite the fact that I make certain to put it on straight (have to look good out there!), it always ends up tilted at a jaunty angle to the right by the end of practice. I notice this because it is always blocking the vision in the right eye a little bit. Why does this happen? Why does it only fall down on one side? I have pondered this during many a lake tour, and all I can figure is that my head is lopsided.