I think if you are down to such small doses only a couple of times per week and still having side effects then you should probably just quit it all together and try something else. You could try oral Minoxidil.
You also have to wonder if it's in your mind rather than an actual effect of Finasteride.