I have this issue as well. I do use my rear mirror. The problem is that the oncoming lights get reflected and it feels like a vehicle is failing to stop behind me. If you have ever been rear ended it triggers your PTSD (or probably not PTSD, but still). Since I was rear ended my on motorcycle years ago I am very sensitive to the idiots who are trying to run me over (everybody else on the road IS an idiot).
I do get used to it, but it is annoying - especially when I am tired, or in heavy rain/snow. Closing the blinds does nothing since the reflection is coming off the glass in front of the blinds.