I had this issue when I checked the focus follows mouse box. When I unchecked that my problems went away. It can be a pain sometimes to have to dbl click on the alt to interact but I have gotten used to it.