So I think I've figure it out. It only happens when I have 3 or more loading screens at the same time. When I do this, it does take a lot longer to load. So what I have to do is hearth with 1 character, let that one completely load, then I can hearth with the other 2.

It's almost like that amount of loading at one time bottle necks and can't catch up. I do have an SSD. At the time it was one of the better ones, but it has been almost 5 years. Samsung something 850 evo if I remember right.