"Not enough Memory" - crashes, freezes and more

[WarcraftPundit]

Background: Everything worked well 5x boxing WoW with this system with my old GPU (AMD FirePro V5900 2GB) but performance was poor, as might be expected with such a GPU.

System specs: DELL Precision Workstation T3500, XEON X5675 3.07GHz 6-core, AMD RADEON RX580 8GB, 20GB RAM, Windows 10 Pro 64-bit 10.0.17134
--------
Some images of the error messages:
https://imgur.com/a/MkP67Af
--------
ISBoxer Diagnostics Report generated 09/15/2018 8:14 PM by ISBoxer 42.7.822.2:
https://pastebin.com/1EpEvWQ1
--------

There are several problems I am experienced since upgrading my GPU. I will attempt to describe them as best as I can.

Situation 1: One of the five clients will freeze. This wouldn't be such a big deal except that whilst it is froze the entire system is slowed beyond usability. Previously I would simply End Task of the frozen client in Task Manager but now everything gets nearly completely stuck. The mouse pointer moves extremely slowly and choppy, it takes about 30 to 45 seconds to respond to input (such as ALT + TAB). I am unable to gather a crash dump when this happens as the only way to get out of this start is a hard restart on the system.

Situation 2: One of the five clients will produce an error box stating it doesn't have enough memory. In this case I can force close the client and restart it by loading my IS Boxer team. However, until I restart the system this will repeatedly occur with increasing frequency.

Situation 3: Unknown crashes. Apparently the display driver crashes because the two displays I have attached will flicker and the sound for hardware being added/removed in windows will play. Then the displays both go to sleep and show no input for a moment and display 2 will then turn back on but not display anything. The screen is just black. The system has not crashed at this point. The keyboard and mouse input is still working. I can open things by using the start button and the arrow/enter keys, ALT-TAB between apps etc. However, the only way to get the displays to work is to restart the system.

I've done clean installs of the display drivers. I 'reset' WoW as described in their support forums. I have completely deleted WoW and reinstalled - including all temp files/APP data, etc. and I've created new teams/window layoutrs/etc., so I could start with completely fresh installs, no addons and no potential glitches in the team(s). I can easily open and run five instances of the WoW client with this setup without using IS Boxer and have yet been unable to reproduce the errors without InnerSpace/ISBoxer.

PLEASE Help! I'm nearly unable to play WoW for over a week and all I am doing is troubleshooting & getting angry and angrier each time. I'm quite sure this post isn't properly composed and that I have failed to provide something crucial. Let me know and I will attempt to get any information needed.

ISBoxer Diagnostics Report generated 09/15/2018 8:14 PM by ISBoxer 42.7.822.2

[Toned]

You need to have 32gigs of ram to play smoothly with 5 clients, 64 for 10 clients.
If you only have 20gb, go into This PC > Properties > Advanced System Settings > Advanced Tab > Under Performance Section Settings > Advanced Tab > Virtual Memory > Change Paging File for All drives to a custom size 16-24gb > click Okay, Okay ... restart / try again :P

[WarcraftPundit]

You need to have 32gigs of ram to play smoothly with 5 clients, 64 for 10 clients.
If you only have 20gb, go into This PC > Properties > Advanced System Settings > Advanced Tab > Under Performance Section Settings > Advanced Tab > Virtual Memory > Change Paging File for All drives to a custom size 16-24gb > click Okay, Okay ... restart / try again :P

Thanks for the info. Unfortunately I'm unable to do any better on hardware at this time. It's just not possible, yet!

What is confusing me here is that if I remove the new GPU and replace it with the old 2GB one everything works well. Just without decent graphics settings and a limited frame rate. Additionally, if I do not run the clients using ISBoxer it also works even with the new GPU running all five clients. But without ISBoxer I can't do much. =(

I've tested again today with virtual memory set to 16gb and separately with 20gb and 24gb. The same issues still happen with all three. I'm going to keep trying some things and see what happens though.

[Nezreck]

mm i'm running 8 accounts smooth as butter with only 24 gigs

[mbox_bob]

If you have replaced the GPU, you will need to create a new Window Layout in ISBoxer, as the device references will have changed and this can have a detrimental performance impact when Windows attempts to write to a device which is no longer there. Use the Window Layout Wizard, and create a New layout (don't choose one which says (existing)) from the drop down. If it is not the issue, you can always swap back to your old layout.

Either that or it is paging to disk a lot due to lack of ram, but then some basic monitoring would show up the lack of ram quite easily and it doesn't really gel with what you say about swapping graphics cards.

/e
heh, after re-reading your original post, I see you say you've tried a fresh profile. Well, that just makes it really weird then, and perhaps you should be getting a compatibility diagnostic instead. http://isboxer.com/wiki/Diagnostics

[WarcraftPundit]

If you have replaced the GPU, you will need to create a new Window Layout in ISBoxer, as the device references will have changed and this can have a detrimental performance impact when Windows attempts to write to a device which is no longer there. Use the Window Layout Wizard, and create a New layout (don't choose one which says (existing)) from the drop down. If it is not the issue, you can always swap back to your old layout.

Either that or it is paging to disk a lot due to lack of ram, but then some basic monitoring would show up the lack of ram quite easily and it doesn't really gel with what you say about swapping graphics cards.

/e
heh, after re-reading your original post, I see you say you've tried a fresh profile. Well, that just makes it really weird then, and perhaps you should be getting a compatibility diagnostic instead. http://isboxer.com/wiki/Diagnostics

Thanks for taking the time to answer and try to help. I appreciate it. I have indeed tried all new layouts, profiles and so on. I am also monitoring the memory usage with Open Hardware monitor.

As for the compatibility diagnostics, I can get that but I cannot get crash dumps when it freezes. Until the one WoW client which freezes is terminated the system is literally unusable. I went through all the jagged mouse movement, waiting for the screen to redraw, watching the spinning balls and all that long enough to get through all the motions needed to finally try and get the crash dump from ISBoxer and it failed. That process (no joke) took over 45 minutes with as slow as everything was running. That is why when it does this the only real option is to have the hardware reset with a power cycle and reboot the system.

Yes this is very strange. Like I mentioned too it works very well, just with reduced FPS and such, with the old GPU. It also works with the new GPU if I do not run ISBoxer, that is five clients will all run for as long as I want.

In Open Hardware Monitor it never shows more than about 60% of memory usage. Here are some screen shots of that and task manager whilst all five clients have been running for a while: https://imgur.com/a/rh7pbi5

https://www.dual-boxing.com/attachment.php?attachmentid=2099&stc=1
https://www.dual-boxing.com/attachment.php?attachmentid=2100&stc=1
and finally the GPU in task manager where I notice something interesting but of unknown relevance. It says the GPU has 10GB of shared memory. I assume this means in addition to it's 8GB of VRAM(?) but I don't see that in the monitoring of the RAM usage anywhere. It only shows a small percentage of it in use so maybe it is reserved or potentially allocated if needed?
https://www.dual-boxing.com/attachment.php?attachmentid=2101&stc=1

[WarcraftPundit]

mm i'm running 8 accounts smooth as butter with only 24 gigs

Nice. What CPU/GPU are you using?

[mbox_bob]

Your screen shots show both the CPU and GPU topping out at 100%. Which would make me think that the issue is occurring when that is happening. I shall await the compatibility diagnostic....

/e ok, it was pointed out you provided one initially and I was too dumb to see it.

Other than the excessive amount of Anti Malware processes running, which might be indicative of something funky going on (and potentially bogging the bejesus out of your computer periodically), not much really shows up.

[WarcraftPundit]

Your screen shots show both the CPU and GPU topping out at 100%. Which would make me think that the issue is occurring when that is happening. I shall await the compatibility diagnostic....

/e ok, it was pointed out you provided one initially and I was too dumb to see it.

Other than the excessive amount of Anti Malware processes running, which might be indicative of something funky going on (and potentially bogging the bejesus out of your computer periodically), not much really shows up.


My guess is I was rambling and you simply missed it in the original post =)
I'm trying to have everything that would be relevant so those like yourself who try to help have something to go by other than "it don't work! WAH!"

Re CPU/GPU utilization maxing at 100%: That only happens when I launch teams. After the clients initially load the utilization rarely hits:
RAM: 60%
CPU: 40%
GPU: 30%
As best as I can tell the resources aren't even being maxed when the crashes/freezes occur. In the background the logging still occurs and doesn't show anything unusual. That does give me a thought though. I'm going to reset OHM after I launch the team and see where it is when the issues next occur.


RE antimalware processes. Yeah I noticed that too and have been separately trying to figure out WTH is with it. It doesn't seem to affect anything at all. I'm considering a fresh install of Win 10 and WoW on a spare HDD and see if it shows that after all the updates and such.

/e crashed again, adding resource monitor from right after.
https://www.dual-boxing.com/attachment.php?attachmentid=2102&stc=1

[Nezreck]

Nice. What CPU/GPU are you using?
i have a gtx 1070 and a ryzen 5 1600x

[Ughmahedhurtz]

Will your PC survive overnight Prime95 torture tests?

[mbox_bob]

I think a full reinstall might be in order. I don't like suggesting that as it seems a bit of a cop out, but there be some weird shit going on and it probably needs to be ruled out there is some kind of hangover from the previous setup. Of other note, I can't for the life of me work out AMD driver versions from the paste, because the current "driver version" on their site is like 18.30.17.01, yet none of their actual driver files carry that number. So, can you confirm what driver version (or even Catalyst/Adrenaline/whatever they called it before, version) you are using?

[JohnGabriel]

How big is your power supply? Your new card requires 185 watts and your old <75.

A couple whiles back when I upgraded to dual cards in SLI I was having all sorts of problems and geek squad ended up figuring out my power supply was too small. Might be something for you to look at.

[WarcraftPundit]

Will your PC survive overnight Prime95 torture tests?

I'm not familiar with this at all. I do understand the concept of course but just haven't had any experience with this particular one.

I found this with a bing search: https://www.guru3d.com/files-details/prime95-download.html

I'll download it and try it when next I have a while away from the PC. I should be away in the morrow for about 7 hours, I presume that would be long enough.


How big is your power supply? Your new card requires 185 watts and your old <75.

A couple whiles back when I upgraded to dual cards in SLI I was having all sorts of problems and geek squad ended up figuring out my power supply was too small. Might be something for you to look at.


Interesting, and something I had not thought of. I did checkthat my PC met the system req. for the card, including minimum PSU wattage, before I purchased it. My power supply is a OEM DELL rated at 750W - though I have no way of knowing if that is what it actually does. I have another compatible power supply which is 850 W so I will swap them and see if it makes a difference.


I think a full reinstall might be in order. I don't like suggesting that as it seems a bit of a cop out, but there be some weird shit going on and it probably needs to be ruled out there is some kind of hangover from the previous setup. Of other note, I can't for the life of me work out AMD driver versions from the paste, because the current "driver version" on their site is like 18.30.17.01, yet none of their actual driver files carry that number. So, can you confirm what driver version (or even Catalyst/Adrenaline/whatever they called it before, version) you are using?

Thank you for all your help and no worries over the recommendation being a cop out. I appriciate your (and everyone's) help here!

AMD Driver versions make no sense to me since they have a different version for various parts of it. This is what is reported as the driver version currently.

Driver Packaging Version
18.30.11.01-180827a-332601E-RadeonSoftwareAdrenalin

The previous driver for the FirePro V5900 card: 15.201.2401.1010

[Ughmahedhurtz]

I'm not familiar with this at all. I do understand the concept of course but just haven't had any experience with this particular one.

I found this with a bing search: https://www.guru3d.com/files-details/prime95-download.html

I'll download it and try it when next I have a while away from the PC. I should be away in the morrow for about 7 hours, I presume that would be long enough.
The official site is https://www.mersenne.org/download/#download but that should work. It just exercises your CPU cores and RAM such that if you have bad memory or flaky devices due to poor power supply output or some other flaky hardware or a terrible heatsink setup, it'll fail the test and report an unexpected result. Or it'll BSOD. Or overheat. Watch it for 15 minutes or so and if the thermals don't climb past 65-70, it should be OK for longer tests.

Something to add to the power supply comment above is that if you stack a bunch of devices up on the same rail e.g. 4 HDDs on the same SATA cable bundle, etc. sometimes that will overload just that rail and the associated outputs. Your manual will usually describe which rails are common and how much they're rated for, though being a Dell, maybe not as I don't think those are modular.

The "common" thing also applies to PCI-E SSDs and SATA ports, depending on the motherboard. Some of my MB's SATA ports for example become unavailable when I connect a PCI-E SSD on one of the M.2 ports. This usually results in a simple failure to enumerate the affected SATA ports, but I can't speak to other shenanigans that might happen if you accidentally comingle those.

[WarcraftPundit]

The official site is https://www.mersenne.org/download/#download but that should work. It just exercises your CPU cores and RAM such that if you have bad memory or flaky devices due to poor power supply output or some other flaky hardware or a terrible heatsink setup, it'll fail the test and report an unexpected result. Or it'll BSOD. Or overheat. Watch it for 15 minutes or so and if the thermals don't climb past 65-70, it should be OK for longer tests.

Something to add to the power supply comment above is that if you stack a bunch of devices up on the same rail e.g. 4 HDDs on the same SATA cable bundle, etc. sometimes that will overload just that rail and the associated outputs. Your manual will usually describe which rails are common and how much they're rated for, though being a Dell, maybe not as I don't think those are modular.

The "common" thing also applies to PCI-E SSDs and SATA ports, depending on the motherboard. Some of my MB's SATA ports for example become unavailable when I connect a PCI-E SSD on one of the M.2 ports. This usually results in a simple failure to enumerate the affected SATA ports, but I can't speak to other shenanigans that might happen if you accidentally comingle those.


I am so glad this has gone in this direction! The PSU comments has led me to some testing and an important discovery indeed.

I believe inadequate power is the problem. This computer was purchased used as a less than adequate replacement for my main PC which was killed in a lightning strike back in early June. It'll be some time yet before I can truly invest in the hardware I need.

Anyway, the power supply was supposed to be 750W. It had a label saying 750W, but under that label is one that says 500W! So with some checking of the part numbers the top label goes to a completely different type. The rating for 12V is only 150W, obviously not enough for 95W CPU & 185W GPU. This would easily explain the insane behavior. Obviously I cannot further troubleshoot or test until I resolve this.

The system will be torn down as soon as I post this and I'll feed the wiring harness through the case and replace the power supply with my tested and working 875W unit.

Next:
I just noticed your name, Ughmahedhurtz (https://www.dual-boxing.com/members/843-Ughmahedhurtz). I love it! It is something like I might have used as a chronic migraineur!

Next again:
Re CPU temps. You state "Watch it for 15 minutes or so and if the thermals don't climb past 65-70..." this makes me curious. My CPU, a XEON X5675, at idle is 33.4°C. During the testing or when I otherwise max all 12 logical cores at 100% it maxes out at 80.1°C. This didn't seem like a problem as the specs for the CPU state the max temp at the heat spreader to be 81.3°C. Am I mistaken in this? 80°C does seem hot but within spec, so I assumed it was fine. If not I will look for a better cooling option.

[Ughmahedhurtz]

Next:
I just noticed your name, Ughmahedhurtz (https://www.dual-boxing.com/members/843-Ughmahedhurtz). I love it! It is something like I might have used as a chronic migraineur!
I have a last name...


Next again:
Re CPU temps. You state "Watch it for 15 minutes or so and if the thermals don't climb past 65-70..." this makes me curious. My CPU, a XEON X5675, at idle is 33.4°C. During the testing or when I otherwise max all 12 logical cores at 100% it maxes out at 80.1°C. This didn't seem like a problem as the specs for the CPU state the max temp at the heat spreader to be 81.3°C. Am I mistaken in this? 80°C does seem hot but within spec, so I assumed it was fine. If not I will look for a better cooling option.TL;DR wall incoming...
Preface question: what does yours do under the max gaming real-world load (as opposed to stress test)?

Here's the thing about the thermals, in my experience. Consider heat soak. Your GPU, SSD/HDD, chipset, and PSU all generate heat. Your PC sitting idle will probably feel cool to the touch. When you hammer it, that spikes the CPU/RAM temps (and maybe GPU if you're running a stress test on that too) which introduces more heat into the chassis. At some point, the amount of heat load in the system may exceed the ventilation system's ability to exhaust it, which ends up causing in effect a thermal runaway. Realistically, few chassis have such poor thermal design that they'd completely runaway to meltdown (*cough*macbook*cough*), but having your fans opposing each other or something like excessive dust buildup can dramatically reduce the effectiveness of fans, heatsinks, etc. HP went from multiple fans to a ducted single-fan design to allow for smaller (read: cheaper) cases/fans/heatsinks to be used on the same CPUs. I like to run H80i CPU coolers with 4x Noctua quiet case fans in mine, which usually results in easy sub-70C stress temps and the loudest thing in the room is the NVidia fan.

I also usually try to allow for fan death or "unexpected results" in my testing as I'd rather have an unexpected spike to 80-85C instead of an unexpected spike to dramatically over-spec temps, especially if something prevents the auto-shutdown feature from saving itself. Again, much less of a problem with more modern stuff but with the way CPU build costs have been elevated lately (yours is or was $1400), I see no reason to risk heat-induced early system issues.

That said, the XEON X5675 specs show a TCase temp of 81.5C. I can't find a reasonable logical relation between TCase temp (the temp at the integrated heat spreader on the CPU itself) and a reading from CoreTemp/etc. Example of where temps are measured, at least on an older system:

https://www.dual-boxing.com/attachment.php?attachmentid=2104&stc=1

Generally, I'd expect ~12C to be a reasonable difference, meaning yours should be *technically* OK up to ~93C core temps, but again that's just a guess based on anecdotal evidence. :P And I sure as hell wouldn't want to run it at 80C+ for extended periods. My hardware engineer colleagues constantly talk about how the lifespan of capacitors/transistors is directly related to how hot they run, and it's not a linear function. ;)

[WarcraftPundit]

Update: Power supply replaced and system up and running again with 875W 80+ Silver power supply instead of the 500W PoS.

Initial tests good, overnight stress tests and then gaming tomorrow if everything works that long.

Previously the system failed the Prime95 tests within minutes. Been running for an hour now with no failures. (crosses fingers)



Preface question: what does yours do under the max gaming real-world load (as opposed to stress test)?


According to OHM log files gaming temps range from 48 - 53C — I'll test this again tomorrow when checking how well it works multiboxing with the new PSU installed.
--
Thanks for all the great information on the thermals. Much more than I expected!
I'm going to look into the cooler/fans you mentioned, if not for this system for whatever I replace it with as I'm able to do so.


https://www.passmark.com/baselines/V9/images/107867746592.png (https://www.passmark.com/baselines/V9/display.php?id=107867746592)

[WarcraftPundit]

Thanks so much to all who have replied and helped with this with the great ideas and suggestions!

After several days of testing with the proper power supply I have had only a single crash which was much easier to recover from. Though it acts very similarly with the slowing of the system until the offending WoW client is killed, it is not entirely unusable. I can live with an occasional crash.

The system remains stable with up to 24 hours straight of Prime95 and other stress testing. I've not had any errors so far in any such testing. Using HWinfo I see that the only error during this crash was CPU 0 DTS: Core #2 Power Limit Exceeded Yes. No other core showed this error. Possibly coincidental I have the CPU cores assigned to individual WoW clients in order. The third client is the one which crashed and that was assigned to CPU core 2, the third one.

[mbox_bob]

Possibly coincidental I have the CPU cores assigned to individual WoW clients in order. The third client is the one which crashed and that was assigned to CPU core 2, the third one.

Unless you are managing all your other background applications and OS functions to make sure they all stick to their cores (network IO / sound / keyboard and mouse input, and anything else that throws interrupts) are using the other cores, I'd suggest against doing that. You are effectively artificially limiting the game, and if the OS grabs your core for something else, your game is SOL and just halts.

As a general rule, you should stick to "All CPUs for All Games" in the CPU strategy wizard of ISBoxer (the last option in the drop down). There are very few reasons these days to limit the cores. This is also the same as just letting Windows, and the smart boffins that wrote the CPU Scheduler manage it (in fact there is another option in the CPU Strategy Wizard of select no CPU's, let windows do it, which is effectively the same as select all CPUs).

Also, it's not clear from what you wrote, but if you select them in order, 1, 2, 3, 4, 5, etc, in the performance tab in ISBoxer, every EVEN number is a hyper thread core, so that game, and the game on the previous odd number are running on the same physical core, but on the two logical cores for that physical. The "rule of thumb" is that a HT logical adds about 30% to the physical, so that would be 1.3 cores for each game. That is a tad limiting, and likely to cause them to run hot when they don't need to.

[Ughmahedhurtz]

If you feel the need to manage your cores on your own, here's a strategy that has worked fine for me on i7-6600k and later models in the series.

Client 1 = 1,2,3,4
Client 2 = 1,2,3,4
Client 3 = 5,6,7,8
Client 4 = 5,6,7,8
Client 5 = 3,4,5,6
dxNothing = 1,2,7,8

I'm sure I'm probably doing something wrong there but I've had no issues running 1080p @ Ultra on the main with the detail macros set up properly to limit the background windows. And I usually watch Netflix/YouTube and have an unholy amount of tabs open while this is going on. I just don't have issues with crashes, stalls, etc.

With your 6/12 CPU, I'd probably run clients 2 and 5 on cores 9,10,11,12 and dxNothing on 1,2,3,4 with client 1. That's pure theorycraft, though, as I've never run anything other than bog-standard i7's.