Jump to content


Photo

out of memory crash...


  • Please log in to reply
26 replies to this topic

#1 boota

boota

    Rubber Ducky

  • Vestrum
  • Pip
  • 39 posts
  • LocationStockholm, Sweden

Posted 05 November 2015 - 15:45

so windows ran out of memory and the game crashed to desktop when i was walking about  propast  and asked shitbot to weld a little. i could not find any dump or crashlog, but i attach the output_log.txt, which conveniently enough ends with a crash message because the memory pool was unsufficient

 

i am running the 3.2.1 beta

 

this is my system:

amd phenom II x4 975 (3,6 GHz)

12 GB ram

Windows 10 Pro 64-bit

 

also, i am not using any page file, since i have an SSD. but it seems wierd that the game would consume consume 6-8 GB memory, and it's the first time i have had an out of memory problem

Attached Files


  • khh likes this

#2 Dreamcatcher

Dreamcatcher

    Arcadia Native

  • Vestrum
  • PipPipPip
  • 148 posts
  • LocationGermany

Posted 05 November 2015 - 17:14

I had performance decrease over time and monitored RAM usage
and I found that the private bytes allocated to Dreamfallchapter.exe
go very very high.

I have a phenomII X4 940
4Gb Ram
win7 64



#3 boota

boota

    Rubber Ducky

  • Vestrum
  • Pip
  • 39 posts
  • LocationStockholm, Sweden

Posted 05 November 2015 - 17:31

to clarify. this is the first time i have had an out of memory issue on this system. technically, the last time i ran out of memory was when trying to launch  wing commander privateer without extending memory first.



#4 khh

khh

    Harbinger of the Balance

  • Moderator
  • 7106 posts
  • LocationNorway

Posted 05 November 2015 - 17:50

I had performance decrease over time and monitored RAM usage
and I found that the private bytes allocated to Dreamfallchapter.exe
go very very high.

How high roughly? Before the beta I never saw it above 2.8G


April Ryan is my friend,
Every sorrow she can mend.
When I visit her dark realm,
Does it simply overwhelm.


#5 Morten

Morten

    RTG Programmer

  • RTG Staff
  • 1058 posts
  • LocationOslo, Norway

Posted 05 November 2015 - 18:35

The Unity version we're currently building with is known to leak memory while rendering shadows (aka. all the time), which is totally bonkers.

You can see this yourself by just loading up a scene and stand around for a while. You'll see the memory usage is gradually climbing at a pretty high rate (couple of megabytes every 1-2 seconds). Sometimes the Garbage Collector can dump a small chunk, but not enough to counteract the memory leak.

 

"So, why are you building with this version of Unity which has a known memory leak?", you say. Well. It's kind of sad, I'm afraid.

The whole Unity 5 update cycle has been full of bugs in almost every aspect of the engine. Multiple times when updating and testing out a recently released patch, we've seen that they have fixed some stuff (which is often very insignificant fixes compared to some of the major bugs in the engine), but the problem is that some of these patches introduce even more bugs, often major ones.

This has, to say the least, been a frustration beyond imagination.

Now, in the version we're currently using, some things were fixed that we needed to have for DFC. But of course this version also introduced the aforementioned shadow rendering memory leak.

So basically, this is the most stable Unity 5.2 patch to date for us, even if it has the memory leak.

 

There is a recently released patch of Unity where the memory leak has been fixed, but that version also had a regression bug (among others) which re-introduced a bug where realtime lights in a scene will incorrectly be culled by occlusion culling even though they are in plain view. /facepalm

So yeah, we can't use that version either.

 

Some major - no, MAJOR - bugs that we are still struggling with in the engine we've seen reports of as far back as March of this year, and even they have yet to be fixed.

 

--

 

Anyway, sorry for the digression, but I just wanted to explain why we're building the game with such a bug in the engine. :)

 

With regards to your reported issue, boota:

The error at the bottom of the log you attached doesn't seem like the sort of entry one would get from a normal out of memory issue. 

I can't say for certain that is has nothing to do with memory, because it might very well be as a lot of weird stuff will happen as an application starts to use a good chunk of memory, nearing the limit.

Also, it seems like at the point of the crash, you've played for quite some time which may mean that the memory leak mentioned has already started to grab a hold of a lot of your available memory, which then lead to the issue causing the crash.

 

I've seen this kind of issue only a couple of times before, and it seems to be quite a rare one. It might have something to do with creating memory references to materials or such in-game, but there has been no real clarification of this issue since the first occurrence were reported to Unity staff.

 

You should, however, be able to re-open the game and continue on from your last saved game. 
I'm very sorry for the inconvenience, and I hope we'll be able to squash this one for good sometime soon.


  • khh, Dreamcatcher, CosmicD and 6 others like this

Red Thread Games PR0grammer & Naturally Selected Part-Time "Keep Track of Forum"-person / Community Overseer


#6 Dreamcatcher

Dreamcatcher

    Arcadia Native

  • Vestrum
  • PipPipPip
  • 148 posts
  • LocationGermany

Posted 05 November 2015 - 19:38

If this has to do with rendering shadows could reduce shadow distance or setting it to 0 improve this or temporarily be a workaround?



#7 boota

boota

    Rubber Ducky

  • Vestrum
  • Pip
  • 39 posts
  • LocationStockholm, Sweden

Posted 05 November 2015 - 19:48

The Unity version we're currently building with is known to leak memory while rendering shadows (aka. all the time), which is totally bonkers.

You can see this yourself by just loading up a scene and stand around for a while. You'll see the memory usage is gradually climbing at a pretty high rate (couple of megabytes every 1-2 seconds). Sometimes the Garbage Collector can dump a small chunk, but not enough to counteract the memory leak.

 

"So, why are you building with this version of Unity which has a known memory leak?", you say. Well. It's kind of sad, I'm afraid.

The whole Unity 5 update cycle has been full of bugs in almost every aspect of the engine. Multiple times when updating and testing out a recently released patch, we've seen that they have fixed some stuff (which is often very insignificant fixes compared to some of the major bugs in the engine), but the problem is that some of these patches introduce even more bugs, often major ones.

This has, to say the least, been a frustration beyond imagination.

Now, in the version we're currently using, some things were fixed that we needed to have for DFC. But of course this version also introduced the aforementioned shadow rendering memory leak.

So basically, this is the most stable Unity 5.2 patch to date for us, even if it has the memory leak.

 

There is a recently released patch of Unity where the memory leak has been fixed, but that version also had a regression bug (among others) which re-introduced a bug where realtime lights in a scene will incorrectly be culled by occlusion culling even though they are in plain view. /facepalm

So yeah, we can't use that version either.

 

Some major - no, MAJOR - bugs that we are still struggling with in the engine we've seen reports of as far back as March of this year, and even they have yet to be fixed.

 

--

 

Anyway, sorry for the digression, but I just wanted to explain why we're building the game with such a bug in the engine. :)

 

With regards to your reported issue, boota:

The error at the bottom of the log you attached doesn't seem like the sort of entry one would get from a normal out of memory issue. 

I can't say for certain that is has nothing to do with memory, because it might very well be as a lot of weird stuff will happen as an application starts to use a good chunk of memory, nearing the limit.

Also, it seems like at the point of the crash, you've played for quite some time which may mean that the memory leak mentioned has already started to grab a hold of a lot of your available memory, which then lead to the issue causing the crash.

 

I've seen this kind of issue only a couple of times before, and it seems to be quite a rare one. It might have something to do with creating memory references to materials or such in-game, but there has been no real clarification of this issue since the first occurrence were reported to Unity staff.

 

You should, however, be able to re-open the game and continue on from your last saved game. 
I'm very sorry for the inconvenience, and I hope we'll be able to squash this one for good sometime soon.

I guessed memory leak since i had played for a while.

 

You don't need to be sorry. it's a beta. i know what i'm in for (I had to manage alpha/beta testing of a custom built CRM/ERP in my last job. Without training or any real programming skills at that. i'm a network admin /support tech. but when did any manager ever care about that? you can do it, they said. you're the one in IT with the best knowledge of our routines, they said... do it or you'll have to rewire the office they said... so after one of the worst years of my professional life I left for college, the project was cancelled a couple of months later and when I graduate I will never have to know jack about computers ever again.)

 

I just wanted to report it to make you aware of it / add it to already similar reports so that you could prioritize it properly, on the odd chance that there was something truly bad going on in the background.

 

happy bughunting! i will continue to report bugs when i find them.


Edited by boota, 05 November 2015 - 19:50.

  • Morten and Martin like this

#8 Ikon

Ikon

    Harbinger of the Balance

  • Istrum
  • PipPipPipPipPipPip
  • 2788 posts

Posted 05 November 2015 - 19:56

I can confirm the issue as well. I was running around Propast with Task Manager displayed on my other monitor and, sure enough, the memory usage kept climbing. It started around 2.5GB and was almost 2.8GB went I quit. I guess this is a situation where my having 16GB or RAM is a distinct advantage :)


  • Mr_Russ and Mixthos like this

If at first you don't succeed, do it like your mother told you.


#9 Dreamcatcher

Dreamcatcher

    Arcadia Native

  • Vestrum
  • PipPipPip
  • 148 posts
  • LocationGermany

Posted 05 November 2015 - 20:25

I can confirm the issue as well. I was running around Propast with Task Manager displayed on my other monitor and, sure enough, the memory usage kept climbing. It started around 2.5GB and was almost 2.8GB went I quit. I guess this is a situation where my having 16GB or RAM is a distinct advantage :)

There are different ways to monitor Memory usage or types of memory to monitor. 

Working set and private bytes for example. I confuse them all the time and do not completly understand the differences myself.

 

All I can say: My peak usage of private bytes was around 10Gb in marcuria at one time.

 

And so far (running DFC on a second PC right now) setting shadow distance to 0 seems to improve it.

private bytes are roughly at 2Gb in propast and seem stable.

So if someone runs into this it might be worth to try reducing shadow distance.


  • Morten likes this

#10 Mixthos

Mixthos

    Rubber Ducky

  • Vestrum
  • Pip
  • 37 posts
  • LocationPaderborn, Germany

Posted 06 November 2015 - 00:09

also, i am not using any page file, since i have an SSD. but it seems wierd that the game would consume consume 6-8 GB memory, and it's the first time i have had an out of memory problem

Windows can have weird issues if you don't use a page file. If I remember correctly it can throw a Out Of Memory Exception even if your RAM is only filled with cached data that would otherwise just be replaced. You could try using a small (~1GB) page file and see if that helps.

 

PS: Closing Firefox/Chrome can also help. ;)


  • Mr Moo likes this

#11 CosmicD

CosmicD

    Vestrum Herald

  • Shifter
  • PipPipPipPipPip
  • 1555 posts
  • LocationHasselt, Limburg, Belgium

Posted 06 November 2015 - 01:35

I will do a test of this tomorrow morning (aka in 7 or 8 hours :P)

I've had dfc sessions up to 3 hours (where lots of the time, I had it minimized in desktop and reopened after 2 hours). It lasted long to restore and refocus but it did work. My memory usage  In the memory usage monitor (which I didn't really show pictures on when posting about fps increase) it gradually increased. But I'd like to know if I can get my 4gb filled and provoke a crash.

Perhaps when you have a pagefile it happens less quickly as things can be cashed in the pagefile. However, I'd like to test this.

BTW if they manage to fix all bugs without introducing older ones, do you need to repack the entire game again ? 

I kinda wonder, that you're making this game to behave in a way that it shows less issues with the bugs. But if those bugs get fixed, will the game be buggy again because of fixed things that get unfixed because they fixed it ? (lol)



#12 boota

boota

    Rubber Ducky

  • Vestrum
  • Pip
  • 39 posts
  • LocationStockholm, Sweden

Posted 06 November 2015 - 10:15

Windows can have weird issues if you don't use a page file. If I remember correctly it can throw a Out Of Memory Exception even if your RAM is only filled with cached data that would otherwise just be replaced. You could try using a small (~1GB) page file and see if that helps.

 

PS: Closing Firefox/Chrome can also help. ;)

The thing is i want to minimize the number of read/writes to my ssd (due to deterioration), and ditching the page file will improve system performance if you have the RAM to support it. If you don't, well, you run out memory.

 

I have not experienced any out of memory issues running without a page file in either windows 7, 8, 8.1 or 10 (Often playing long sessions). So this tells me that the issue is that there is a memory leak in the application (which to me is fine as long as it doesn't brake the game too soon, i think i had been playing the game for about 4 hrs when it crashed). The same thing would probably have happened even if i had a page file. it would just have taken an even longer time. So the easy solution here is to not play all the books in one sitting. the reason i reported it is so that if RTG realize this is an issue on machines with less memory, or feel that this is something they need to fix, they will have any information i had. (and also i like the attention).


  • Morten likes this

#13 Morten

Morten

    RTG Programmer

  • RTG Staff
  • 1058 posts
  • LocationOslo, Norway

Posted 06 November 2015 - 10:43

I kinda wonder, that you're making this game to behave in a way that it shows less issues with the bugs. But if those bugs get fixed, will the game be buggy again because of fixed things that get unfixed because they fixed it ? (lol)

No, we of course try to avoid this. :)

 

For instance, one of the MAJOR bugs was one that was not converting the directional lightmap textures - after a lightmap bake - to the correct color space (lightmaps are basically a texture containing pre-rendered/static lighting data for a scene, so that we can reduce the amount of realtime lights used, among other benefits). So, we baked the lightmaps, and it looked totally fine on the computer that baked them. However, on another computer - after syncing the change over our version control server - the lightmaps looked totally borked. For some reason, the lightmap textures were incorrectly imported to Unity on the new computer, setting the texture to contain colors in the incorrect color space.

Basically, one can say it's a matter of setting a value to 1 instead of 0..

 

So what we did was, we made a workaround for it: Whenever anyone on the team is importing/reimporting a directional lightmap texture, we check if that texture has the correct color space settings. If not, we change it locally on that computer. If it already has the correct settings; leave it be.

 

This seems like an easy fix and it goes to show how stupid some of the bugs in the engine are. However, it took us a long time to dig into it and find out the reason why our lightmaps were borked in the first place.

When we first found it, I couldn't believe something like this could even make it through one testing cycle at Unity. It's such an integral part of a game. 

 

Anyhow, that's how we worked around one of the major issues we had earlier on. And this workaround will not have any impact once Unity fixes the bug (yes, it's reported personally to Unity), since then the lightmap textures will have the correct settings when reimporting and the workaround code will have to do nothing.


  • khh, CosmicD and Mixthos like this

Red Thread Games PR0grammer & Naturally Selected Part-Time "Keep Track of Forum"-person / Community Overseer


#14 CosmicD

CosmicD

    Vestrum Herald

  • Shifter
  • PipPipPipPipPip
  • 1555 posts
  • LocationHasselt, Limburg, Belgium

Posted 06 November 2015 - 11:19

Oh the joy of finding out why something bugs out and the stress and uncertainty that goes with it. it was killing us as a mod team. It will surely kill the mood of a professional team that has to deliver a game.

We had stuff when we wanted to pack the l4d expansion, the game would crash when we looked at a certain undefined thing. After months we found out that if we used models compiled with pre-orange box toolset it crashed. Not only did I build bsp after bsp with some functions turned off and on, We had to scavange forums for it and eventually I made the connection because one guy was saying that "old hl2 native props would crash their campaign when they packed it in vpk". So one of our modelers DID made some custom props for an old mod that we used, but he did it with the source engine before the orange box update. I turned off all these models.. voila, fixed!

Good times, valve responded also and fixed some other stuff in the process that prevented us to do sound properly in custom campaigns. But our voice actor would get angry at us because we couldn't enclude and distribute the voice acting he did before valve fixed all this stuf.

YAY


  • Morten and boota like this

#15 CosmicD

CosmicD

    Vestrum Herald

  • Shifter
  • PipPipPipPipPip
  • 1555 posts
  • LocationHasselt, Limburg, Belgium

Posted 06 November 2015 - 16:48

Tried it:

It appearently did crash. All I had to do was go stand in propast and go shopping, after an hour it crashed (just by standing still). The thing is,  I can play and alt tab and it would remain open longer, seems that when it's out of focus you prevent it from running out of memory



#16 Ikon

Ikon

    Harbinger of the Balance

  • Istrum
  • PipPipPipPipPipPip
  • 2788 posts

Posted 06 November 2015 - 18:52

seems that when it's out of focus you prevent it from running out of memory

 

That would make sense. Sounds exactly like what Windows does to balance CPU usage: i.e. de-allocate resources to inactive & minimized apps. I do suspect, however, that this might change depending on whether the Windows "Adjust for best performance of:" setting is configured to give preference to Programs or Background Services.


If at first you don't succeed, do it like your mother told you.


#17 khh

khh

    Harbinger of the Balance

  • Moderator
  • 7106 posts
  • LocationNorway

Posted 06 November 2015 - 18:56

That would make sense. Sounds exactly like what Windows does to balance CPU usage: i.e. de-allocate resources to inactive & minimized apps. I do suspect, however, that this might change depending on whether the Windows "Adjust for best performance of:" setting is configured to give preference to Programs or Background Services.

No, I think it's probably because the game doesn't render frames when it's running as a fullscreen application and is in the background. And it's almost certainly during the rendering that it's leaking memory, since Morten says it's related to the shadows.


  • Morten likes this

April Ryan is my friend,
Every sorrow she can mend.
When I visit her dark realm,
Does it simply overwhelm.


#18 Ikon

Ikon

    Harbinger of the Balance

  • Istrum
  • PipPipPipPipPipPip
  • 2788 posts

Posted 06 November 2015 - 19:21

That's an interesting idea. Does that mean it would continue to render frames if it's run as a windowed app in the background?


If at first you don't succeed, do it like your mother told you.


#19 khh

khh

    Harbinger of the Balance

  • Moderator
  • 7106 posts
  • LocationNorway

Posted 06 November 2015 - 19:27

That's an interesting idea. Does that mean it would continue to render frames if it's run as a windowed app in the background?

I'm not sure, actually. If any part of it was showing it'd have to render that part, and Windows has the live preview thing. But if it's properly minimized? Not sure.


April Ryan is my friend,
Every sorrow she can mend.
When I visit her dark realm,
Does it simply overwhelm.


#20 boota

boota

    Rubber Ducky

  • Vestrum
  • Pip
  • 39 posts
  • LocationStockholm, Sweden

Posted 06 November 2015 - 22:21

It's probably easier and less taxing on the system to just draw the full window than masking the parts that are obscured.

I guess dynamic lighting and stuff like that would still require the full scene to be rendered




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users