[Noted] [Debugging Tool] Memory Leak

[2McCoy7Member]

CTD’s are still occurring during saves and starting battles and ending turns.

I am not sure what language the game is written in but I figured I’d post this Memory Leak tool that can be purchased from Intel. If the game does happen to be written in C, C++ or C# this tool can find all the different types of Memory Leaks and even give you an exact line. I’m no salesmen just did some research on Memory Leaks and they seem to be extremely hard to diagnose and fix so if there is something out there that may make your life easier its worth a try! Please note that the program is expensive however there is an option for a free trial.

Memory and Thread Debugger

Debug memory errors like leaks and allocation errors and threading errors like data races and deadlocks
No special builds! Use your normal debug or optimized build

https://software.intel.com/en-us/intel-inspector-xe

[NarvekKeymaster]

Hi 2McCoy7,

if you have any saves, logs and/or dumpfiles relating to the crashes you experience could you email us about it? Can you also provide us with your system specs?
Check the bug reporting guide here: http://ageofwonders.com/support-2/

For detecting leaks etc we have our own tools, but thank you for the tip.

[LaschaModerator]

Leak should be improved in next build. Please let me know if you still have trouble. Press CTRL+ALT+M while playing in debug mode to keep track of memory usage. Memory usage shouldn’t increase much, if you see it increase 100+mb in ~50 turns then there’s definitely a problem. Use strategic map to check memory usage, because in tactical combat it will increase, obviously, since more content is loaded at that point, after which the memory should be freed again.

[2McCoy7Member]

@narvek: I have already sent a savegame and my computer specs a long time ago. All the issues I have been having are identical. I can reload the last save game and keep playing. I will however start sending in all my saves and logs if I encounter any of these issues again

@lascha: I just downloaded Eternal lords and played for a couple hours. No issues so far but will do a longer continuous play through on my days off. As far as memory usage I never did see it creep up. I even shut off all essential services and was only using 1.5gb of the 8gb’s I have total on my system (holding steady) with the game running and it was still CTD’ing with the out of memory error.

But hopefully as you said its a thing of the past.
Nice job on the add-on by the way. Only a couple hours in and I’m liking the new content, especially the racial governance bonus.

Any plans to let the linux users do some modding?

[LaschaModerator]

2McCoy7 wrote:

Any plans to let the linux users do some modding?

Sadly, no 🙁 Mac & Linux ports were never planned to begin with. Porting the tools aswell is very out of scope for us. We intend to take it into account when starting with future games though.

[2McCoy7Member]

Sadly Eternal Lords still has memory leak
I emailed the first log and dmp file of many yet to come…

log0420_1406.txt:157: [14:08:42]Heap: World – Leaking memory! (Allocations left: 70)
log0420_1406.txt:185: [14:08:54]Heap: World – Leaking memory! (Allocations left: 93)
log0420_1406.txt:243: [14:11:44]Heap: World – Leaking memory! (Allocations left: 154)
log0420_1406.txt:289: [14:14:26]Heap: World – Leaking memory! (Allocations left: 100)
log0420_1406.txt:330: [14:14:41]Heap: World – Leaking memory! (Allocations left: 70)
log0420_1406.txt:390: [14:18:32]Heap: World – Leaking memory! (Allocations left: 61)
log0420_1406.txt:467: [14:23:20]Heap: World – Leaking memory! (Allocations left: 9)
log0420_1406.txt:499: [14:23:53]Heap: World – Leaking memory! (Allocations left: 19)
log0420_1406.txt:554: [14:25:11]Heap: World – Leaking memory! (Allocations left: 182)
log0420_1406.txt:614: [14:26:48]Heap: World – Leaking memory! (Allocations left: 181)
log0420_1406.txt:682: [14:28:15]Heap: World – Leaking memory! (Allocations left: 43)
log0420_1406.txt:727: [14:28:27]Heap: World – Leaking memory! (Allocations left: 218)
log0420_1406.txt:755: [14:28:31]Heap: World – Leaking memory! (Allocations left: 94)
log0420_1406.txt:796: [14:28:38]Heap: World – Leaking memory! (Allocations left: 71)
log0420_1406.txt:865: [14:32:23]Heap: World – Leaking memory! (Allocations left: 20)
log0420_1406.txt:893: [14:32:26]Heap: World – Leaking memory! (Allocations left: 41)
log0420_1406.txt:917: [14:32:39]Heap: World – Leaking memory! (Allocations left: 36)
log0420_1406.txt:995: [14:39:46]Heap: World – Leaking memory! (Allocations left: 123)
log0420_1406.txt:1034: [14:39:57]Heap: World – Leaking memory! (Allocations left: 62)
log0420_1406.txt:1132: [14:42:51]Heap: World – Leaking memory! (Allocations left: 467)
log0420_1406.txt:1179: [14:43:07]Heap: World – Leaking memory! (Allocations left: 218)
log0420_1406.txt:1210: [14:46:54]Heap: World – Leaking memory! (Allocations left: 19)
log0420_1406.txt:1232: [14:47:03]Heap: World – Leaking memory! (Allocations left: 36)
log0420_1406.txt:1264: [14:47:07]Heap: World – Leaking memory! (Allocations left: 13)
log0420_1406.txt:1317: [14:51:50]Heap: World – Leaking memory! (Allocations left: 69)
log0420_1406.txt:1366: [14:56:56]Heap: World – Leaking memory! (Allocations left: 7)
log0420_1406.txt:1408: [14:57:10]Heap: World – Leaking memory! (Allocations left: 218)
log0420_1406.txt:1475: [14:57:12]Heap: World – Leaking memory! (Allocations left: 70)
log0420_1406.txt:1568: [15:00:17]Heap: World – Leaking memory! (Allocations left: 201)
log0420_1406.txt:1616: [15:03:01]Heap: World – Leaking memory! (Allocations left: 218)
log0420_1406.txt:1667: [15:07:38]Heap: World – Leaking memory! (Allocations left: 119)
log0420_1406.txt:1706: [15:09:23]Heap: World – Leaking memory! (Allocations left: 45)
log0420_1406.txt:1734: [15:09:43]Heap: World – Leaking memory! (Allocations left: 16)
log0420_1406.txt:1762: [15:09:47]Heap: World – Leaking memory! (Allocations left: 61)
log0420_1406.txt:1798: [15:09:49]Heap: World – Leaking memory! (Allocations left: 20)
log0420_1406.txt:1826: [15:09:59]Heap: World – Leaking memory! (Allocations left: 58)
log0420_1406.txt:1844: [15:10:08]Heap: World – Leaking memory! (Allocations left: 35)
log0420_1406.txt:1913: [15:13:09]Heap: World – Leaking memory! (Allocations left: 252)
log0420_1406.txt:2020: [15:14:56]Heap: World – Leaking memory! (Allocations left: 44)
log0420_1406.txt:2044: [15:15:27]Heap: World – Leaking memory! (Allocations left: 38)
log0420_1406.txt:2100: [15:22:30]Heap: World – Leaking memory! (Allocations left: 23)
log0420_1406.txt:2195: [15:25:38]Heap: World – Leaking memory! (Allocations left: 235)
log0420_1406.txt:2272: [15:30:31]Heap: World – Leaking memory! (Allocations left: 98)
log0420_1406.txt:2360: [15:31:27]Heap: World – Leaking memory! (Allocations left: 20)
log0420_1406.txt:2396: [15:31:56]Heap: World – Leaking memory! (Allocations left: 13)
log0420_1406.txt:2452: [15:35:08]Heap: World – Leaking memory! (Allocations left: 142)
log0420_1406.txt:2508: [15:36:12]Heap: World – Leaking memory! (Allocations left: 182)
log0420_1406.txt:2558: [15:40:53]Message: Out of memory!

[LaschaModerator]

Are you using cloud saves by any chance? There’s a leak there in the current build which will accumulate pretty fast since it usually goes off at least once per turn

[2McCoy7Member]

Nope but as you can see in the logfile I did forget to turn autosaves off…
Graphics are on ultra. Not sure if that makes a difference other then speed up the ctd. Vertical sync box is also checked..

Not sure what other info might be useful

[2McCoy7Member]

I’m not sure if I’m the only Linux user having issues but if so I just thought of another consistency.

I always play a random generated map with my own custom leader. No big deal but the leader is always the sorcerer class. I spam Glyphs of warding on all my cities. Perhaps there is something linked with those particles that float around all my cities that has something to do with the leak…
As far as my armies go I use mostly Earth Elementals, perhaps there could be something linked with those also.

^^This all might be useless info but I am not seeing anyone else having issues anymore and I know its not my OS. I played Empire total war all day when it was newly ported to Linux without any issues…

I’ll post again if I remember anymore of my play-style consistencies…

[tezeriuszMember]

Hi, I have crashes due to memleaks too, and funny parts is that I noticed it after grabbing new dlc.

Attachments:

1. log0422_1034.txt
2. launcher0422_1034.txt

[2McCoy7Member]

@tezeriusz: I’ve always had ctd’s right from the linux beta release. Perhaps you have started playing more consecutive hours then before? or turned the graphics up to a higher setting?

@devs: Emailed my second CTD. Out of memory when entering manual battle. My first one occurred after I saved and ended my turn.

[tezeriuszMember]

@2mccoy7: Setting are that same. Yes I had longer session.

[LaschaModerator]

Thought there was another memory leak but there’s a serious fragmentation issues on Linux OS. Some fixes were made for the linux build to prevent some fragmentation when a save goes off. It’s not perfect, but it definitely helped in my tests. The new build is: Steam Library > AoW3 > Properties > BETAS > open_beta

PS. You can ignore this message. Those “leaks” actually do get cleaned up afterwards.
log0420_1406.txt:2508: [15:36:12]Heap: World – Leaking memory! (Allocations left: 182)

[2McCoy7Member]

Have you guys ruled out asking fellow Linux devs for help? Perhaps the Valve or Feral Devs wouldn’t mind sharing advice on how to best handle this fragmentation issue. They all must have encountered the same road blocks and eventually found a solution. It’s definitely in Valves best interest to share that kind of info and I’m actually surprised they don’t auto send info like that to devs who are new to tackling Linux games.

Glad you have the leaks under control. Too bad there wasn’t a global command to reset the memory (virtual reboot) to fix the fragmentation issue. Autosaves would be a perfect time to implement some sort of fix like that.

I’ll test the new build. Good luck! I definitely appreciate your efforts.

[2McCoy7Member]

Lascha wrote:

Some fixes were made for the linux build to prevent some fragmentation when a save goes off. It’s not perfect, but it definitely helped in my tests. The new build is: Steam Library > AoW3 > Properties > BETAS > open_beta

^^There are two Beta branches. I am assuming it is the “OSX and Linux fixes” branch I am supposed to select? If I assume correctly when I press play nothing happens (no launcher). When I revert back to default the launcher works again…

Sorry for the late response…knowing I would have to start a new game was depressing so I spent some time trying to get the Map Editor functional in wine. Almost but not quite. It freezes when you try and create a new map…oh well.

Anyways hopefully someone is able to get that executable up and running. Tomorrow is my last day off and won’t be able to test for a week once I’m back at work

[LaschaModerator]

Nope not that branch. It’s the open_beta latest awesomeness branch.

[2McCoy7Member]

Blackscreen freeze on end turn using beta branch. This time it was unique in that it produced a .dmp file but did not actually ctd. Definitely the first time this has happened in all my games.

I emailed the log and dmp files and as you can see at the end of the log file it spams the memory leak. I will keep playing to confirm but think this is the least stable build so far. Encountered the blackscreen in less then 1 1/2hrs of play

[LaschaModerator]

Thanks for testing McCoy. Looks like it’s still related to out of memory. Any chance you could help keep an eye on the memory usage while playing so we can get an understanding of ‘what’ else could cause the memory usage to increase so much?

[2McCoy7Member]

Name a program and a terminal command for that program that will generate a logfile with information that will be useful to you. I have manually observed my memory and it does NOT creep up. It stays at the same which is why the fragmentation makes complete sense.

^^It seems if you give me a command that returns a log with a timestamp with info that is relevant to memory fragmentation then you can compare the times on the memory log and the game log to find the issues

That said this is my last day off so no rush. I’m done testing for at least a week

[tezeriuszMember]

@2mccoy7: They using mempool we can’t see how system memory is affected (one big allocation at start).

I got nice hangout with latest beta – 15931.
Attached stuff as usual – most likely you have deadlock.

@lascha: Do you have still problems with mem fragmetation? Or there are normal memleaks? Maybe extra thread that will print memory usage/state of mempool isn’t that bad idea?

Attachments:

1. launcher0425_1318.txt
2. pstacks2.txt
3. log0425_1153.txt

[LaschaModerator]

You can keep track of the memory with CTRL+ALT+M hotkeys in-game (while in debug mode).

At the bottom of the list you’ll see total memory. Just keep an eye on it while playing and see what action increases it permanently. In that case there’s a mem-leak. For example, I noticed that CTRL+S (quicksave) would increase the memory usage permanently.
That was fixed in last patch. Another time I noticed moving a unit over the map (long distance) would leak memory, fixed as well. Reports like this would really help.

You’ll also see we have a specific memory heap for the World simulation (stuff on the strategic map & tactical map). Unline on Windows, we disabled it for other systems on Linux & Mac because these OS’es any outside middleware can hook into our pool and corrupt it (for example a graphics driver). On the World heap there’s no such outside middleware so that’s fine.

[Author]

Posts