Date: 29/3/2006
| I added an optional memory usage debugger to Lgi over the last 2 days, just in case that Btree memory hog thing turns up again. What it does is override the normal new/delete/malloc/free functions to all go through 2 functions lgi_malloc and lgi_free. These functions add some tracking data onto the start of each block and also keep everything in a big list as each block has a next/prev pointer making it a doubly linked list (easy O(1) insert/delete). So when lgi_malloc is called it allocs memory enough for the data and the stats block. The stats block has the size of the data and a stack trace. The stack trace stores the EIP pointer for the top 8 functions on the call stack. This means later on we can track down who allocated the memory. Then it adds the block to the linked list. lgi_free obviously just removes the block from the list and calls "free" on it.
Thus at any point in the execution you can call LgiDumpMemoryStats which walks the current linked list of memory blocks and dumps it to file "stats.mem". While it does that it also looks up the module and source file/line information for each EIP in the stack trace for the block. Thus the dump file looks like this:
Block 65536 bytes
Lgid.dll C:\Code\Lgi\src\common\General\GContainers.cpp:1434
Lgid.dll C:\Code\Lgi\src\common\General\GContainers.cpp:1644
Lgid.dll C:\Code\Lgi\src\common\General\GContainers.cpp:1878
Lgid.dll C:\Code\Lgi\src\common\Gdc2\Font\GFontCodePages.cpp:1544
Lgid.dll C:\Code\Lgi\src\common\Gdc2\Font\GFontCodePages.cpp:817
Block 64 bytes
Lgid.dll C:\Code\Lgi\src\common\General\GContainers.cpp:1644
Lgid.dll C:\Code\Lgi\src\common\General\GContainers.cpp:1878
Lgid.dll C:\Code\Lgi\src\common\Gdc2\Font\GFontCodePages.cpp:1544
Lgid.dll C:\Code\Lgi\src\common\Gdc2\Font\GFontCodePages.cpp:817
Block 272 bytes
Lgid.dll C:\Code\Lgi\src\common\General\GContainers.cpp:600
Lgid.dll include\common\GContainers.h:103
Lgid.dll C:\Code\Lgi\src\win32\Gdc2\Gdc2.cpp:2191
Lgid.dll GApp15::GApp15+0x16
Lgid.dll C:\Code\Lgi\src\common\Gdc2\15Bit.cpp:71
So you can see that this is interesting but it's also not very easy to digest. For example the dump from Scribe just after starting up has 140,000 or so blocks taking up 64mb of dump file. Good luck sorting through that!
So of course you need something to summerize that into "information" instead of "data". And that tool is built on to LgiIde:
As you can see this is much easier to digest. You can see some solid stats on where the memory is being allocated and where some optimization is needed. The great thing about having the stack trace is that we can jump over the container classes and get the real point that the allocation happened. I've already identified some areas for improvement in Scribe, in particular the XML parser and the storage sub-system.
Initially this system put quite a bit of extra load on the program while it was running but I've managed to get it to a point where you barely notice the performance overhead during runtime, although obviously you use more memory than normal. So it can be left on during normal debugging so that you can at any time dump the current memory usage and analyse it in LgiIde.
I intend to also make this functionality able to dump leaked memory at program exit (using the atexit method).
The other good thing about this code is that it's not too closely tied to Lgi, it should be possible to separate it out and use it in any software. With a little work.
Update: atexit is what I was looking for getting the memory leak detection going.
|
