Page: 0 ... 5 ... 10 ... 15 ... 20 ... 25 ... 30 ... 35 ... 40 ... 45 50 51 52 53 54 55 56
Memory Leak Removal
Date: 1/4/2006
With the code I talked about in the previous blog entry I've been able to start removing lots of memory leaks from Scribe. I've added the leak check dumping that I talked about. It's so easy to use I'm just amazed and it's so low impact on the software during use. It was inspired somewhat by valgrind's leak check tool, but this is (much?) better in that it's runtime impact is much lower than valgrind. Of course valgrind has memcheck, which I can't compete with.

However there is some work ahead, because to clear the app of leaks you have to exercise the code over and over to get coverage, i.e. all of the paths through the code getting executed. But now that the tools are so good, I'll leave it running for the next few weeks and get the codebase nice and tight with memory. I also want to reduce the block count, as lots of small blocks of memory are inefficent, in that each block has a number of bytes in overhead and takes time to allocate and free. And beyond that it also slows application shutdown because if the app is swapped out to disk and you shut it down, lots of small memory blocks take much longer to free than a few large blocks. Each block has to be read/written to free it, and if it has to be loaded from virtual memory first it takes longer. Yeah? The number one offender in the billions of tiny blocks of memory is the XML parser. So that'll get some string pooling options like the hash table.
(1) Comment | Add Comment

Memory Usage Optimization
Date: 29/3/2006
I added an optional memory usage debugger to Lgi over the last 2 days, just in case that Btree memory hog thing turns up again. What it does is override the normal new/delete/malloc/free functions to all go through 2 functions lgi_malloc and lgi_free. These functions add some tracking data onto the start of each block and also keep everything in a big list as each block has a next/prev pointer making it a doubly linked list (easy O(1) insert/delete). So when lgi_malloc is called it allocs memory enough for the data and the stats block. The stats block has the size of the data and a stack trace. The stack trace stores the EIP pointer for the top 8 functions on the call stack. This means later on we can track down who allocated the memory. Then it adds the block to the linked list. lgi_free obviously just removes the block from the list and calls "free" on it.

Thus at any point in the execution you can call LgiDumpMemoryStats which walks the current linked list of memory blocks and dumps it to file "stats.mem". While it does that it also looks up the module and source file/line information for each EIP in the stack trace for the block. Thus the dump file looks like this:
Block 65536 bytes
	Lgid.dll C:\Code\Lgi\src\common\General\GContainers.cpp:1434
	Lgid.dll C:\Code\Lgi\src\common\General\GContainers.cpp:1644
	Lgid.dll C:\Code\Lgi\src\common\General\GContainers.cpp:1878
	Lgid.dll C:\Code\Lgi\src\common\Gdc2\Font\GFontCodePages.cpp:1544
	Lgid.dll C:\Code\Lgi\src\common\Gdc2\Font\GFontCodePages.cpp:817

Block 64 bytes
	Lgid.dll C:\Code\Lgi\src\common\General\GContainers.cpp:1644
	Lgid.dll C:\Code\Lgi\src\common\General\GContainers.cpp:1878
	Lgid.dll C:\Code\Lgi\src\common\Gdc2\Font\GFontCodePages.cpp:1544
	Lgid.dll C:\Code\Lgi\src\common\Gdc2\Font\GFontCodePages.cpp:817

Block 272 bytes
	Lgid.dll C:\Code\Lgi\src\common\General\GContainers.cpp:600
	Lgid.dll include\common\GContainers.h:103
	Lgid.dll C:\Code\Lgi\src\win32\Gdc2\Gdc2.cpp:2191
	Lgid.dll GApp15::GApp15+0x16
	Lgid.dll C:\Code\Lgi\src\common\Gdc2\15Bit.cpp:71

So you can see that this is interesting but it's also not very easy to digest. For example the dump from Scribe just after starting up has 140,000 or so blocks taking up 64mb of dump file. Good luck sorting through that!

So of course you need something to summerize that into "information" instead of "data". And that tool is built on to LgiIde:

As you can see this is much easier to digest. You can see some solid stats on where the memory is being allocated and where some optimization is needed. The great thing about having the stack trace is that we can jump over the container classes and get the real point that the allocation happened. I've already identified some areas for improvement in Scribe, in particular the XML parser and the storage sub-system.

Initially this system put quite a bit of extra load on the program while it was running but I've managed to get it to a point where you barely notice the performance overhead during runtime, although obviously you use more memory than normal. So it can be left on during normal debugging so that you can at any time dump the current memory usage and analyse it in LgiIde.

I intend to also make this functionality able to dump leaked memory at program exit (using the atexit method).

The other good thing about this code is that it's not too closely tied to Lgi, it should be possible to separate it out and use it in any software. With a little work.

Update: atexit is what I was looking for getting the memory leak detection going.
(1) Comment | Add Comment

Scribe v1.89 Test1 Release Notes
Date: 28/3/2006
Well it's done. I'm not making any promises on code quality or memory usage though. I've seen some problems in the few days of testing the final codebase but no bugs, just optimization issues. However thats mostly tied to the use of the new incremental bayesian word database updates. This feature is off by default, due to concerns about speed but you can turn it on in the bayesian settings dialog. It updates the word databases on the fly as you read your email and bin stuff as spam.

It's a little slow at times on older hardware, on my 1.4ghz/512mb machine it gets in the way of usability but on the 3.06ghz/2gb machine it's not noticable at all. The backend of the word DB's is currently a Btree but I'm thinking of playing with Sqlite as the backend and see what the performance is like.

Sometimes it uses upwards 40mb of memory too, but I havn't reproduced this yet. Usually it uses about 3-4mb. Let me know your experiences with the new bayesian filter.

You need to rebuild your word lists at least once with v1.89 to initialize the new format files, and then either use the incremental updates or the regular way of rebuilding the lists every so often.

There is also a number of API changes that have bubbled changes throughout the code base, and while it's been running well for me in the last month it could mean that rarely used functions are still broken. It's a monumental task to try every menu option, command, filter condition/action and type of email and connection available in the software. So I'm relying somewhat on the user base for feedback.

I ripped out the old XML parser which has a side effect of meaning the format of the lgi.conf has changed a little. All the lgi.conf options that had a "." in the element name should be converted to "-", so "font.fixed" becomes "font-fixed" etc.

Looking forward to your comments.
(8) Comments | Add Comment

PayPal Robots
Date: 27/3/2006
PayPal seem to be employing robots to do their support work. Their menus are displaying incorrectly in Firefox so I thought I'd be a good net citizen and report the problem to them. Firstly there is no section in the support options for "website". Which was the first ominous sign that this wasn't going to go well. So I send off a brief but polite description of the problem and hope for the best:

Hello my name is Brian, I will be happy to assist you with your question.

In review you will want to clear cache and cookies and log in using

If you are experiencing problems with our website that have not been 
encountered before, it is likely that you have a corrupted cookie. The 
easiest way to clear cookies is to remove them from the PC rather than the 

Follow these simple steps to clear your cookies:

Yeah ok, just to placate you I'll do all that. So I clear the cache and all the cookies and re-login to PayPal. Unsurprisingly the same problem appears and I email them back:

I've cleared my cache and cookies and logged in again to 
and I'm still getting the same issue.
Attached is a screen shot of the problem.

Then I get this back from them:

Thank you for contacting PayPal.

I was unable to open the attachment you had sent due to security 
precautions. Please contact PayPal in text form at and we will be happy to assist you 

Thank you for being part of the PayPal community.  

Community satisfaction and your experience with PayPal are very important 
to me. You may receive a survey from our third party vendor, Benchmark 
Portal, about the service you received.

Awesome! I will club them to death with my PNG of malware!

It was a PNG image file... it's not going to bite. PNG files can't have 
virus' or malicious payloads.
Screw this, if you don't want help fixing your own damn site then fine I 
can spend my time elsewhere.

That reset the robot:

Hello my name is Mahak, I will be happy to assist you with your question 
regarding account problem.

I do apologize for the inconvenience that you are experiencing with this 
situation. To better assist you please be specific about any one of your 
transaction that you are looking for.

Thank you for being part of the PayPal community.  

Community satisfaction and your experience with PayPal are very important 
to me. You may receive a survey from our third party vendor, Benchmark 
Portal, about the service you received.

Haha. So I guess I should get used to the bustedness of, all I have to do is repeat to myself "I do not care. I do not care. I do not care."
(0) Comments | Add Comment

Date: 23/3/2006
The bug in the Btree code that I was using valgrind to find wasn't there. It was merely some old debugging code that was getting in the way. Doh!

So now there is no reason not to release v1.89-test1. I just need to clean up some bits and peices and we're good to go.

The effort spent on valgrinding the bayesian code did turn up some bugs which I've fixed. So it wasn't a fruitless waste of time. But I'm sure it would've been nicer to have a release out last week eh?
(1) Comment | Add Comment

Date: 22/3/2006
Lgi now builds and runs on cygwin with GCC. I tested this by building the IDE with Visual C++ and then loading the XML project files that LgiIde uses, and building it again with gcc. After a while I got it to a point where the GCC version doesn't crash. It's not fully tested but at least something is working. Most of the issues were with using mis-matched DLL's, ie some that were compiled with VC and some were GCC. This obviously "Does Not Work(TM)".

A release of Lgi will be out shortly.
(0) Comments | Add Comment