Page: 0 ... 5 ... 10 ... 15 ... 20 ... 25 ... 30 ... 35 ... 40 ... 45 49 50 51 52 53 54 55 56 57
Dump Server
Date: 13/4/2006
When you write client/server network software like an email client it's inevitable that some user, somewhere, is going to try and connect to a server that your app doesn't work with. And so it's normal to include logging in your app to capture the conversation between the client and server to a file. Scribe has an option to dump all the bytes to a log file in hex format, e.g.:
Read = 18
00000000	2B 4F 4B 20 48 65 6C 6C 6F 20 74 68 65 72 65 2E  +OK Hello there.
00000010	0D 0A                                            ..
Write = 11
00000000	55 53 45 52 20 74 65 73 74 0D 0A                 USER test..
Read = 24
00000000	2B 4F 4B 20 50 61 73 73 77 6F 72 64 20 72 65 71  +OK Password req
00000010	75 69 72 65 64 2E 0D 0A                          uired...
Write = 11
00000000	50 41 53 53 20 74 65 73 74 0D 0A                 PASS test..
Read = 16
00000000	2B 4F 4B 20 6C 6F 67 67 65 64 20 69 6E 2E 0D 0A  +OK logged in...
Write = 6
00000000	53 54 41 54 0D 0A                                STAT..
Read = 18
00000000	2B 4F 4B 20 39 33 32 20 32 35 30 34 39 31 31 39  +OK 932 25049119
00000010	0D 0A                                            ..
Write = 6
00000000	55 49 44 4C 0D 0A                                UIDL..
Read = 1024
00000000	2B 4F 4B 0D 0A 31 20 55 49 44 39 36 30 34 2D 31  +OK..1 UID9604-1
00000010	30 36 37 36 31 30 34 33 35 0D 0A 32 20 55 49 44  067610435..2 UID
00000020	39 36 30 36 2D 31 30 36 37 36 31 30 34 33 35 0D  9606-1067610435.
00000030	0A 33 20 55 49 44 39 36 31 30 2D 31 30 36 37 36  .3 UID9610-10676
00000040	31 30 34 33 35 0D 0A 34 20 55 49 44 39 36 31 31  10435..4 UID9611
...etc etc

Thats nice by itself but it really only lets you look at the data. What would be really nice is to actually be able to run your software locally under debug mode with something that made the same responses as the server in the field. So thats what "DumpServer" does, it reads in the log file and pretends to be the server in the log file, sending all the bytes under each read to the client. And it has a nice GUI as well:

Now I can fire up Scribe and get it to interact with the "server" just like the user does and hopefully find the point where things go off the rails in my own code.

How cool is that?
(1) Comment | Add Comment

Review: Belkin KVM (F1DL102U)
Date: 10/4/2006
Product: Belkin 2 port KVM (F1DL102U)

Score: 1/5

  • Cheap (paid ~$70 AUD)
  • Neat "all in one" cables.
  • No software support for Linux or Mac.
  • Hangs regularly
  • No reset button
After buying a Mac mini I needed a way to switch between my PC and the new Mac, so I did a little research and bought the Belkin 2 port USB KVM through a local retailer. After getting it home and plugging it in it all worked as expected. The software that allows soft switching via the keyboard only works with windows so I didn't even bother with that and I've only used the hardware switch on top of the unit. The KVM has two lights on the switch that light up when one or the other computer has the "focus". The cable system is tidy and comes with the unit, no need to buy extra cables. I like how that you have a single cable that has a bunch of fly leads on the end for VGA/USB/audio. That design is just brilliant, gets rid of a bunch of cable clutter.

Thats where the good news stops I'm afraid. Because the unit's actual job of switching between 2 computers is well lacking in a number of fundamental ways. First and foremost is the "blinking lights" problem that comes in a couple of different flavours. First version is that BOTH active lights come on at "half strength" and the KVM refuses to switch to the PC, well it does switch, just for a second and then switches back to the Mac. This usually happens when the Mac is on and selected by the KVM, and the PC boots in the other channel but it's not limited to that scenario. The other flavour is just delightful, the KVM can't decide which computer is active so it switches from one to the other every second. Back and forth endlessly switching. Both machines are on and functioning normally but the KVM just keeps switching as fast as it can. For either of these maladies the only solution is to unplug it from the computers to power cycle it and then plug it back in again. As it gets power off the USB ports, you only need to plug the USB plug on each computer. And this is happening a few times a week, so it's not some rare hardware glitch that I can ignore.

Then there is the issue of screen detection that I talked about in a previous entry where it seems that when Windows can't detect what monitor is attached, i.e. when the KVM is switched to the other computer, it decides to pick a refresh rate out of thin air. Which generally causes the average LCD to give an "Input Out Of Range" type error. Sometimes windows even chooses a different screen resolution as well. Now one would think, even assume, that Windows would be able to "Just Use The Friggin Display Options I Set(TM)" but that would be far too easy and would put whole swathes of MSCE's out of work so thus it was decreed that Windows should suck and not use the right settings. However that might not be the fault of the KVM, I expect that smarter KVM engineers would be able to come up with a solution to the windows problem. Belkin should find some of these smarter engineers and employ them. If you find yourself in this ugly sitution I heartily suggest using the RefreshForce to reign in Window's taste for trying new refresh rates.

So all in all, not much to recommend it. I'd try anything else first, and only get this unit if all else fails.
(2) Comments | Add Comment Scam
Date: 4/4/2006
If you ever get an invite from delete it immediately. You really don't want to get sucked in the vortex of spam that creates. The main problem with is that they can send you premium SMS messages that cost you money to receive, and you can get these SMS messages without ever signing up with them (e.g.). can also spam all you IM buddies inviting them to join merely by giving them your IM user id (e.g.).

Not only does run an incredibly dishonest spam business but they also threaten you with legal action if you expose their true nature.

So here I am exposing their true nature :D
(0) Comments | Add Comment

Site Traffic Spike
Date: 1/4/2006
Traffic was up 22% this month over the previous busiest month. But it wasn't only just popular with regular visitors, but it was popular with the bots as well. With robots now accounting for over half of all hits on this site. Maybe in 10 years time no-one will read memecode except millions of automated software.
(0) Comments | Add Comment

Memory Leak Removal
Date: 1/4/2006
With the code I talked about in the previous blog entry I've been able to start removing lots of memory leaks from Scribe. I've added the leak check dumping that I talked about. It's so easy to use I'm just amazed and it's so low impact on the software during use. It was inspired somewhat by valgrind's leak check tool, but this is (much?) better in that it's runtime impact is much lower than valgrind. Of course valgrind has memcheck, which I can't compete with.

However there is some work ahead, because to clear the app of leaks you have to exercise the code over and over to get coverage, i.e. all of the paths through the code getting executed. But now that the tools are so good, I'll leave it running for the next few weeks and get the codebase nice and tight with memory. I also want to reduce the block count, as lots of small blocks of memory are inefficent, in that each block has a number of bytes in overhead and takes time to allocate and free. And beyond that it also slows application shutdown because if the app is swapped out to disk and you shut it down, lots of small memory blocks take much longer to free than a few large blocks. Each block has to be read/written to free it, and if it has to be loaded from virtual memory first it takes longer. Yeah? The number one offender in the billions of tiny blocks of memory is the XML parser. So that'll get some string pooling options like the hash table.
(1) Comment | Add Comment

Memory Usage Optimization
Date: 29/3/2006
I added an optional memory usage debugger to Lgi over the last 2 days, just in case that Btree memory hog thing turns up again. What it does is override the normal new/delete/malloc/free functions to all go through 2 functions lgi_malloc and lgi_free. These functions add some tracking data onto the start of each block and also keep everything in a big list as each block has a next/prev pointer making it a doubly linked list (easy O(1) insert/delete). So when lgi_malloc is called it allocs memory enough for the data and the stats block. The stats block has the size of the data and a stack trace. The stack trace stores the EIP pointer for the top 8 functions on the call stack. This means later on we can track down who allocated the memory. Then it adds the block to the linked list. lgi_free obviously just removes the block from the list and calls "free" on it.

Thus at any point in the execution you can call LgiDumpMemoryStats which walks the current linked list of memory blocks and dumps it to file "stats.mem". While it does that it also looks up the module and source file/line information for each EIP in the stack trace for the block. Thus the dump file looks like this:
Block 65536 bytes
	Lgid.dll C:\Code\Lgi\src\common\General\GContainers.cpp:1434
	Lgid.dll C:\Code\Lgi\src\common\General\GContainers.cpp:1644
	Lgid.dll C:\Code\Lgi\src\common\General\GContainers.cpp:1878
	Lgid.dll C:\Code\Lgi\src\common\Gdc2\Font\GFontCodePages.cpp:1544
	Lgid.dll C:\Code\Lgi\src\common\Gdc2\Font\GFontCodePages.cpp:817

Block 64 bytes
	Lgid.dll C:\Code\Lgi\src\common\General\GContainers.cpp:1644
	Lgid.dll C:\Code\Lgi\src\common\General\GContainers.cpp:1878
	Lgid.dll C:\Code\Lgi\src\common\Gdc2\Font\GFontCodePages.cpp:1544
	Lgid.dll C:\Code\Lgi\src\common\Gdc2\Font\GFontCodePages.cpp:817

Block 272 bytes
	Lgid.dll C:\Code\Lgi\src\common\General\GContainers.cpp:600
	Lgid.dll include\common\GContainers.h:103
	Lgid.dll C:\Code\Lgi\src\win32\Gdc2\Gdc2.cpp:2191
	Lgid.dll GApp15::GApp15+0x16
	Lgid.dll C:\Code\Lgi\src\common\Gdc2\15Bit.cpp:71

So you can see that this is interesting but it's also not very easy to digest. For example the dump from Scribe just after starting up has 140,000 or so blocks taking up 64mb of dump file. Good luck sorting through that!

So of course you need something to summerize that into "information" instead of "data". And that tool is built on to LgiIde:

As you can see this is much easier to digest. You can see some solid stats on where the memory is being allocated and where some optimization is needed. The great thing about having the stack trace is that we can jump over the container classes and get the real point that the allocation happened. I've already identified some areas for improvement in Scribe, in particular the XML parser and the storage sub-system.

Initially this system put quite a bit of extra load on the program while it was running but I've managed to get it to a point where you barely notice the performance overhead during runtime, although obviously you use more memory than normal. So it can be left on during normal debugging so that you can at any time dump the current memory usage and analyse it in LgiIde.

I intend to also make this functionality able to dump leaked memory at program exit (using the atexit method).

The other good thing about this code is that it's not too closely tied to Lgi, it should be possible to separate it out and use it in any software. With a little work.

Update: atexit is what I was looking for getting the memory leak detection going.
(1) Comment | Add Comment