MemeCode - Blog

Scribe Folder Load Speed

Date: 26/5/2006

It seemed to me that over the years the speed of loading folders in Scribe has got slower and slower. To the point that I grimice inside everytime that stupid progress bar comes up. Well today something very interesting happened. I was testing the mail2 export code (now almost finished!) by exporting the spam folder to a empty mail2 file. Then loading it in a clean install of i.Scribe. And I knew that with about 7000 and something messages, loading that folder should take 10 seconds or so. And you know what. It took about a second. Huh I thought?

So thus the little light went on in my head. I tried it again, still uber fast. I loaded up the dev build of InScribe and.... slooooow. i.Scribe... fast... InScribe... slooooow. Ok so what the hell is going on here?

Firstly I recompiled the dev version in release mode. Just to rule out the different between release and debug builds. That didn't make a whole lot of difference. Then I rebuilt as i.Scribe and that didn't change anything. So I tried clean installs, removing the accounts, trying every different combination until I hit upon the problem.

Everytime I load an email the to and from address are turned into objects that have links back into the contact database. The action of creating these links while loading mail in a folder is somewhat time consuming when done en masse.

The solution? Do it on demand. It then only gets done on a few email at a time, spread out over time so it's almost negligable. But that frees up the folder load to run at full speed, not bogged down with all this searching around in the address book. And man does it fly now. A folder of 7000 odd messages loads in about a second. And a lot of folders that would progress bar me every day now load instantly.

This little edit might break a few things here and there. But in the long run it'll be good for everyone. I'm sure we'll iron out the bugs in no short time with a little feild testing.

Yay for speed.

Update: Well over the last few days where I've got to use the code more and see it working on a different system it's not all as good as I'd hoped. I think the issue is that the disk cache is making any sort of meanful measurement of folder load speed meaningless. When I initially load a folder after system boot, nothing is cached and it's only marginally faster than before. Not instant. However later on after loading things a few times, it's fetching from cache and it's a lot faster. Every now and then windows dumps the cache and it slows down. So I'm not sure how much faster it really is. I expect it's really somewhere in the order 30-50% faster than before. Does anyone know how to profile code that accesses the disk? Is there a way of flushing the disk cache before running your tests?

Comments:

fret
27/05/2006 5:08am

The mail2 export code got a nice speed boost from the folder load running faster. The average transfer rate of new email has gone from ~70/sec to around ~250/sec. Now that's what I'm talkin 'bout.

I've just implemented the Contact export as well. So it's basically finished enough to release. I'll add support for other object types later I think. Otherwise it'll be another week before any feild testing happens.

SnappyCrunch
29/05/2006 11:03pm

Sounds like it'd reduce the memory footprint on load as well.

Hooray for speed!

From:

Email (optional):

(Will be HTML encoded to evade harvesting)

Message:

Remember username and/or email in a cookie. Notify me of new posts in this thread via email.	BBcode: [q]text[/q] [url=link]description[/url] [img]url_to_image[/img] [pre]some_code[/pre] [b]bold_text[/b]

Sitemap