Blog
Page: 0 ... 5 ... 10 ... 15 19 20 21 22 23 24 25 26 27 ... 30 ... 35 ... 40 ... 45 ... 50 ... 55 ... 60 ... 65 ... 70 ... 75 ... 80 ... 85
Scribe v2
Date: 26/11/2008
So over the last week or so I've brought online a new mail storage back end that I'm calling "Mail3". There are already some release notes in the v2 history page about it but I thought I'd talk a little bit about it in the blog. Basically a whole bunch of things are happening in the v2 code base all at the same time.

Firstly I'm really hammering on the memory leaks, trying to get them under control. I've fixed a lot already but the new code that I've written this week still leaks a fair bit. However the leak detection and fixing is working really well so it's almost fun.

Secondly I've finally rid myself of the hideous mime parser from Scribe v1.xx (and early alphas/betas of v2.00). Basically it sat inside the Mail object class and was well not well written. It's probably some of the oldest code in Scribe, dating back to when I sucked more at coding. Anyway I wrote a new MIME parser back in 2006 or something but couldn't work out how to integrate it without breaking everything so it stayed unused for years. Anyway I though while I was doing major earthworks in the code I'd blow the old parser away at the same time. So I did that and moved the ugly MIME flattening code into the Mail2 back end where it belongs.

Thirdly, Mail3. What is Mail3? Well Mail3 is the new mail store back end that I'm working on as the default mail store for Scribe v2. It's primarily a Sqlite database in a folder. I've chosen Sqlite for several reasons; it's small, it's cross platform (both the data format and the code), it's reasonably fast/robust and finally it doesn't have a restrictive license. Also you can also upgrade the format of objects (tables) easily enough. So it fulfills all the requirements for a new mail store. I may still choose to store large attachments outside the Sqlite database, I'm some what undecided on that at this point. Hence having the database in a folder.

So the Mail3 back end is working, you can create the mail store, manage the folder hierarchy, store mail into it etc. But the code doesn't support other types of objects yet, just folders and mail. One of the great things about Mail3 is that finally I'll have a 1:1 mapping between an incoming rfc822 email "image" and how it's stored in the database. So I'll be able to write out an exact (or functionally the same) version of a received email. Which is something that the Mail2 back end couldn't do because it munged the MIME segment hierarchy into a flat list of attachments. The Mail3 back end stores the MIME segments as rows in a table, with ParentId pointers to reconstruct the tree of segments.

Forthly, I've moved the parsing of email from the GUI thread into the worker thread doing the email download.



This should reduce the amount of GUI lockup during mail downloads.

Oh and lastly, the current code base can load more than one mail store at the same time. For instance you can have a mail2 file and a mail3 database open at the same time and they both appear in the folder tree as separate hierarchies of folders. There is only one problem with that, and that is the addressing of folders via paths is going to have to change. At the moment I'm not sure how that is going to play out but basically now that you can have multiple folders loaded things like filters and so on that reference a folder by it's path need to pre-pend a prefix on to the path to define which mail store the folder is in.

A v1 folder name:
/Inbox/Subfolder


Now needs a prefix to determine which store the Inbox belongs to:
/Store2/Inbox/Subfolder


However I'm not totally sure what that 'Store2' part should be. Should it be a complete path and file name to the store? or some arbitrary name that the user calls each mail store? or some UID that Scribe itself generates?

Yeah not so simple (to me at least).
(1) Comment | Add Comment

Svn Repository
Date: 21/11/2008
Something has corrupted my Svn Repository and it's unrecoverable. Most recent backup is 3 months old, and I still have recent checkout's of most of my apps so I don't think I'll actually lose anything other than the hours of sorting out the mess.
  • Checkout old copy.
  • Merge changes from working copy (I love you WinMerge!).
  • Check for new files/folders, add them to svn.
  • Check that it builds.
  • Commit up to date copy.
  • Repeat for all known checkouts (there are a 100 or so!)
*sigh*

Not what I wanted to do today. I plan to make more frequent backups too. In fact I have mostly finished writing a backup application which I'll be launching before year's end. I should've made sure that was live last week, but who knew?
(1) Comment | Add Comment

Valgrind.... MacOSX.... OH YES OH YES OH YES!
Date: 11/11/2008
Someone has done a preliminary port of Valgrind to MacOSX. It's kinda incomplete and buggy but it exists, which means other devs should get on the wagon and get it to production level soon enough.

This is just awesome news. Valgrind is a great tool and seeing it come to Mac is very good news for software quality on the Mac. Mine included.

Last night I was looking at a bug in Scribe/Mac, wishing that I had Valgrind. Well now I do! Well... almost, it's Leopard only at this point so it looks like I'll have to finally upgrade from Tiger. I really love Tiger, in my experience it's a lot more stable than Leopard. So I'm hoping the more recent builds of Leopard are better because I really didn't care for the earlier ones.

Valgrind is something I'd pay for. It's that good.
(0) Comments | Add Comment

Rocket Science
Date: 28/10/2008
There has been two recent events in rocket science that have really advanced the state of the art for private enterprise getting into space. This is exciting on a number of levels, both from the little kid point of view, "wow rockets!" and the more mature, "finally someone other than the government doing it right" point. Generally governments are not good at long term development, and when politics get involved things go sideways (e.g. the shuttle debacle) for a long time. Anyway, here they are:

Armadillo won the level 1 prize of the Lunar Lander Challenge this week. I've been watching Armadillo for a very long time and they finally won something, which is well deserved for all the blood, sweat and tears they put into their rockets. Well done Armadillo!

Secondly, SpaceX put their Falcon1 into orbit, which is a great step forward towards cheap privately funded space access. SpaceX got it right on try number 4 after going so close on the previous 2 attempts. And it's really good to see them get to orbit.

Obviously Scaled Composites won the X Prize a while back, but great things are still happening in the space industry.
(0) Comments | Add Comment

Looking for Scribe v1.88 Users
Date: 24/10/2008
I've had some bug reports from people upgrading between v1.88 to v1.89 or v1.90 where the folder names disappear or other problems with accessing their mail and contacts. I want to look into this issue and sort it out, so I've been running v1.88 in parallel with v2 just to collect some mail and try the upgrade myself. So far so good, the upgrade is perfect, all email, folders and so on usable in later version (although I can't open the upgraded folders in v1.88 anymore).

If you have some mail folders from v1.88 that you haven't opened in a more recent version that doesn't have anything too confidential that you wouldn't mind donating a copy of to improve the Scribe software I'd be interested. Especially if it doesn't upgrade well. To test if it upgrades well or not, MAKE A COPY of your Scribe folder, install a later version (i.e. the latest v1.90 build) into that copied folder and run the new version. If there are problems then you can help! Drop me an email and we'll sort out something in terms of file transfer. If you zip the OLD v1.88 folder you'll probably be able to upload it to a free file sharing site (google it) and post me a URL to download.

That way I can sort out the bugs and stop others from experiencing them.
(1) Comment | Add Comment

Gcc Pain
Date: 22/10/2008
It seems that gcc even in it's more recent versions (v4.2 in my case) has an obnoxious little bug in that 2 classes defined locally in separate C++ files can "intermingle" in the global namespace and cause chaos. e.g.
// File1.cpp
class ClassA
{
    int a;

public:
    ClassA()
    {
         a = 1;
    }

    ~ClassA()
    {
    }
};

void MethodA()
{
    ClassA instance1;
}
And...
// File2.cpp
class ClassA
{
    char *b;

public:
    ClassA()
    {
         b = strdup("Hello");
    }

    ~ClassA()
    {
         free(b);
    }
};

void MethodB()
{
    ClassA instance2;
}
In Visual C++ this is fine, the 2 classes have scope limited to their local C++ file and using them causes no issues.

However in gcc C++ destroying ClassA in one C++ file can result in the destructor from the "other" ClassA to be called, i.e. File1.cpp's trying to destroy the stack variable instance1 and ends up in File2.cpp's ClassA::~ClassA() implementation, which tries to free the string and crashes.

Awesome. And this goes on with no compile or link time errors or warnings. You just find out the hard way at runtime. And if the classes are similar enough you might not even notice the problem until later.

Very insidious indeed. Just beware.

This week I found 2 pairs of classes with the same name, one of which causes a crash, and the other I only noticed because of valgrind complained I was writing off the end of a block of memory. In both cases there was one original copy of the code and I made of a copy of it to implement a different version or mode of operation, each with a local helper class, with the same name. One was the old scripting engine vs the new one, and the other case was the Mail2 backend (Scribe v2) vs the Imap backend.

(Of course I could've missed something really basic with my makefile's or compile options, and I'm happy to take blame if it's me doing something dumb. But I think I'm doing fairly mundane and reasonable things in those areas so I doubt it's my fault in this case)
(0) Comments | Add Comment