MemeCode - Blog

Page: 0 ... 5 ... 10 ... 15 ... 20 21 22 23 24 25 26 27 28 29 ... 30 ... 35 ... 40 ... 45 ... 50 ... 55 ... 60 ... 65 ... 70 ... 75 ... 80 ... 85

Mac Port
Date: 29/12/2008	This week I'm working on the Mac port of Scribe again. And I've almost got native scroll bars intergrated with the Lgi objects and running nicely. Almost. Makes a big difference to the look and feel of the app to have native scroll bars. I will be looking at some other native controls over the next few days. Buttons, checkboxes, radio controls and groups, tab control etc. I'm beginning to wonder if the Mac method of doing scrollbars would work on windows. If it did I could change the architecture of scrolling so that the "Mac" way was native to Lgi and other platforms have to support it or hack it in. The difference between Mac and Windows is that on Mac, the scroll view is one control (i.e. control == HWND) and the content is a separate control, which is then just moved so that the right part of the control is in view. On windows the scrollbars are built into your HWND (if you want them) and so you have just one control/HWND and you draw in a smaller client rectangle, with you drawing offset (by setting the origin of the HDC) by the right amount. I'm wondering if you can plug a separate child control into a HWND with scrollbars, the same as the Mac. Then I could rejig the API to do that all the time. It might even have some efficency gains with scrolling blocks of graphics memory around instead of re-drawing everything. At the moment I've made the Mac way work by reimplementing the GLayout class to shoehorn the Mac API into the way that Lgi is expecting things to work. This means that the GLayout control is owning 2 controls instead of one, and hides that fact from the layers above. But it works so I'll stick with that till I either run into a showstopper issue or I re-write it to be the native way of doing things in Lgi. Hmmmm.... native Mac controls.
(0) Comments \| Add Comment

Linux Ports
Date: 2/12/2008	The Linux ports of my apps have recently become very unstable on current distros. I suspect that the rules for using xlib have changed or something. But xlib was never really thread safe anyway so I've decided to stop avoiding the inevitable rewrite to XCB and just get on with it. My apps aren't about to stop being threaded. So yes, the XCB re-write is happening right now. I expect that it will take a few months before anything really comes of it, but rest assured that something good is happening in Memecode's Linux world. In a year or 3 there will most likely be a similar post about the eventual rewrite of the Mac ports to cocoa and Objective C++. (I'm not really looking forward to that either, I hate big re-writes).
(12) Comments \| Add Comment

Temp files vs in memory.
Date: 28/11/2008	Most application programmers will eventually come to a point where they need to choose whether to store some data in memory or whether to write it to a temporary file. In these sorts of cases you generally don't know how much data you have to handle so the safe thing to do it to write it to disk. However that has it's downsides, it's slower, you have to make sure you clean up the file afterwards and it can get intercepted by all sorts of other processes like anti-virus. I'm currently working on the code that handles email download in Scribe v2 and basically there needs to be a container that stores the email as it's streaming in over the socket. Initially I just used a standard in memory container (Lgi object GStringPipe) which is a FIFO sort of thing that puts it's data in fixed sized blocks. Which is kinda cool because it doesn't have to re-allocate a single block of memory to "grow" it when it runs out of space. It just allocates another block and writes to that. Anyway thats fine for about 99.8% of email. Then there is the edge case. The really insanely large email... 111mb? 456mb? I've seen some whoppers. You usually don't see those on the internet, but in a LAN situation it's not unreasonable to email a large file to someone. You know they are not particularly bandwidth limited. And I've always made it a point to handle that gracefully, both for sending and receiving. In the receiving case I stream into memory until I hit 4mb, then I create a temp file and write all the data in memory to disk, and continue streaming remaining data straight to disk. At the end of the email download I parse it in the worker thread. The cool thing about the MIME parser is that it accepts a generic stream class as input and I can give it a memory stream or a disk stream without the MIME parser caring where the data is stored. The graceful part of that system is that if you have a large email, it's not hanging around in memory, or worse virtual memory, making your system slow down. I have had bugs before where I wrote an email to disk and for a moment closed the file, then a second later attempted to open it and it's gone. Deleted. This happens when some zealous anti-virus process decides that the email is infected. So I do take writing email to disk seriously, and in general I choose to keep the file handle open to avoid losing the data. I'm not particularly worried about infected email considering Scribe's fairly good at detecting executable attachments and blocking or deleting them. The 4mb cut over limit is somewhat arbitrary. But I think it covers the normative case nicely.
(0) Comments \| Add Comment

Scribe v2
Date: 26/11/2008	So over the last week or so I've brought online a new mail storage back end that I'm calling "Mail3". There are already some release notes in the v2 history page about it but I thought I'd talk a little bit about it in the blog. Basically a whole bunch of things are happening in the v2 code base all at the same time. Firstly I'm really hammering on the memory leaks, trying to get them under control. I've fixed a lot already but the new code that I've written this week still leaks a fair bit. However the leak detection and fixing is working really well so it's almost fun. Secondly I've finally rid myself of the hideous mime parser from Scribe v1.xx (and early alphas/betas of v2.00). Basically it sat inside the Mail object class and was well not well written. It's probably some of the oldest code in Scribe, dating back to when I sucked more at coding. Anyway I wrote a new MIME parser back in 2006 or something but couldn't work out how to integrate it without breaking everything so it stayed unused for years. Anyway I though while I was doing major earthworks in the code I'd blow the old parser away at the same time. So I did that and moved the ugly MIME flattening code into the Mail2 back end where it belongs. Thirdly, Mail3. What is Mail3? Well Mail3 is the new mail store back end that I'm working on as the default mail store for Scribe v2. It's primarily a Sqlite database in a folder. I've chosen Sqlite for several reasons; it's small, it's cross platform (both the data format and the code), it's reasonably fast/robust and finally it doesn't have a restrictive license. Also you can also upgrade the format of objects (tables) easily enough. So it fulfills all the requirements for a new mail store. I may still choose to store large attachments outside the Sqlite database, I'm some what undecided on that at this point. Hence having the database in a folder. So the Mail3 back end is working, you can create the mail store, manage the folder hierarchy, store mail into it etc. But the code doesn't support other types of objects yet, just folders and mail. One of the great things about Mail3 is that finally I'll have a 1:1 mapping between an incoming rfc822 email "image" and how it's stored in the database. So I'll be able to write out an exact (or functionally the same) version of a received email. Which is something that the Mail2 back end couldn't do because it munged the MIME segment hierarchy into a flat list of attachments. The Mail3 back end stores the MIME segments as rows in a table, with ParentId pointers to reconstruct the tree of segments. Forthly, I've moved the parsing of email from the GUI thread into the worker thread doing the email download. This should reduce the amount of GUI lockup during mail downloads. Oh and lastly, the current code base can load more than one mail store at the same time. For instance you can have a mail2 file and a mail3 database open at the same time and they both appear in the folder tree as separate hierarchies of folders. There is only one problem with that, and that is the addressing of folders via paths is going to have to change. At the moment I'm not sure how that is going to play out but basically now that you can have multiple folders loaded things like filters and so on that reference a folder by it's path need to pre-pend a prefix on to the path to define which mail store the folder is in. A v1 folder name: /Inbox/Subfolder Now needs a prefix to determine which store the Inbox belongs to: /Store2/Inbox/Subfolder However I'm not totally sure what that 'Store2' part should be. Should it be a complete path and file name to the store? or some arbitrary name that the user calls each mail store? or some UID that Scribe itself generates? Yeah not so simple (to me at least).
(1) Comment \| Add Comment

Scribe v2

Date: 26/11/2008

So over the last week or so I've brought online a new mail storage back end that I'm calling "Mail3". There are already some release notes in the v2 history page about it but I thought I'd talk a little bit about it in the blog. Basically a whole bunch of things are happening in the v2 code base all at the same time.

Firstly I'm really hammering on the memory leaks, trying to get them under control. I've fixed a lot already but the new code that I've written this week still leaks a fair bit. However the leak detection and fixing is working really well so it's almost fun.

Secondly I've finally rid myself of the hideous mime parser from Scribe v1.xx (and early alphas/betas of v2.00). Basically it sat inside the Mail object class and was well not well written. It's probably some of the oldest code in Scribe, dating back to when I sucked more at coding. Anyway I wrote a new MIME parser back in 2006 or something but couldn't work out how to integrate it without breaking everything so it stayed unused for years. Anyway I though while I was doing major earthworks in the code I'd blow the old parser away at the same time. So I did that and moved the ugly MIME flattening code into the Mail2 back end where it belongs.

Thirdly, Mail3. What is Mail3? Well Mail3 is the new mail store back end that I'm working on as the default mail store for Scribe v2. It's primarily a Sqlite database in a folder. I've chosen Sqlite for several reasons; it's small, it's cross platform (both the data format and the code), it's reasonably fast/robust and finally it doesn't have a restrictive license. Also you can also upgrade the format of objects (tables) easily enough. So it fulfills all the requirements for a new mail store. I may still choose to store large attachments outside the Sqlite database, I'm some what undecided on that at this point. Hence having the database in a folder.

So the Mail3 back end is working, you can create the mail store, manage the folder hierarchy, store mail into it etc. But the code doesn't support other types of objects yet, just folders and mail. One of the great things about Mail3 is that finally I'll have a 1:1 mapping between an incoming rfc822 email "image" and how it's stored in the database. So I'll be able to write out an exact (or functionally the same) version of a received email. Which is something that the Mail2 back end couldn't do because it munged the MIME segment hierarchy into a flat list of attachments. The Mail3 back end stores the MIME segments as rows in a table, with ParentId pointers to reconstruct the tree of segments.

Forthly, I've moved the parsing of email from the GUI thread into the worker thread doing the email download.

This should reduce the amount of GUI lockup during mail downloads.

Oh and lastly, the current code base can load more than one mail store at the same time. For instance you can have a mail2 file and a mail3 database open at the same time and they both appear in the folder tree as separate hierarchies of folders. There is only one problem with that, and that is the addressing of folders via paths is going to have to change. At the moment I'm not sure how that is going to play out but basically now that you can have multiple folders loaded things like filters and so on that reference a folder by it's path need to pre-pend a prefix on to the path to define which mail store the folder is in.

A v1 folder name:

/Inbox/Subfolder

Now needs a prefix to determine which store the Inbox belongs to:

/Store2/Inbox/Subfolder

However I'm not totally sure what that 'Store2' part should be. Should it be a complete path and file name to the store? or some arbitrary name that the user calls each mail store? or some UID that Scribe itself generates?

Yeah not so simple (to me at least).

(1) Comment | Add Comment

Svn Repository
Date: 21/11/2008	Something has corrupted my Svn Repository and it's unrecoverable. Most recent backup is 3 months old, and I still have recent checkout's of most of my apps so I don't think I'll actually lose anything other than the hours of sorting out the mess. Checkout old copy. Merge changes from working copy (I love you WinMerge!). Check for new files/folders, add them to svn. Check that it builds. Commit up to date copy. Repeat for all known checkouts (there are a 100 or so!) sigh Not what I wanted to do today. I plan to make more frequent backups too. In fact I have mostly finished writing a backup application which I'll be launching before year's end. I should've made sure that was live last week, but who knew?
(1) Comment \| Add Comment

Valgrind.... MacOSX.... OH YES OH YES OH YES!
Date: 11/11/2008	Someone has done a preliminary port of Valgrind to MacOSX. It's kinda incomplete and buggy but it exists, which means other devs should get on the wagon and get it to production level soon enough. This is just awesome news. Valgrind is a great tool and seeing it come to Mac is very good news for software quality on the Mac. Mine included. Last night I was looking at a bug in Scribe/Mac, wishing that I had Valgrind. Well now I do! Well... almost, it's Leopard only at this point so it looks like I'll have to finally upgrade from Tiger. I really love Tiger, in my experience it's a lot more stable than Leopard. So I'm hoping the more recent builds of Leopard are better because I really didn't care for the earlier ones. Valgrind is something I'd pay for. It's that good.
(0) Comments \| Add Comment

Sitemap