Scribe Gmail Support Update
Google has switched off so called "insecure" authentication methods, including "PLAIN", which Scribe relied on.

Today I got that draft implementation working enough to be able to login to the Gmail IMAP server using OAuth2. It does take you via a detour into your default browser but in the end it works. It's also quite the hack job at this point so I want to clean up the code and make the error handling at least very verbose. I will be making a release in the next few days off the stable branch that includes functional OAuth2 support.

It has been argued that Google's motivation in doing this is to force people onto the web and out of installed desktop / mobile clients, for the purpose of making Gmail's web UI more palatable. I remain unconvinced about this argument in that there seems to be another reason that makes more sense to me. OAuth2 doesn't require the client to store the plain text password. Thus reducing the possible points of failure for security. Most installed clients are bad at storing persistent account credentials in a fully secure fashion. So by removing that attack vector it could be argued that Google is doing App authors a favour.


I still find OAuth2 quite needlessly complicated. And it's very dependent on the client having lots of pre-configured knowledge about the server it's authenticating with. I mean for every service that Scribe will support OAuth2 authentication I have to have a unique ClientID and ClientSecret, as well as the token URI and access URI... none of which are "discoverable" on the fly, but hard coded in. So you can't connect Scribe to some arbitrary new service that supports OAuth2. I have to manually create support each new service. I don't yet understand how that is a reasonable state of affairs.

Tags: scribe oauth2
Scribe Gmail Support
So Google have changed the authentication support for Gmail again, breaking the ability for Scribe to log in to any of the Gmail related services. Which means I'm going to have to put aside the v2.1 work temporarily to work on getting one of the support authentication methods working in Scribe.

The options are: I'll try and keep this post updated with progress on that front.

The list of all official SASL mechanisms is here. PLAIN-CLIENTTOKEN is not mentioned at all. I wonder what implements that? Also of note, both XOAUTH and XOAUTH2 are marked "OBSOLETE". Nice one Google, supporting only obsolete, undocumented or non-functional authentication methods.

Tags: scribe
Scribe v2.1 Update
It may look like nothing is happening but really, there is. I have been toiling away on features and bug fixes in the v2.1 "trunk". Also of note, I've removed all the ANSI api support in the Windows implementation of LGI which means that any hope of running it on Windows 9x is gone for good. XP has been the official minimum for a while now, and now it's cemented in the code as well.

Tags: scribe
The Current State of Scribe
Maybe it's time to just bundle OpenSSL in Scribe? I'm thinking of statically linking it so that it's always available and there will never be any trouble with missing or mis-matched libraries. The downside is the download size will blow out a bit. Over 2 MiB for the Windows build, maybe even 3. Recently I've had a very bad time with people have all sorts of weird crashes, odd behaviour and connection issues. And it seems that these things are next to impossible to reproduce. Maybe it's the target market changing on me as the profile of the client gets higher.

One of those odd behaviours is Scribe not saving it's settings between sessions. I suspect this has to do with the install folder being write only, but the software thinking it's in portable mode (i.e. store settings in the install folder). I'd love for someone to be able to reproduce that and tell me how.

So I'm going to look into ways of avoiding pitfalls. I'm starting to think that making automatic crash reporting default to "on" is for the best. I'm simply not getting enough data to respond to problems.

The HTML editing functionality is coming along nicely. I managed to send quite a few HTML messages over the last few weeks. But it's still pretty flakey. And also there are some thorny issues to sort out. However it's a lot better than even a month ago so there is hope.

Also I finally fixed an old crash in the GWindow destructor on the Linux/GTK2 build. The other main sticking point there is the popup handling under the GTK2 platform isn't great. But I didn't see the point in tackling that when the software was still crashing all the time.

Edit: So I think I've worked out the procedure for the not being able to save your settings bug: Yeah I'm going to have to make sure that Scribe is aware that during startup that it may have ADMIN permissions, and not to assume it will be able to write to anything in the Program Files folder tree. However it should totally take advantage of having ADMIN rights to setup the registry for being the default client / registered client.

Tags: scribe
Scribe v2.1
So the v2.0 builds are basically stable as far as I can tell. Although there is some issues with some people connecting to SSL servers and one report of a crash at startup on MacOSX 10.10 (which I'm investigating). In the light of that I'm shifting my attention to the features in the v2.1 branch. I want to go through the things I'm aiming to get into that release: If you think I'm missing something really obvious that chime in either on email or in the comments on this post.

I'm looking to get the first build out sometime in Feburary.

Tags: scribe
"Invalid HTTP Request Headers" in Firefox
For most of 2014 I've been using Firefox because I feel it's doing a better job of looking after my privacy. However it's not without it's cost. For instance today I found some pretty quirky behaviour in the Mac build of Firefox v34.0.5:

Basically on this site it's dropping the leading "G" of the "GET" request. To make matters worse the problem is intermittent. I can reproduce with a clear cache. Internally Firebug is reporting the correct packet, but Wireshark shows the missing byte.

Tags: firefox
Scripting Debugger

And extern library function support!

Tags: scribe scripting


16/12/2014 10:18am

Portable OpenSSL on Mac OS X
So it's been brought to my attention that Scribe's self install of OpenSSL on Mac doesn't actually work in v2.0.x, so the last week I've been looking into what it takes to do that.

The main issue is that load paths, are by default, absolute and hardcoded. Which works against you when you want a portable install of something. Secondly the OpenSSL .dylib is made of 2 libraries; libssl and libcrypt. And when you dlopen the first (libssl) the system needs to be able to find libcrypt. The link is embedded in the Mach-O image as a LC_LOAD_DYLIB segment. You can see that by issue the command:
otool -l ./libssl.1.0.0.dylib
Part of the output you'll get:
cmdsize 64
   name /usr/lib/libcrypto.1.0.0.dylib (offset 24)
   time stamp 2 Thu Jan  1 10:00:02 1970
That absolute path is what is causing the dlopen of libssl to fail. So we need to fix that. And the tool to do it 'install_name_tool', which is provided from Apple. Now I had two problems. Firstly what do you change the path to? And how do you actually get the tool to change the path?

Firstly Apple has several ways of locating shared libraries:
  1. @executable_path : relative to the main executable
  2. @loader_path : relative to the referring binary
  3. @rpath : relative to any of a list of paths.
What I wanted is the path to be relative to the executable, so I want:

But how to call install_name_tool to actually do that? Well to start with I tried:
install_name_tool -change /usr/lib/libcrypto.1.0.0.dylib @executable_path/libcrypto.1.0.0.dylib ./libssl.1.0.0.dylib
Which is actually the right command. But because the new string is longer than the old string the command fails (silently I might add). The trick to getting it to work is to add the linker command:
to the CFLAG variable in the OpenSSL root Makefile. This causes the load segments to have extra padding bytes so that you can change the paths as much as you want later. Now after rebuilding the OpenSSL dylibs you can reissue the install_name_tool command and the link path changes:
MacOS matthew$ otool -l ./libssl.1.0.0.dylib
cmdsize 64
   name @executable_path/libcrypto.1.0.0.dylib (offset 24)
Then I could dlopen 'libssl.1.0.0.dylib' and it would pull in the right libcrypt file (both as sitting in the same folder as the executable).

I'm putting this here so I don't forget how to do this next time. Ha.

Tags: dylib openssl


Yakov Eremin
10/04/2015 8:09pm
Yakov Eremin
12/04/2015 2:14am

Calling Msys and MingW commands from outsite their environment (i.e. CreateProcess)
So I wanted to check in on how Lgi apps built on Msys / MingW these days, particularly to have a look at implementing a GDB wrapper in the LgiIde application so that I can debug stuff directly in Linux without having to drop down to the command line. But never mind that convoluted explanation.

The problem that I initially saw when trying to call Msys's 'make' was this:
0 [main] make.exe" 4408 fhandler_base::dup: dup(some disk file) failed, handle 0, Win32 error 6
/bin/sh: line 1: /c/Code/Lgi/trunk/LgiIde/C:/MinGW/msys/1.0/bin/make.exe: No such file or directory
Then after I started using a different CreateProcess wrapper I got this:
-f: Nothing to be done for `'.
I checked and re-checked that the path pointed to both the Msys and MingW bin directories in make's environment. But it still wouldn't work. From the command line it was fine, but doing a CreateProcess on make failed.

It would appear that Msys make really needs to be run inside a shell of some sort. Even if it's just windows build in cmd.exe. So I ended up with code that looks like this:
CreateProcess("C:\\Windows\\System32\\cmd.exe", "/C make.exe -f", ...);
Which works well enough for me.

Tags: mingw


18/11/2014 9:58am

The Abomination Of Wingdings And The Shortfalls Of Outlook
It's fairly well accepted that the font WingDings should not be used in webpages. Many modern cross platform browsers don't even support it. One of the reasons is that supporting WingDings means supporting creating symbol fonts which requires adding the SYMBOL_CHARSET define to the call to CreateFont instead of the normal ANSI_CHARSET. Due to their cross platform nature they want as much code as possible to be the same regardless of platform. So instances like this where a Windows specific font needs some special handling are often unimplemented to keep things simple. It may even be a strategic design decision made to deliberately weaken the support for it hoping that it's usage would fade away.

In my email client I have two places that I have to special case code just for WingDings. Firstly the above mentioned CreateFont call needs a special flag added so that when rendering HTML email that uses WingDings the correct glyphs are placed on the screen/page. The second place is during conversion from HTML to (unicode) text. Of course the actual bytes to render in the HTML in the document are not unicode or some easily known charset, so you have to convert from "WingDingsCharset" to Unicode using some hacked together table.

Now the question is "Who is creating all this content that uses the WingDings font?"

That's an excellent question dear reader, and the answer is: Outlook. In 2014 Outlook still happily converts:
Into a WingDings HTML font tag containing a single uppercase 'J'. That will be rendered as a smiley face by Outlook and a capital J by most other software. Especially software on Macs and Linux that don't have access to that font by default. Never mind that Unicode has a perfectly functional smiley face character (☺) which could be rendered correctly by pretty much any software these days.

Outlook's character set issues don't stop there though. Take the case that I ran into recently where Outlook was used to send an email containing a URL that included a Unicode character. Now that in itself is somewhat dubious and indeed the URL referred to a Microsoft content server that should've known better than to allow a unicode character in the URL (especially when a perfectly suitable ascii character was available). Now that Unicode URL was specified used proper HTML entity encoding, however the meta charset of the HTML aaaaand the Content-Type of the MIME segment both stated the document was in us-ascii. And well, that Unicode character got converted to garbage at some point. Instead Outlook should make the charset of the HTML "utf-8" or something so that the character can exist in the specified charset. I've taken the approach of assuming the attributes of tags like 'src' and 'href' are in unicode despite the prevailing HTML charset.

Another aspect of the poor charset implementation in Outlook is the tendency to use undefined characters in the specified charset. Take for instance an Outlook generated email says it's using the ISO-8859-1 charset and then goes on to use characters in the 0x80 to 0x9F address space which is according to ISO-8859-1, undefined. Internally Outlook is using the Windows-1252 charset, which DOES define characters in that range. However instead of marking the email as "Windows-1252" it puts the incorrect ISO-8859-1 charset in the headers (they are essentially the same outside of that range). This commonly manifests as smart quotes (0x91 and 0x92) getting rendered incorrectly. Scribe of course has the ability to let the user override the charset which fixes the problem.

Now I'm in no way saying Scribe is perfect and lets all bash on Outlook. But I feel these bugs have been there for over 10 years and still haven't been addressed and something needs to be said. I doubt they will ever fix their software. Which is sad, because it makes the rest of the email software community look bad when we have to deal with the broken emails coming out of Outlook. I try and take a very literal interpretation of the incoming data. So that the user sees the content warts and all. I'm not a fan of sweeping bugs in other clients under my carpet. That has had a very poor history in the web-browser world. Ending in content bugs lasting far longer than they should have.

And if this ever gets back to someone working on the Outlook team: fix thy bugs. It's not even hard... these are really basic problems and it would most likely take a day or less to address.

Tags: email charsets Outlook


13/06/2014 1:56pm

Very high Windows 7 idle hard disk utilization
For months now I've noticed that when I leave my Windows 7 machine alone for over 30 minutes the hard disk starts grinding away relentlessly. Of course if you go to investigate what it's doing all activity ceases immediately. Preventing you from finding out exactly what is going on.

I had these thoughts of possible malware or a virus infection cross my mind and so downloaded and ran Process Monitor and then left the machine sit for 50 minutes. When I came back to it sure enough the hard disk was grinding away. But now I've been logging the activity! So what's the secret process using so much disk I/O?
9:49:49.6359004 PM	MsMpEng.exe	948	ReadFile	
9:49:49.6465209 PM	MsMpEng.exe	948	ReadFile	
9:49:49.6465457 PM	MsMpEng.exe	948	ReadFile	
9:49:49.6540933 PM	MsMpEng.exe	948	ReadFile	
9:49:49.6541138 PM	MsMpEng.exe	948	ReadFile	
9:49:49.6577645 PM	MsMpEng.exe	948	ReadFile	
9:49:49.6600897 PM	MsMpEng.exe	948	ReadFile	
Oh just Microsoft's anti-malware service checking all my files. "Alright... everything seems to be in order here... as you were".

Gear Unicorns
It seems that my standards are too high, because I've been researching models of various pieces of gear that I'd like to buy at some point and I end up realizing that nothing in the market meets my needs, or if it does it's out of my budget.

Unicorn: Findings:
A: The colour laser printer.
  • ~$250.
  • Reasonably priced toner (haha.. ha.. *sigh*).
  • Good print quality.
  • Robust wireless.
  • Full size A4 tray.
  • Reasonable foot print.
Pretty much everything falls flat on the toner price. But if I remove that filter it's still hard to make all the other checkboxes fit at the same time.
At this point I've given up for a while.
B: The still / video camera.
  • Great DSLR style stills.
  • Good low light performance.
  • Longish zoom... x5.. x8?
  • Fast auto-focus in stills and video modes (inc zooming).
  • <$1000? (Maybe)
  • Smaller is better (might help with longer zoom).
  • Tethered shooting of video.
  • HDMI output.
  • Mic input.
There are a number of micro 4/3rds that kinda come close. And the EOS-M is again, kinda close. But I don't see any of them ticking all the boxes.
Pretty much decided on a Panasonic GH3 for this.
C: Replacement Mac laptop.
  • >= 8GB RAM.
  • USB3.
  • Firewire (I still regularly use a firewire audio interface).
  • At least 1000 vertical lines of display resolution.
  • Upgradable hard disk and RAM (current laptop lasted 7 years because I could fix things myself).
  • Dedicated graphics card.
  • ~$1000
The non-retina mid-2012 models seem to fit this criteria if they have the optional display upgrade, which makes them rare as hens teeth. Which has the side effect of making them much more expensive. Might just wait a while till the used prices drop.
So the old Macbook died an untimely death at my hands and so while I didn't have all the cash to spring for a mid-2012 model I decide that if I dropped the USB3 requirement I could get a much older / cheaper Macbook Pro that still ticked a lot of the boxes.

Or near enough... in fact the display is only 900px high but thats a lot better than the 13" Macbook's display. Ended up getting an early 2008 for $250 + shipping. Probably will need a RAM upgrade but at that price I can afford it.
  • 2.4ghz C2D
  • 2GB presently, 6GB when upgraded
  • 1440 by 900px screen
  • 7200rpm WD black HD (from old machine)
  • 2 x Firewire
  • 2 x USB2
  • Wireless A/G/N
  • GeForce 8600M GT (Dunno how good that'll end up being?)
I'll probably update this as times goes on and I refine criteria.