Blog
Scripting Debugger
16/12/2014


And extern library function support!

Tags: scribe scripting

Comments:

fret
16/12/2014 10:18am

Portable OpenSSL on Mac OS X
12/12/2014
So it's been brought to my attention that Scribe's self install of OpenSSL on Mac doesn't actually work in v2.0.x, so the last week I've been looking into what it takes to do that.

The main issue is that load paths, are by default, absolute and hardcoded. Which works against you when you want a portable install of something. Secondly the OpenSSL .dylib is made of 2 libraries; libssl and libcrypt. And when you dlopen the first (libssl) the system needs to be able to find libcrypt. The link is embedded in the Mach-O image as a LC_LOAD_DYLIB segment. You can see that by issue the command:
otool -l ./libssl.1.0.0.dylib
Part of the output you'll get:
    cmd LC_LOAD_DYLIB
cmdsize 64
   name /usr/lib/libcrypto.1.0.0.dylib (offset 24)
   time stamp 2 Thu Jan  1 10:00:02 1970
That absolute path is what is causing the dlopen of libssl to fail. So we need to fix that. And the tool to do it 'install_name_tool', which is provided from Apple. Now I had two problems. Firstly what do you change the path to? And how do you actually get the tool to change the path?

Firstly Apple has several ways of locating shared libraries:
  1. @executable_path : relative to the main executable
  2. @loader_path : relative to the referring binary
  3. @rpath : relative to any of a list of paths.
What I wanted is the path to be relative to the executable, so I want:
@executable_path/libcrypto.1.0.0.dylib

But how to call install_name_tool to actually do that? Well to start with I tried:
install_name_tool -change /usr/lib/libcrypto.1.0.0.dylib @executable_path/libcrypto.1.0.0.dylib ./libssl.1.0.0.dylib
Which is actually the right command. But because the new string is longer than the old string the command fails (silently I might add). The trick to getting it to work is to add the linker command:
-Wl,-headerpad_max_install_names
to the CFLAG variable in the OpenSSL root Makefile. This causes the load segments to have extra padding bytes so that you can change the paths as much as you want later. Now after rebuilding the OpenSSL dylibs you can reissue the install_name_tool command and the link path changes:
MacOS matthew$ otool -l ./libssl.1.0.0.dylib
...
    cmd LC_LOAD_DYLIB
cmdsize 64
   name @executable_path/libcrypto.1.0.0.dylib (offset 24)
...
Then I could dlopen 'libssl.1.0.0.dylib' and it would pull in the right libcrypt file (both as sitting in the same folder as the executable).

I'm putting this here so I don't forget how to do this next time. Ha.

Tags: dylib openssl
Calling Msys and MingW commands from outsite their environment (i.e. CreateProcess)
16/11/2014
So I wanted to check in on how Lgi apps built on Msys / MingW these days, particularly to have a look at implementing a GDB wrapper in the LgiIde application so that I can debug stuff directly in Linux without having to drop down to the command line. But never mind that convoluted explanation.

The problem that I initially saw when trying to call Msys's 'make' was this:
0 [main] make.exe" 4408 fhandler_base::dup: dup(some disk file) failed, handle 0, Win32 error 6
/bin/sh: line 1: /c/Code/Lgi/trunk/LgiIde/C:/MinGW/msys/1.0/bin/make.exe: No such file or directory
Then after I started using a different CreateProcess wrapper I got this:
-f: Nothing to be done for `Makefile.windows'.
I checked and re-checked that the path pointed to both the Msys and MingW bin directories in make's environment. But it still wouldn't work. From the command line it was fine, but doing a CreateProcess on make failed.

It would appear that Msys make really needs to be run inside a shell of some sort. Even if it's just windows build in cmd.exe. So I ended up with code that looks like this:
CreateProcess("C:\\Windows\\System32\\cmd.exe", "/C make.exe -f Makefile.windows", ...);
Which works well enough for me.

Tags: mingw

Comments:

fret
18/11/2014 9:58am

The Abomination Of Wingdings And The Shortfalls Of Outlook
24/3/2014
It's fairly well accepted that the font WingDings should not be used in webpages. Many modern cross platform browsers don't even support it. One of the reasons is that supporting WingDings means supporting creating symbol fonts which requires adding the SYMBOL_CHARSET define to the call to CreateFont instead of the normal ANSI_CHARSET. Due to their cross platform nature they want as much code as possible to be the same regardless of platform. So instances like this where a Windows specific font needs some special handling are often unimplemented to keep things simple. It may even be a strategic design decision made to deliberately weaken the support for it hoping that it's usage would fade away.

In my email client I have two places that I have to special case code just for WingDings. Firstly the above mentioned CreateFont call needs a special flag added so that when rendering HTML email that uses WingDings the correct glyphs are placed on the screen/page. The second place is during conversion from HTML to (unicode) text. Of course the actual bytes to render in the HTML in the document are not unicode or some easily known charset, so you have to convert from "WingDingsCharset" to Unicode using some hacked together table.

Now the question is "Who is creating all this content that uses the WingDings font?"

That's an excellent question dear reader, and the answer is: Outlook. In 2014 Outlook still happily converts:
:)
Into a WingDings HTML font tag containing a single uppercase 'J'. That will be rendered as a smiley face by Outlook and a capital J by most other software. Especially software on Macs and Linux that don't have access to that font by default. Never mind that Unicode has a perfectly functional smiley face character (☺) which could be rendered correctly by pretty much any software these days.

Outlook's character set issues don't stop there though. Take the case that I ran into recently where Outlook was used to send an email containing a URL that included a Unicode character. Now that in itself is somewhat dubious and indeed the URL referred to a Microsoft content server that should've known better than to allow a unicode character in the URL (especially when a perfectly suitable ascii character was available). Now that Unicode URL was specified used proper HTML entity encoding, however the meta charset of the HTML aaaaand the Content-Type of the MIME segment both stated the document was in us-ascii. And well, that Unicode character got converted to garbage at some point. Instead Outlook should make the charset of the HTML "utf-8" or something so that the character can exist in the specified charset. I've taken the approach of assuming the attributes of tags like 'src' and 'href' are in unicode despite the prevailing HTML charset.

Another aspect of the poor charset implementation in Outlook is the tendency to use undefined characters in the specified charset. Take for instance an Outlook generated email says it's using the ISO-8859-1 charset and then goes on to use characters in the 0x80 to 0x9F address space which is according to ISO-8859-1, undefined. Internally Outlook is using the Windows-1252 charset, which DOES define characters in that range. However instead of marking the email as "Windows-1252" it puts the incorrect ISO-8859-1 charset in the headers (they are essentially the same outside of that range). This commonly manifests as smart quotes (0x91 and 0x92) getting rendered incorrectly. Scribe of course has the ability to let the user override the charset which fixes the problem.

Now I'm in no way saying Scribe is perfect and lets all bash on Outlook. But I feel these bugs have been there for over 10 years and still haven't been addressed and something needs to be said. I doubt they will ever fix their software. Which is sad, because it makes the rest of the email software community look bad when we have to deal with the broken emails coming out of Outlook. I try and take a very literal interpretation of the incoming data. So that the user sees the content warts and all. I'm not a fan of sweeping bugs in other clients under my carpet. That has had a very poor history in the web-browser world. Ending in content bugs lasting far longer than they should have.

And if this ever gets back to someone working on the Outlook team: fix thy bugs. It's not even hard... these are really basic problems and it would most likely take a day or less to address.

Tags: email charsets Outlook

Comments:

wroot
13/06/2014 1:56pm

Very high Windows 7 idle hard disk utilization
15/3/2014
For months now I've noticed that when I leave my Windows 7 machine alone for over 30 minutes the hard disk starts grinding away relentlessly. Of course if you go to investigate what it's doing all activity ceases immediately. Preventing you from finding out exactly what is going on.

I had these thoughts of possible malware or a virus infection cross my mind and so downloaded and ran Process Monitor and then left the machine sit for 50 minutes. When I came back to it sure enough the hard disk was grinding away. But now I've been logging the activity! So what's the secret process using so much disk I/O?
9:49:49.6359004 PM	MsMpEng.exe	948	ReadFile	
9:49:49.6465209 PM	MsMpEng.exe	948	ReadFile	
9:49:49.6465457 PM	MsMpEng.exe	948	ReadFile	
9:49:49.6540933 PM	MsMpEng.exe	948	ReadFile	
9:49:49.6541138 PM	MsMpEng.exe	948	ReadFile	
9:49:49.6577645 PM	MsMpEng.exe	948	ReadFile	
9:49:49.6600897 PM	MsMpEng.exe	948	ReadFile	
Oh just Microsoft's anti-malware service checking all my files. "Alright... everything seems to be in order here... as you were".

Gear Unicorns
5/12/2013
It seems that my standards are too high, because I've been researching models of various pieces of gear that I'd like to buy at some point and I end up realizing that nothing in the market meets my needs, or if it does it's out of my budget.

Unicorn: Findings:
A: The colour laser printer.
  • ~$250.
  • Reasonably priced toner (haha.. ha.. *sigh*).
  • Good print quality.
  • Robust wireless.
  • Full size A4 tray.
  • Reasonable foot print.
Pretty much everything falls flat on the toner price. But if I remove that filter it's still hard to make all the other checkboxes fit at the same time.
At this point I've given up for a while.
B: The still / video camera.
  • Great DSLR style stills.
  • Good low light performance.
  • Longish zoom... x5.. x8?
  • Fast auto-focus in stills and video modes (inc zooming).
  • <$1000? (Maybe)
  • Smaller is better (might help with longer zoom).
  • Tethered shooting of video.
  • HDMI output.
  • Mic input.
There are a number of micro 4/3rds that kinda come close. And the EOS-M is again, kinda close. But I don't see any of them ticking all the boxes.
Pretty much decided on a Panasonic GH3 for this.
C: Replacement Mac laptop.
  • >= 8GB RAM.
  • USB3.
  • Firewire (I still regularly use a firewire audio interface).
  • At least 1000 vertical lines of display resolution.
  • Upgradable hard disk and RAM (current laptop lasted 7 years because I could fix things myself).
  • Dedicated graphics card.
  • ~$1000
The non-retina mid-2012 models seem to fit this criteria if they have the optional display upgrade, which makes them rare as hens teeth. Which has the side effect of making them much more expensive. Might just wait a while till the used prices drop.
So the old Macbook died an untimely death at my hands and so while I didn't have all the cash to spring for a mid-2012 model I decide that if I dropped the USB3 requirement I could get a much older / cheaper Macbook Pro that still ticked a lot of the boxes.

Or near enough... in fact the display is only 900px high but thats a lot better than the 13" Macbook's display. Ended up getting an early 2008 for $250 + shipping. Probably will need a RAM upgrade but at that price I can afford it.
  • 2.4ghz C2D
  • 2GB presently, 6GB when upgraded
  • 1440 by 900px screen
  • 7200rpm WD black HD (from old machine)
  • 2 x Firewire
  • 2 x USB2
  • Wireless A/G/N
  • GeForce 8600M GT (Dunno how good that'll end up being?)
I'll probably update this as times goes on and I refine criteria.

Scribe v2.00 Final
2/12/2013
Unless someone finds some major problem with the Windows v2.00 Beta46 release I'm going to be calling that the final official release of v2.00 in 2 weeks. It's not perfect but most of the issues are minor or only happen very rarely. I'll be trying to get the Mac port up to scratch in the mean time. There are a few known issues and that I need to sort out. (The filter bar is a mess, the edit and checkbox controls should REALLY be native). Then I'll post that hopefully before Christmas.

What that'll mean for customers of InScribe is that your v1.xx key that has been working for all the v2.00 beta's is not going to work for the v2.00 final. For existing customers the upgrade pricing will be $5 USD (50% off the normal price).

Once the v2.00 final is out and the upgrade process is nice and smooth I'll be turning my attention towards the security side of email. I've been watching the news lately and to be honest the revelations coming from Snowden have been eye opening. I think it's time to make privacy the main point, and as the author of an email client I feel it's my responsibility to make that happen. So the stable v2.00 branch will increasingly be focused on seamless encryption of email, and the surrounding RFC standards.

As a side note I'm not super happy about the performance of the Sqlite database layer that makes up the mail3 folder format. So I'm actively playing with options in that space. The client can easily support plug and play storage systems via a simple API (currently implementations are mail2, mail3 and IMAP). So adding more is feasible. Maybe some experimental implementations will appear in the v2.xx releases.

Tags: scribe releases
Why I Use Version Control
21/8/2013
Apart from the replication functionality of version control I got to use the ability to rollback code for real this week. Normally it's a stop gap that you never really need to use, but in this case I came across a problem that had me stumped. Clearly in the past my software has worked fine, and so in the case where you literally have no idea whats wrong, or even what change you made broke things there is version control.

In a nut shell the Mac port of Scribe was broken, it would start and then crash somewhere inside the Carbon library. I tried valgrinding it and that basically led nowhere useful. So I put it off for a bit before deciding to bite the bullet and walk back through my SVN commits and isolate the change that causes the bug. I started by writing a small python script that took a date as a command line parameter, that checks out the source as it was on that date. I started early in the year, and checked the 1st of each month. Compile each checkout and test it for that crash. I worked my way through till the 1/7/2013, where is started crashing... then I went back to 15/6/2013... runs fine. Forward to 23/6/2013, crashes. So some commit between 15/6 and 23/6 was to blame. The revisions in Lgi were 906 to 931 or so.

I began walking the Lgi checkout forward one revision at a time with the command:
svn up -r [revision_number]
Then recompiling after each update and running the software to test for the crash. It didn't take long to figure out that revision 912 was to blame.

Now I was getting somewhere!

So I started looking at all the source code changes in that commit and quickly ruled out all but two. So to test which part of the commit caused the crash I checked out r911 into a temporary location on my Macbook, while having r912 checked out on my PC. Then I proceeded to WinMerge changes one by one over to the r911 checkout on the Mac. That isolated the change to GList.cpp as the cause of the crash.

The final analysis is that a mismatched call to GSurface::ClipRgn(NULL) was doing something bad to the Carbon API and it would all go south from there. Reasonably easy to fix once you know what the issue is.

Yay... for source control! ;-)

Tags: coding crash
Visual Studio 2005 crashes when debugging an application
1/8/2013
I've just spent the better part of a day trying to troubleshoot this crash, so I'm going to post this here so that Google indexes it for the next poor soul that stubs their toe on it. Firstly I'll just outline the possibly relevant pre-conditions for this problem: What happens is that after uninstalling SUA I could no longer debug any application in VS2005. The IDE would simply crash as soon as debugging started. The event log would contain the follow entry:
Faulting application name: devenv.exe, version: 8.0.50727.867, time stamp: 0x45d2c842
Faulting module name: KERNELBASE.dll, version: 6.1.7601.18015, time stamp: 0x50b83c8a
Exception code: 0xe0434f4d
Fault offset: 0x0000c41f
Faulting process id: 0x22e0
Faulting application start time: 0x01ce8e70ca7eef80
Faulting application path: C:\Program Files (x86)\Microsoft Visual Studio 8\Common7\IDE\devenv.exe
Faulting module path: C:\Windows\syswow64\KERNELBASE.dll
Report Id: 0fb5f65e-fa64-11e2-8627-e05815f5174c
No one seemed to have the same issue as me. I tried lots of different things. Anyway I eventually stumbled on the problem by mere blind luck. In the add-in manager for VS2005 there was a "VSAddin" plugin that describes itself as an Extension to the VS debugger to support debugging UNIX based applications. And since I had recently uninstalled SUA to fix an unrelated issue I immediately twigged that this add-in was probably causing the crash. So I disabled it and restarted the IDE.

Now it doesn't crash. *sigh*

Tags: vs2005 crash
Have you ever needed to know what type of memory stick you have?
11/7/2013
Well Wikipedia has a nice SVG file that will tell you. Hold the piece of ram up to the screen and it will match one of the outlines perfectly (if your screen DPI is correct).

I had a piece of DDR2 it turns out.

I'm sorry, but that's just cool.

Tags: hack
LED Ring Working Prototype
2/3/2013
Here is my LED ring from the previous post working in real hardware:



Tags: mc1 axefx led-ring

Comments:

andrewz
05/06/2013 11:40pm

RGB LED ring PCB
24/1/2013
I've been working towards my own RGB LED ring PCB:



Tags: axefx mc1 led