Blog
It's been a fantastic week at Memecode headquarters...
12/6/2015
I bought a cheap new Win8.1 tablet. Which was amazing for about 24 hours until I decided that I didn't want the 5gb restore partition and tried to remove it and resize the C: to fill the available space. Yeah that didn't go so well and I think I've bricked it.

To add insult to injury my main Windows 7 install on the desktop PC died tonight. It boots to a blank black screen. Safe mode hangs with lots of disk activity. It got stuck on the Paragon HFS+ driver, so I booted into Mac and removed that (copied elsewhere). And I'm trying to fix the disks one at a time with Disk Manager. Currently it's in some sort of infinite loop verifying my Win7 NTFS partition. Sigh.

Seems like all my PCs hate me at the moment.

One thing to be happy about is the fixes to HTML table layout I implemented today. There were some bugs related to table layout for spanned cells that have non-dynamic width specifications that are larger than the available space. And also an off by one error in the block element flow code. Slowly that HTML control is getting quite solid.

Tags: windows
SourceForge to Github Migration
4/6/2015
You may or may not already be aware of this but SourceForge has started taking control of popular apps that have abandoned their SourceForge project and moved elsewhere for the purpose of injecting adware into the installer for said products. The Gimp and nmap are some high profile examples.

This of course goes directly against my principles. And I can't in good conscience continue to use SourceForge to host any of my Open Source apps or libraries. So as of today Lgi, i.Ftp, i.Mage and i.Hex are all hosted on Github.

I need some experience using git anyway.

My main concern is that I can't actually delete the projects on SourceForge, and they may decide to take them over and zombify them in the same manner as the Gimp. There is very little I can do about that unfortunately. Personally I will consider SourceForge as a point of last resort when getting software.

Tags: open-source
Supporting Work
1/6/2015
It seems that a few things have gradually become a problem. Firstly customers with accents in their name are getting bad key's generated off the InScribe purchase page that they land on after buying the software. This is caused by less than stella coding on my part. But something that I'm trying to fix. The issue is that the data coming from PayPal is URL encoded and is in an unknown Charset. I've implemented some PHP in the purchase page to find the charset in the headers and do all the conversion for the $name. However in trying to test it I ran into a fairly annoying PayPal sand box bug.

The PayPal Sand Box Bug

If you go to create a new test user with an accent in their name you'll get a field validation error along the lines of "Only use letters". Which makes it hard to test purchase scripts with non-latin characters in the buyers name. I did however find a way around this, and that is to use the bulk upload tool to create the users. You can download a .tsv template direct from the PayPal sand box site on the "Create New User" page. From that fill out the details, and supply whatever name you want. Because the file is saved in unicode it bypasses the broken field validation stuff. Then upload that and you got a test user with accents. (Haven't tried Asian characters yet, but I will at some point). My new problem is that even after doing that and putting a sandbox button on my site the purchase script gets an empty response from PayPal when it checks the transaction. Fun times.

The Installer/Uninstaller

Everyone knows that the official way to uninstall a Windows application is to use the Control Panel uninstall page. It had completely escaped my attention until now that Scribe provides no such uninstaller entry in that list (Thank you dear user for reminding me). So I've gone and added the right registry entries to list the uninstaller, with help / support links, and the current version. The version is read straight from the .exe during installer compile time so it's always correct. Also the installer has been refactored to have a Desktop/Portable switch baked right in. Huzzah.

The Help

Also the help files shipping with Scribe have drifted from reality. So I'm in the process of reading through everything and updated all the parts that aren't correct anymore. Some areas might need fleshing out as well.

OAUTH2 Support

An initial draft implementation of OAUTH2 support for Gmail IMAP was shipped in the most recent build of Scribe. I'm quite keen to get feedback about it. So if you have tried to get it working and either succeeded or failed. Let me know.

Tags: scribe

Comments:

Scott
06/06/2015 7:02am
fret
07/06/2015 8:32am
Scott
10/06/2015 9:38am

Stabilizing GH3 Video For Sony Vegas
19/5/2015
I have a GH3 camera that takes pretty nice video. Because I cheaped out and got a non-Panasonic prime for the lens, I don't have in lens stabilization. What I do have is a bunch of footage that needs stabilization so I've been looking around for software based solutions.

The first (free) one I heard about was Blender and it's point tracking. After downloading that and playing around with it I found it very limited in what footage it can work with. As soon as you pan or zoom it becomes very difficult to keep the footage zooming and panned correctly. There are scripts to help but because it's not core functionality it just gets complicated fast.

Then I heard about Deshaker, which primarily runs as a plugin for VirtualDub. A tool which I haven't used in a very long time. So I downloaded both of those and got to work testing it on my footage. Immediately it was obvious that it was vastly better and more powerful that Blender.

The first issue I faced when using VirtualDub was that the input video was .MOV files off the camera. This is the format from ffmpeg:
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'bec+tanner 038.MOV':
  Duration: 00:09:03.36, start: 0.000000, bitrate: 49552 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuvj420p(pc, bt709), 1920x1080 [SAR 1:1 DAR 16:9], 48004 kb/s, 25 fps, 25 tbr, 90k tbn, 50 tbc (default)
    Stream #0:1(und): Audio: pcm_s16be (twos / 0x736F7774), 48000 Hz, 2 channels, s16, 1536 kb/s (default)
    Stream #0:2(und): Data: none (tmcd / 0x64636D74), 0 kb/s (default)
And VirtualDub uses .AVI files. Next step is converting the video to an AVI without re-encoding it. And that is best accomplished by my old friend ffmpeg. Which has the ability to convert between compatible containers without loss of quality and time caused by re-encoding the video. My initial attempt failed because the AVI container doesn't support big endian audio (the default off the camera). So I converted that to little endian using this command:
ffmpeg -i "input.mov" -vcodec copy -codec:a pcm_s16le "output.avi"
Now I could load my files into VirtualDub and stabilize them. The process basically involved adding 2 instances of the Deshaker plugin to the filter pipeline. The first one is set to "pass 1" and enabled. Disable the 2nd instance. Then go to the start of the video and runing the Output Playback mode. This generates all the motion vectors. Disable the first pass instance of Deshaker and enable the 2nd pass instance. Now you can export the video using your desired compression. I don't like any of the built in compressors so I downloaded the x264 codec and used that.

But the output had a problem. When the camera panned to track people walking, they appeared to jump back and forth in the stabilized output. Very odd looking. So I went frame by frame and noticed that in the source video each pair of frames coming through was the same. Deshaker then would get confused by this and shift the intermediate copy of the frame by more than the new frame. I thought the best way to get around this was to delete these redundant frames from the GH3. Fortunately VirtualDub comes with just such a filter: interpolate

By putting it in front of the Deshaker instances I got nice clean output, free of stuttering. At least as far as it would play in VLC. However when I loaded that into Sony Vegas there was big problems playing the AVI files back. Basically it would drop to a slide show, and on top of that there was corruption in the rendered output. Mulling my options I started looking for a way of getting the video out of VirtualDub without using the AVI container format. Fortunately there is a way. By installing an external encoder, which in my case was more just redirecting output to ffmpeg.

I found that by adding the right external encoder you can call ffmpeg to encoder the video straight from VirtualDub. The process involves writing out a "ffmpeg-1.vdprof" text file with the content from that link. Then importing that into VirtualDub and editing the details a bit. I decided to lower the -crf parameter to 16 to make it pretty much visually lossless. I don't want the stabilization to drop the quality at all.

Now I can bring that footage into Vegas and it plays smooth as butter. But it might have sent a few hairs grey.

Tags: video
Scribe Gmail Support Update
4/5/2015
Google has switched off so called "insecure" authentication methods, including "PLAIN", which Scribe relied on.

Today I got that draft implementation working enough to be able to login to the Gmail IMAP server using OAuth2. It does take you via a detour into your default browser but in the end it works. It's also quite the hack job at this point so I want to clean up the code and make the error handling at least very verbose. I will be making a release in the next few days off the stable branch that includes functional OAuth2 support.

It has been argued that Google's motivation in doing this is to force people onto the web and out of installed desktop / mobile clients, for the purpose of making Gmail's web UI more palatable. I remain unconvinced about this argument in that there seems to be another reason that makes more sense to me. OAuth2 doesn't require the client to store the plain text password. Thus reducing the possible points of failure for security. Most installed clients are bad at storing persistent account credentials in a fully secure fashion. So by removing that attack vector it could be argued that Google is doing App authors a favour.

Maybe.

I still find OAuth2 quite needlessly complicated. And it's very dependent on the client having lots of pre-configured knowledge about the server it's authenticating with. I mean for every service that Scribe will support OAuth2 authentication I have to have a unique ClientID and ClientSecret, as well as the token URI and access URI... none of which are "discoverable" on the fly, but hard coded in. So you can't connect Scribe to some arbitrary new service that supports OAuth2. I have to manually create support each new service. I don't yet understand how that is a reasonable state of affairs.

Tags: scribe oauth2
Scribe Gmail Support
24/4/2015
So Google have changed the authentication support for Gmail again, breaking the ability for Scribe to log in to any of the Gmail related services. Which means I'm going to have to put aside the v2.1 work temporarily to work on getting one of the support authentication methods working in Scribe.

The options are: I'll try and keep this post updated with progress on that front.

The list of all official SASL mechanisms is here. PLAIN-CLIENTTOKEN is not mentioned at all. I wonder what implements that? Also of note, both XOAUTH and XOAUTH2 are marked "OBSOLETE". Nice one Google, supporting only obsolete, undocumented or non-functional authentication methods.

Tags: scribe
Scribe v2.1 Update
21/4/2015
It may look like nothing is happening but really, there is. I have been toiling away on features and bug fixes in the v2.1 "trunk". Also of note, I've removed all the ANSI api support in the Windows implementation of LGI which means that any hope of running it on Windows 9x is gone for good. XP has been the official minimum for a while now, and now it's cemented in the code as well.

Tags: scribe
The Current State of Scribe
23/2/2015
Maybe it's time to just bundle OpenSSL in Scribe? I'm thinking of statically linking it so that it's always available and there will never be any trouble with missing or mis-matched libraries. The downside is the download size will blow out a bit. Over 2 MiB for the Windows build, maybe even 3. Recently I've had a very bad time with people have all sorts of weird crashes, odd behaviour and connection issues. And it seems that these things are next to impossible to reproduce. Maybe it's the target market changing on me as the profile of the client gets higher.

One of those odd behaviours is Scribe not saving it's settings between sessions. I suspect this has to do with the install folder being write only, but the software thinking it's in portable mode (i.e. store settings in the install folder). I'd love for someone to be able to reproduce that and tell me how.

So I'm going to look into ways of avoiding pitfalls. I'm starting to think that making automatic crash reporting default to "on" is for the best. I'm simply not getting enough data to respond to problems.

The HTML editing functionality is coming along nicely. I managed to send quite a few HTML messages over the last few weeks. But it's still pretty flakey. And also there are some thorny issues to sort out. However it's a lot better than even a month ago so there is hope.

Also I finally fixed an old crash in the GWindow destructor on the Linux/GTK2 build. The other main sticking point there is the popup handling under the GTK2 platform isn't great. But I didn't see the point in tackling that when the software was still crashing all the time.

Edit: So I think I've worked out the procedure for the not being able to save your settings bug: Yeah I'm going to have to make sure that Scribe is aware that during startup that it may have ADMIN permissions, and not to assume it will be able to write to anything in the Program Files folder tree. However it should totally take advantage of having ADMIN rights to setup the registry for being the default client / registered client.

Tags: scribe
Scribe v2.1
27/1/2015
So the v2.0 builds are basically stable as far as I can tell. Although there is some issues with some people connecting to SSL servers and one report of a crash at startup on MacOSX 10.10 (which I'm investigating). In the light of that I'm shifting my attention to the features in the v2.1 branch. I want to go through the things I'm aiming to get into that release: If you think I'm missing something really obvious that chime in either on email or in the comments on this post.

I'm looking to get the first build out sometime in Feburary.

Tags: scribe
"Invalid HTTP Request Headers" in Firefox
9/1/2015
For most of 2014 I've been using Firefox because I feel it's doing a better job of looking after my privacy. However it's not without it's cost. For instance today I found some pretty quirky behaviour in the Mac build of Firefox v34.0.5:



Basically on this site it's dropping the leading "G" of the "GET" request. To make matters worse the problem is intermittent. I can reproduce with a clear cache. Internally Firebug is reporting the correct packet, but Wireshark shows the missing byte.

Tags: firefox
Scripting Debugger
16/12/2014


And extern library function support!

Tags: scribe scripting

Comments:

fret
16/12/2014 10:18am

Portable OpenSSL on Mac OS X
12/12/2014
So it's been brought to my attention that Scribe's self install of OpenSSL on Mac doesn't actually work in v2.0.x, so the last week I've been looking into what it takes to do that.

The main issue is that load paths, are by default, absolute and hardcoded. Which works against you when you want a portable install of something. Secondly the OpenSSL .dylib is made of 2 libraries; libssl and libcrypt. And when you dlopen the first (libssl) the system needs to be able to find libcrypt. The link is embedded in the Mach-O image as a LC_LOAD_DYLIB segment. You can see that by issue the command:
otool -l ./libssl.1.0.0.dylib
Part of the output you'll get:
    cmd LC_LOAD_DYLIB
cmdsize 64
   name /usr/lib/libcrypto.1.0.0.dylib (offset 24)
   time stamp 2 Thu Jan  1 10:00:02 1970
That absolute path is what is causing the dlopen of libssl to fail. So we need to fix that. And the tool to do it 'install_name_tool', which is provided from Apple. Now I had two problems. Firstly what do you change the path to? And how do you actually get the tool to change the path?

Firstly Apple has several ways of locating shared libraries:
  1. @executable_path : relative to the main executable
  2. @loader_path : relative to the referring binary
  3. @rpath : relative to any of a list of paths.
What I wanted is the path to be relative to the executable, so I want:
@executable_path/libcrypto.1.0.0.dylib

But how to call install_name_tool to actually do that? Well to start with I tried:
install_name_tool -change /usr/lib/libcrypto.1.0.0.dylib @executable_path/libcrypto.1.0.0.dylib ./libssl.1.0.0.dylib
Which is actually the right command. But because the new string is longer than the old string the command fails (silently I might add). The trick to getting it to work is to add the linker command:
-Wl,-headerpad_max_install_names
to the CFLAG variable in the OpenSSL root Makefile. This causes the load segments to have extra padding bytes so that you can change the paths as much as you want later. Now after rebuilding the OpenSSL dylibs you can reissue the install_name_tool command and the link path changes:
MacOS matthew$ otool -l ./libssl.1.0.0.dylib
...
    cmd LC_LOAD_DYLIB
cmdsize 64
   name @executable_path/libcrypto.1.0.0.dylib (offset 24)
...
Then I could dlopen 'libssl.1.0.0.dylib' and it would pull in the right libcrypt file (both as sitting in the same folder as the executable).

I'm putting this here so I don't forget how to do this next time. Ha.

Tags: dylib openssl

Comments:

Yakov Eremin
10/04/2015 8:09pm
Yakov Eremin
12/04/2015 2:14am