Why We Reinvent The Wheel
Date: 22/12/2006
The issue of creating a new scripting language for Scribe has been on my mind lately, and as Ben simply points out in his comment here: "didn't you think about using an already existing scripting language to do the job?", it's clearly on the mind of my users and readers as well. And I thought I'd talk about how I got to become one of those geeky language types that talk about lexing and parse trees and so on.

The decision to roll my own language was in part a natural progression from i.Mage's scripting that just got a bit out of control. I started with an ultra simple syntax that I could parse in 3 lines of code: <command> <argument>[ <argument>] <crlf>. Pretty simple eh? Well then I thought about that and wanted to add some variables so I could do some simple expressions in the command's arugments. At this point spending weeks trying to intergrate a fully fledged scripting language seemed stupid, I only wanted some basic expressions. How long could it take? So a few hours later I had a basic expression evaluator and bing I was done. So I seemed to have staved off a good amount of work and still had the feature I wanted. Still I did think about it, considered the trade off and it was simpler to do it myself.

Now sometime later I decided that a loop command might be useful. So I add the "for" command which looped over a set of commands. This proved to be a little harder but not overly difficult, another few hours passed and I had loops in my language. Still I was convinced it was much easier than intergrating someone elses code. I had already suffered with the ICI scripting language for a Scribe plugin, and it was been painful to intergrate. It just didn't want to map to the internal object addressing method I wanted. So in fact I had tried to intergrate a script language with my code before and KNEW how much work was involved.

Sometime later again someone bugged me about writing some complicated filter in Scribe and I thought to myself "that ICI stuff was a pain, maybe I should look at bringing i.Mage's scripting language over into Scribe". So I did, it was trivial, with some changes here and there both apps could reuse the compiler/executor. In the meantime I had refined the syntax a lot to make it very Javascript like in an effort to stave off having to talk people into learning a new script syntax. It seemed like a good way to justify the language.

Still over the months it's existed I guess I've now spent about the same amount of time I would've had to spend to intergrate a external language. However I still have very tight binding to native C++ code, and some features that wouldn't map well to external languages, and I can change the language to suit my needs at any point. Because I understand it from a core level.

For instance I love being able to define a method that exists in a different DLL and call it through script. VB did this from very early on and it's brilliant. It means that you can glue different bits of your code in different langauges together very easily. This clearly rubbed off on me because I added it to my language fairly early on. I seriously looked into Python at one point and I wasn't happy with the work required to call external C libraries. I even thought about hacking Python's interpreter to do my bidding. But it was clearly a scary proposition.

I'm not sure I did the right thing in writing my own. But I'm certainly happy with the results, and the code base will be very valuable in a lot of the software I'm writing. The expression evaluator will most likely end up in i.Hex and so on. And I feel that compilers aren't so scary anymore. So there is a lot of first hand knowledge that I've gained from the process.
22/12/2006 10:10am
Yes, compilers are fun to build. Reminds me of a project I had to do in college.

I was in a Robotics program and we had a simple robot that was supplied and we had to create a programming language for it. Everyone else went out and made huge "Case" statements, but me, having heard somewhere about lexical analyzers decided to go to the library.

I found the O'Reilly book, "lex & yacc" and proceeded to create my own language. Hmm, fun times.

I agree, it's sometimes easier to roll it yourself than to use prepackaged languages. You know exactly what it's doing and can fix it yourself when it breaks, rather than waiting for someone else to help, or wading through 1000's of lines of someone else's code.

Good for you. I appreciate all the effort that goes into your programs.
25/12/2006 7:32pm
Been there, done that... rolled my own scripting language, because integrating with for example Mozilla's SpiderMonkey JS-engine would have been overkill and pain for the thing I wanted to have in my program. Should you ever want to rewrite your scripting engine, take a look at the Spirit parser ( which lets you define rules and actions in C++ (using some nifty template constructions)!
Email (optional): (Will be HTML encoded to evade harvesting)
Remember username and/or email in a cookie.
Notify me of new posts in this thread via email.