J5

Simultaneoous Background Process Monitoring in Bash using Multitail

2011-11-24T00:30:00.000-05:00

I'm working on an installer for the development server environment we use at www.salsalabs.com

There are time-consuming tasks which are part of the installation process which can be done simultaneously: installation of program dependencies (with brew) and download of code repositories (using git).

I backgrounded the first task in my installation script using the usual & and let the script continue on and complete the second task "normally". After the second task completed, a wait command makes sure that the script does not proceed until the first task is also complete.

That's all well and good. It works just dandy... except that the output of these two processes is intermixed and due to each process providing feedback about the process of individual steps ... that output is very very very very very messy.

So I looked for a solution and I found MultiTail. Awesome! And available via brew! Awesome!

Except ... I don't have log files, and I don't want to generate log files. I want to have two different output streams and use multitail to view them simultaneously in one terminal window.

Two different output streams is easy enough, we've got file descriptors for that, right? (check out the exercise here, it makes my brain hurt). No, we need two different files that are going to act like streams. Oh! We want buffers!

(Cue the superhero music)

Here come FIFOs to the rescue! Use the handy-dandy mkfifo command and you can turn any output stream like stdout, stderr into a buffer that looks like a file to the operating system. Yay!!!!

So, our code, something like this:

install_dependencies.sh &
install_packages.sh
wait

install_something_else.sh

becomes:

mkdir /tmp/fifos
mkfifo "/tmp/fifos/Dependencies"
mkfifo "/tmp/fifos/Code Installation"

install_dependencies.sh > "/tmp/fifos/Dependencies" &
install_packages.sh > "/tmp/fifos/Code Installation"
wait

install_something_else.sh

And we add on multitail:

mkdir /tmp/fifos
mkfifo "/tmp/fifos/Dependencies"
mkfifo "/tmp/fifos/Code Installation"

install_dependencies.sh > "/tmp/fifos/Dependencies" &
install_packages.sh > "/tmp/fifos/Code Installation" &
multitail -ts --basename "/tmp/fifos/Dependencies" "/tmp/fifos/Code Installation"
wait

install_something_else.sh

The problem, now? multitail is never going to exit on it's own, we'll have to press "q" to end the process when the other stuff is done ... assuming we can tell for sure. So we never get to the wait command and we never get on to the next steps of our installation. Let's send an email to the developer and ask him for that feature ... but we'll make our own solution in the meantime.

So, here presented for you is my solution for monitoring a group of parallel background processes using multitail and ending the monitoring when all processes are complete.


PROCESS_COMPLETED_COUNT_TMP_FILE=$(mktemp -t "proc-count")
echo 0 > $PROCESS_COMPLETED_COUNT_TMP_FILE

function killtail {
    MUST_COMPLETE=$1
    PROCESS_COMPLETED=$(cat $PROCESS_COMPLETED_COUNT_TMPO_FILE)
    ((PROCESS_COMPLETED++))

    if [ PROCESS_COMPLETED -ge $MUST_COMPLETE ]; then
        # dangerous if there are multiple multitails running at the same time
        TAIL_PID=$(pidof multitail)
        kill $TAIL_PID
        rm -rf $PROCESS_COMPLETED_COUNT_TMP_FILE
    fi
}

mkdir /tmp/fifos
mkfifo "/tmp/fifos/Dependencies"
mkfifo "/tmp/fifos/Code Installation"

# subshell for everything involved with installing dependencies
(
    install_dependencies.sh > "/tmp/fifos/Dependencies"
    killtail 2
)> "/tmp/fifos/Dependencies" &

# subshell for everything involved with installing code
(
    install_packages.sh > "/tmp/fifos/Code Installation" &
    killtail 2
)> "/tmp/fifos/Code Installation" &

multitail -ts --basename "/tmp/fifos/Dependencies" "/tmp/fifos/Code Installation" 2> /dev/null
wait

rm "/tmp/fifos/Dependencies"
rm "/tmp/fifos/Code Installation"

install_something_else.sh

Now the subshells are backgrounded instead of the scripts themselves. Both "steps" are backgrounded, not just the first one. The command to increment the kill counter and check for the appropriate number of processes to have completed is added to the block of code. As well as backgrounding the subshells, we're directing stdout output from the subshells into our FIFOs, instead of directing the output of individual commands.

We pass the --basename option to multitail so that we see only the name of our FIFO in the multitail windows... and we've used nice human readable names for these temporary buffer files, so isn't that special?

When the kill counter hits the magic number, a SIGTERM will be sent to the multitail that we get back from pidof multitail. This makes multitail error, so we're throwing away the error output from multitail.

Finally, we still have a "wait" statement, just in case our multitail monitoring of the processes doesn't work for some reason. We still want to make sure they complete before moving forward with more steps. We remove the FIFOs and move on to the next steps in our installation.

Happy process monitoring!

Question: How do you convert a Java String object to a Javascript String in Rhino?

2009-12-21T15:46:00.001-05:00

Convert any Java object with a toString() method into a JS string by calling String(object) in JS


var javaString = new java.lang.String("Java String");
var javascriptString = String(javaString);

Javascript for the Mad Scientist: advanced javascript for jQuery

2009-12-16T12:25:00.001-05:00

I had a great time last night presenting a talk on advanced js for jQuery to the Frederick Web Tech. Meetup.

Christopher Thatcher of env.js and jQuery-Claypool fame and I hung out after the meeting and had drinks and lots of great conversation. Because of family and holidays and such-like many of the regulars couldn't come or couldn't stay for after-meeting socializing.

You can view or download/fork the presentation "slides" if you are interested in looking at the material.

jQuery Namespacing / Child Plugins Update

2009-12-12T10:28:00.000-05:00

I've updated the example namespacing plugin to use new arguments.callee() instead of extending the method and prototype individually. This should allow for proper inheritance.

It does require that the method also wrap the plugin behavior in

if (this.jquery) { ... }

jQuery Namespacing / Child Plugins / Modularization

2009-12-08T02:27:00.002-05:00

When writing jQuery plugins, you'll find that you want to group related functionality into a single "namespace" or "module", that acts as a parent for multiple child plugins.

There is not a lot of information readily accessible about the subject, from what I can tell. So little in fact that only after I came up with a quick and dirty method for myself did I dive deep enough to find anything.

Malnota's Wordpress Blog shows examples of three plugins that aim to do the job. They aren't reviews so much as they are demonstrations.

First up is jQuery.NameSpace by Gilberto Saraiva. Gilberto posts an example along with his download link, but to be honest with you, I don't follow any of it. Could be the moving stuff in the background of the blog. Could be the hour that I'm writing this at (I really should be in bed, kids!).

Malnota writes up the features a little more clearly in his review/article but again, I don't get a sense of what it's doing under the hood or how well it performs. Except that Malnota says that the "child" plugin did not receive arguments that were passed to it.

UH-OH!

A relatively casual review of the code shows that Saraiva is going through a lot of complicated eval gymnastics, and hasn't commented a single thing that he's doing. Double UH-OH. Seriously, though this looks like it took a lot of effort to complete and using my spidey sense, it might be close to a neat solution to the problem. I'll tell you that I like the fact that you don't need to call the parent/namespace as a function in order to use it. If nothing else, the approach taken here is interesting because of that aspect.

Next we have a solution created by the master himself, John Resig. Is this a good time to mention that my friends used to call ME ejohn? I wonder if I could have snapped up the domain name before he got to it and somehow switched places with him like he was my Dad and it was Vice Versa. That's right, I too wish that I could be John Resig.

John's solution is jQuery.space and the thing to say about it is that he has his own ideas about how to approach the situation. From the way that I read the code.. he's doing the same thing that Gilberto does in jQuery.NameSpace, but in a much more elegant fashion. It looks to me like he's creating a new copy of jQuery.fn for each namespace. Then when you want to stop working that space, you do the magical endSpace call and this code runs:

newjQuery.endSpace = function(){
  newjQuery.fn = oldFn;
 };

Just like magic you climb out of the rabbit hole. Hey, that's pretty neat. That's why they call him "The King".

Last out of the gate is jQuery.Modularize by Arial Flesler. I analyzed this one more than the other two to figure out what it was doing. Essentially, it creates an inner copy of the namespace, and stores the "this" (jQuery object) that is normally expected there and currying each function so that it will be called with the stored "this".

If you have a namespace "foo" (called via jQuery.foo() natch!) and methods "bar" and "baz", jQuery.Modularize creates "foo._" and "foo._.bar" and "foo._.baz" and "foo._._s_" (this). It only does the magic currying part of the algorithm once, or when you specify "lazy: true".

I find the cryptic ._. and _s_ and _o_ stuff to be offputting when studying the source code, but overall this is quite the nifty piece of craftsmanship. I was going to complain about the storing of _s_ (data) with the namespace (code) but then I remembered that javascript doesn't have threads so there's no way that it can collide with a simultaneous call for the same namespace with different data.

What a world.

If you've made it this far and want to see what I did, have a peek at the madscience namespace plugin in my github.

My approach is really freaking simple. I was happy I came up with anything. Since anything I write isn't likely to have 10,000 child plugins, I think that simplicity and terseness might make up for the lack of wowee zowee shown. Certainly I'll be leaving it as is for the purposes it was written ...

And I'll probably add links to this blog post and/or the two top namespacing plugins reviewed into the comments. So that the people who see my code can go see that there are other, more subtle ways to solve this problem.

P.S. John's plugin isn't up at plugins.jquery.com ... and unfortunately that site is a little bit app-store-licious ... I rarely find anything good searching it.

UPDATE: I think that there could be a collision in the stored jQuery object _s_ in the jQuery.Modularize plugin if you were using the same namespace in multiple asynchronous callbacks/timers. Something like this:

jQuery("#content").foo().bar(function () { jQuery("#heading").foo().bar(); this.baz(); });

If I am reading the control flow correctly, this.baz() will act on "#heading" and not "#content". I'll ping Flesler and see whether this has been considered or is a real life (edge) case or not.

Splitting Directives - Modular "Molecule" Configurations of Apache httpd

2009-12-04T13:07:00.003-05:00

At my 9 to 5, I am currently wearing a sys admin hat (this is something between a welder's mask and sherlock holmes' tweed cap, I think). I love process automation (the foundation of programming, I'd say) so everything is done with a bash script for maximum reuse.

My vision is to be able to rebuild a new version of my httpd configurations and deploy them to the target servers in one command (the testing process happens BEFORE the final build and deploy, silly!)

Why? One of the problems that I've seen in past projects is lack of configuration management and revision control for infrastructure applications like httpd or websphere. Worse, different tiers in the enterprise (boldly going, anyone?) and different servers might have different configurations because of lack of strictness in implementing changes methodically.

Awesome sauce local geek-about-town Jason Leveille turned me on to a tool called Puppet by Reductive Labs. I am impressed with the featureset that I have seen, but alas, the method of delivery can't be used for my purposes (client daemons running on all the boxes).

If you have complete control of your deploy environment and want to solve lots of other problems besides automating and modularizing updates/builds of Apache httpd, I think that you should definitely look at Puppet. I will likely be diving deeper even without having a clear need right now that it can fill.

Enough about Puppet, though. This post is called Splitting Directives and there hasn't been one mention of httpd.conf yet! What gives? Well if you've ever met me I'm either listening and agreeing with what you say or I'm telling it the long way. Life is a journey, right? Forgive me my exposition and proceed if you want to know the method I cooked up for splitting common and specific Apache httpd.conf directives and for building and deploying them.

First, you have to reorganize your httpd.conf file and bring related directives close together. Good examples of this are that you would move entries that refer to virtual host docroots into the VirtualHost directives. You would put the Listen and NamedVirtualHost directives right before the VirtualHost block that they apply to. Move IfModule blocks to occur right after the module is loaded (be careful if the stuff in the IfModule has multiple dependencies). That kind of thing.

Next, mark up with comments how you can split the httpd.conf up into multiple files. Put a start and end for each section you think is "modular molecule".

Split the httpd.conf using the start and end markers into separate files, put the new files in a subdirectory called "common" or similar underneath /apacheroot/conf. Replace (or comment out and replace) the original directives in the httpd.conf file with Include directives that match your "modular molecule" filenames.

For example, here is a very basic version of the httpd.conf I am currently working with:

# ServerRoot, ServerName, DocumentRoot, ServerAdmin directives
Include conf/common/main.conf
# All the LoadModule directives
Include conf/common/modules.conf
# LoadModule and WebspherePluginConfig directives 
Include conf/common/websphere.conf
# User and Group directives, with IfModule blocks around them
Include conf/common/runas.conf
# VirtualHost blocks and associated Listen and NamedVirtualHost records, one per file.
# Remember you can repeat directives like Listen and NamedVirtualHost with the same value without harm,
# So make it one VirtualHost block per file even if they share an IP and port.
# For extra "security" you can explicitly include these, or include one file at conf/common/vhosts.conf that specifies all the others.
Include conf/common/vhosts/*.conf

OK, so at this point we just have a bunch of files and a bunch of includes and there's not much to it. Certainly it can't be used across the enterprise in this form, no matter how fancy the deployment script is. The configuration is just not suitable for more than one computer (unless they all use the same hostnames and all IPs are bound the same, etc. etc.). I am assuming that like me, your configurations should all be substantially the same and that there are only a few things that you need to modify per-host. Things like IPs and hostnames.

That is what the next step addresses. We create another folder underneath /apacheroot/conf called "config" or similar. We now go into each of the files we created earlier, and reorganize them according to whether the directive is a specific one or a common one. The difference between the previous method of organization and this one? This time, wrap the specific directives around the common ones. Where common and specific directives occur in the same file, create a heirarchy with common directives at the core.

We're running a decorator pattern on the configurations ... the common directives are wrapped and decorated by the specific directives. Specific directives go into the conf/config folder and common directives stay in the common folder, albeit in an "unwrapped" (and probably unusable) state.

Edit your httpd.conf to reflect where you have included "modules" that are only specific directives, or wrap specific directives. Omitting the comments for brevity, you now have something like this:

Include conf/config/main.conf
Include conf/common/modules.conf
Include conf/config/websphere.conf
Include conf/config/runas.conf
Include conf/config/vhosts/*.conf

A sample vhost.conf might look like this:

Listen 127.0.0.1
NamedVirtualHost 127.0.0.1:80
<VirtualHost 127.0.0.1:80>
   ServerName localhost
   ServerAlias local.dev mysite.dev mysite.local foo.bar
   Include conf/common/vhosts/sample.vhost.conf   
</VirtualHost>

Run apachectl -t from bin to check your configuration. Working? Awesome. Tweak as necessary if not.

This is a configuration good for one single host, but all of the specific directives have been isolated on the filesystem to the "conf/config" directory. We can now use whatever means we want to organize multiple configurations on the filesystem -- based on the same common directives with similar specific directives.

At the simplest, we just copy the config directory and make a different configuration there. We started with a "dev" configuration we use on our localhost, but what about when we want to deploy this out onto the QA server(s)?

cp conf/config conf/qa1
cp conf/config conf/qa2

We need to store these in version control for later use. And that is going to factor in to the deployment method. This project is using svn so I'd run these commands, too:

svn add conf/qa1
svn add conf/qa2
svn commit conf -m "Adding configurations for qa1 and qa2"

Now you have to go in and edit them to have the right IPs and hostnames and what-not and then commit changes after.

On your buildserver, you should have all these configurations checked out somewhere local. Let's say in /build/apache/httpd/modular/ and that the source code repository is at svn://localhost/apache . You should also have a working area outside of SVN where you can build a configuration locally before you deploy it across the wire to a target. Something like /build/apache/httpd/configs

The build script is pretty simple. Designed to be run from /build/apache/httpd and for there to be directories "modular" and "configs" for the configurations and build directories:

#! /bin/bash
# $1 config to build
SVNREPO="svn://localhost/apache"
CONFIG="$1"

# make directory for configuration
mkdir -p configs/$CONFIG

# export the common directive files to the build dir for this config
svn export $SVNREPO/common configs/$CONFIG/common
# export the configuration specific files to the build dir for this config as "conf/config"
svn export $SVNREPO/$CONFIG configs/$CONFIG/config

Run this script with the parameter "qa1" and there is now a functional configuration for your qa1 server at /build/apache/httpd/configs/qa1. There are a lot of ways to deploy this configuration to a target. I am currently using scp directly but you could just as easily compress the config with tar or another program and move it across as one file.

Example:

#! /bin/bash
# $1 config $2 user $3 host $4 apache root path on host
CONFIG="$1"
USER="$2"
HOST="$3"
PATH="$4"

scp -r configs/$CONFIG/* $USER@$HOST:$PATH/conf

# test and then restart apache here too?
# ssh $USER@$HOST "$PATH/bin/apachectl -l"
## should check the response above before proceeding ... you can roll that yourself :)
# ssh $USER@$HOST "$PATH/bin/apachectl graceful-restart"

There are a lot of configuration directives that are tied to a particular site or a particular layer of the enterprise (dev, qa, stage, production, etc.) and not just the specific server. The method given above can be extended to allow for more complex heirarchies of configuration to be built. Simply repeat the steps but consider commonalities between "stage1" and "stage2" that have analogs in "prod" layer as well. Separate them from the "config" level as you did with the "common" directives. Make a template set as "layer" or "mode" or whatever you think of the steps in the deployment cycle as. Copy that template set on the filesystem as you did with the "config" folder.

Consider the best way to nest your files for your use when modifying configurations. All of the intermediary levels between the most specific and the most general directives need to stay as closely in sync as they can for this system to be useful in avoiding misconfigurations and configuration entropy.

I think a good general rule would be that each server has a "site" role (when there are multiple httpd installs for instance), a "layer" role, and a "server" role. That would correspond to a structure like this:

conf/common
conf/site (template for conf/sites)
conf/sites/site1
conf/sites/site2
conf/layer (template for conf/layers)
conf/layers/layer1
conf/layers/layer2
conf/server (template for conf/servers)
conf/servers/server1
conf/servers/server2

The build script then must be adapted to take each of these into account when reducing the set of configurations down into the configuration needed for this particular host.

#! /bin/bash
# $1 server to build
# $2 layer to build
# $3 site to build
SVNREPO="svn://localhost/apache"
SERVER="$1"
LAYER="$2"
SITE="$3"

BUILDDIR="$SITE/$LAYER/$SERVER"

# make directory for configuration
mkdir -p configs/$BUILDDIR

# export the common directive files to the build dir for this config
svn export $SVNREPO/common configs/$BUILDDIR/common
# export the server specific files to the build dir for this config as "conf/server"
svn export $SVNREPO/servers/$SERVER configs/$BUILDDIR/server
svn export $SVNREPO/layers/$LAYER configs/$BUILDDIR/layer
svn export $SVNREPO/sites/$SITE configs/$BUILDDIR/site

Each site may have different configurations for each layer. If the way that the Include statements in your configuration files works there are lots of levels of heirarchy, you may wish to use a heirarchical folder structer. That is the fundamental model that I used, but it is more complex to illustrate conceptually and so I will leave illustrating it to a later post (if there is demand).

I hope this post helps someone else who is looking for information on build scripts, modular configurations, modular directives, magic modular molecules or general httpd common configuration tomfoolery. Good luck and take care! Considered actions are the only defensible ones.

Google Chrome Frame

2009-11-29T13:58:00.000-05:00

Since my Web Worker shim relies on Google Gears plugin and the Google Chrome Frame plugin may make Gears obsolete (at least for this use case), I have temporarily suspended work on my html5 shims while I wait for that issue to shake out.

I may push harder for a non-Gears, native solution based on Statified JS. There is also a possibility to offer server-side worker implementations, which dovetails nicely with my recent research about the state of server-side JS.

People seem to be very excited about Node, but I prefer env.js and the Claypool JS framework from what I can see.

I think that server-side DOM emulation is the "BIG DEAL™" and that server-side JS should build on an existing server stack (which was one of the nicest points in Jaxer's favor when it was in active development).

Is anyone working on making any of the other server-side JS servers apache httpd modules? I haven't seen any indication that they are interested in this kind of integrated approach.

First Test Suite Complete

2009-09-18T15:45:00.000-04:00

The first version of the test suite is completed now and everything is passing, testing 6 levels of child worker nesting, and some core functionalities such as importScripts, self alias for the worker global scope, the lack of "window" and hiding the worker global scope constructors.

The results of the test suite are this:

worker received: worker
worker: JSUNITY: Running unnamed test suite
worker: JSUNITY: 4 tests found
worker: JSUNITY: [PASSED] testImport
worker: JSUNITY: [PASSED] testWindow
worker: JSUNITY: [PASSED] testSelf
worker: JSUNITY: [PASSED] testWGSVisibility
worker: JSUNITY: 4 tests passed
worker: JSUNITY: 0 tests failed
worker: JSUNITY: 2 milliseconds elapsed
[object Object]
worker: JSUNITY: Running unnamed test suite
worker: JSUNITY: 1 test found
worker: JSUNITY: [PASSED] testInnerWorkerExists
worker: JSUNITY: 1 test passed
worker: JSUNITY: 0 tests failed
worker: JSUNITY: 0 milliseconds elapsed
[object Object]
worker-child queue: worker-child
worker received: another message from parent
worker received: worker-child received: worker-child
worker received: worker-child: JSUNITY: Running unnamed test suite
worker received: worker-child: JSUNITY: 4 tests found
worker received: worker-child: JSUNITY: [PASSED] testImport
worker received: worker-child: JSUNITY: [PASSED] testWindow
worker received: worker-child: JSUNITY: [PASSED] testSelf
worker received: worker-child: JSUNITY: [PASSED] testWGSVisibility
worker received: worker-child: JSUNITY: 4 tests passed
worker received: worker-child: JSUNITY: 0 tests failed
worker received: worker-child: JSUNITY: 3 milliseconds elapsed
worker received: [object Object]
worker received: worker-child: JSUNITY: Running unnamed test suite
worker received: worker-child: JSUNITY: 1 test found
worker received: worker-child: JSUNITY: [PASSED] testInnerWorkerExists
worker received: worker-child: JSUNITY: 1 test passed
worker received: worker-child: JSUNITY: 0 tests failed
worker received: worker-child: JSUNITY: 0 milliseconds elapsed
...

I need to add tests for the other portions of the API that are completed as well as the portions which are not. Once I do that, I will start working on using a MessagePort implementation for the communication between the worker and worker global scope. This should allow me to plug in communication based on local storage or web database, which is necessary for implementing SharedWorker.

The work I have been doing on the postMessage and onmessage events and how they queue when the objects are not ready should make it simpler to implement the MessagePort / MessageChannel structure.

Right now the most glaring problem is that there is significant duplication of the onmessage code between DedicatedWorker and WorkerGlobalScope. I was allowing this because the code in WorkerGlobalScope uses a closure to remember the scope inside the workerPool onmessage method and the DedicatedWorker didn't need it, but there shouldn't be any reason not to just store the scope reference for DedicatedWorker, too.

I need to experiment with where I can store that function for both objects to use, however. I am finding that the structure of the current objects and how they are passed around is proving quite limiting in terms of allowing for shared functions. Solving this will be even more important as I design a larger framework that includes other parts of the HTML5 and related APIs, available inside and outside of the worker scope.

I will try to create an object called html5shims and attach all of the shim API implementations to the global scope from within the closure surrounding html5shims. The current implementation of Worker uses a style of function assignment which is not compatible, so I will have to check again which browser was requiring that and see if I can find a different way.

Here's that style if you're curious:

Worker = (function InitWorker(window,navigator,wgsSource) {
                  function DedicatedWorker (url) {
...
                  }
              return DedicatedWorker;
          })(this,...);

with (foo) { function bar() {} } fubared

2009-09-17T21:38:00.000-04:00

The last two days I've been working on a test suite for my webworker shim and found a couple of wrinkles.

The first was that messages were being lost when they were sent immediately after the worker creation.

For example:

var w = new Worker("foo.js");
w.onmessage = function (event) {
    alert(event.data);
};
w.postMessage("foo");

The message "foo" would never reach the worker. This was solved simply enough by queueing messages when the communication channel is not ready. Which brought me back to looking at the spec for MessagePort and MessageChannel, but that's not important right now.

The really wrinkly thing that I'm seeing now is based on this test:

function testImportScripts () {
   importScripts("../scripts/import.js"); // declares function importedFunction
   assertNotUndefined(importedFunction,"imported function is defined");
   ...
}

First I was getting reference errors telling me that importScripts was undefined. I was able to get around that by surrounding the call to importScripts in a with (this) {} block. That worried me. And importedFunction was still undefined.

So I write a simple worker script:

onmessage = function (event) {
 postMessage("received: " + event.data);
};

importScripts("../scripts/import.js");
importedFunction();

No importedFunction. However, with importedFunction assigned instead of declared -- viola! function declarations inside of with blocks are not supported according to ECMA-262 3rd edition. It makes sense, but it puts a bit of a crimp in my plans. (I found this post which confirmed my suspicion, I didn't actually dig through the spec to find out!

The current version of the WorkerGlobalScope uses this code inside of it's _executed method for all internal executions:

Function("with (this) { " + source + " }").call(this);

I strongly prefer to maintain the use of with so for now I am modifying the code passed to the worker to make function declarations function assigments. The latest version of the code and the tests so far are checked in to SVN.

Web Worker API Shim Demo Posted

2009-09-13T22:08:00.001-04:00

I finally got the Web Worker API shim demo posted on Google Code tonight.

There is an interesting trick to posting the demo HTML page where you need to set the svn:mime-type property to "text/html" or it won't be served as HTML. Makes sense but confused me for a while and made me sidetrack looking at cleaning up my old portfolio site.

Not that that wouldn't be a great idea, too.

Web Worker Demo

To see the shim, visit that URL with a non-supporting browser that has Google Gears installed. You may also be interested in comparing it against browser implementing web workers natively such as FF 3.5.

The shim is implemented via two classes. Probably the simplest way to understand how they work is from the inside out. Implementing the Worker object and the worker-to-worker-thread messaging mechanism on top of Gears is pretty simple. But the API doesn't just cover instantiating a worker, giving it some work to do and talking to it. The API defines a window-like environment that the worker thread operates in, where other APIs are available:

WorkerGlobalScope

Once the WorkerGlobalScope is created, the Worker itself is a dispatcher, sending and receiving messages. So what does the WorkerGlobalScope look like and how do we make one?

The HTML5 Web Workers API spec covers the WorkerGlobalScope in detail. WorkerGlobalScope.js is my implementation. In it, we have constructor functions for each of the WorkerGlobalScopes (Dedicated, Shared and the "base") which handle setting instance variables both private and public. We also have DedicatedWorkerGlobalScope onmessage prototype and a prototype object which handles all of the methods available in any WGS, including empty Worker and SharedWorker constructor methods.

But the WorkerGlobalScope established by the constructor and prototype is incomplete -- not only because I haven't finished everything! It is missing the very crucial piece of providing a working Worker implementation.

Of course, we can't have a WorkerGlobalScope without having already created a Worker. The implementation of Worker is provided not directly by WorkerGlobalScope but by the calling Worker. The DedicatedWorker.js file bootstraps the Worker into the originating window environment with the InitWorker method that returns the DedicatedWorker constructor.

Simplified:


 Worker = (function InitWorker(window) {
  function DedicatedWorker(url) {
  ...
  }
  return DedicatedWorker;
 })(this,navigator,WorkerGlobalScopeSource);

The InitiWorker method provides closure around the bootstrap parameters. "this", the global scope for workers. "navigator", the navigator object passed through from the global scope. The source which can be used to create the WGS. Because it is named, it can be easily passed as a string via function decompilation into the Gears worker pool thread. It is my understanding that because it is defined as part of an assignment operation, it does not pollute the namespace (I think there are some finer points there, but I don't know them as well as I would like).

When we create the code to execute our worker's payload in the workerPool, which must instantiate the worker global scope and make all of the APIs available to the worker, we basically repeat the exact same construction. There it is text and not source code, but it is still the same thing. In this way, we share the source code for Workers and WorkerGlobalScope with all child instances even though we cannot directly pass the objects.

The code which is executed by the Gears workerPool thread:


this._source = [
 wgsSource,
 "var wgs = new DedicatedWorkerGlobalScope(\""+url+"\");",
 "wgs.navigator = " + toSource.call(navigator)+";",
 "wgs.navigator.online=true;",
 "Worker=wgs.Worker=("+InitWorker+")(wgs,"+ toSource.call(navigator) +",'"+ escapeQuotes(wgsSource)+"');",
 "wgs._loadSource({url: ['" + url + "'], "+
 "callback: function(scripts){ wgs._scripts = scripts; wgs._execute('importScripts(\""+url+"\")'); } });"
].join("\n");

Matters are complicated by the fact that the HttpRequest object (analog to XHR) available in Gears does not allow for synchronous operations but that is a story for another day.

As the credits on the demo say, thanks to Andrea Giammarchi for pointing me at the need for this shim.

HTML5 Shims, Shivs, Fallbacks and Compatibility Layers

2009-09-10T10:47:00.000-04:00

I am working on my implementation of HTML5 WebWorker shim via Google Gears plugin (currently). Part of what is interesting is that the WebWorker has a lot of the other HTML5 APIs available internally, so I am also looking to bring in other shims / write my own if I want the project to be complete and provide an actual fully-functioning Worker-alike and SharedWorker-alike.

As a result, I've decided to start a project to collect the available shims. Rather than just the Web Worker, I will try to bring together as much of the HTML5 js apis as possible. The project is called "html5-shims" and is hosted on Google Code.

I have added a page to the wiki for my worker shim/html5 shims project which lists other shims available, the relevant specs, etc.

HTML5 Shims : Links & Resources