Create Blog » Web

For the Love of a Deity, Please Prove Me Wrong: IE Alpha

Alex Iskander — Tue, 29 Dec 2009 19:06:38 +0000

Please no. Just, no. I mean, honestly, how far can IE sink? I really hope I am wrong here, but…

If you check my test case, you may notice that it looks different in Internet Explorer than it does in sane browsers. What is the culprit?

Simple: filter:alpha.

Supposedly, making elements somewhat transparent is possible in everyone’s favorite browser. Well, it is, but you have to use Microsoft’s proprietary “filter” CSS property.

Okay, fine, at least it works. Except in one case, from what I can tell: it will not apply transparency to absolutely positioned child elements of the element with the “filter” property set.

Why is this an issue? Let’s take the simple example of a button control (specifically, SproutCore’s SC.Button, but it should apply to other button controls as well): you have a left part (with nice rounded corners), a center part (that can be stretched), and a right part (again with the rounded corners). One part—for instance, the right part—is either positioned relatively or is the parent element itself, and simply has a background-position:right. The left and center parts, however, are positioned absolutely; something like left: 10; right: 10 for the center part; right:0; width:10; for the right part (I just made these numbers up! They aren’t real!)

So, what happens if you try to fade the button out in Internet Explorer? Well, it looks really odd. In the above example, the right part fades out, but the left and center do not. They just stay as they were.

There is only one workaround that I know of: seting filter:alpha on every element. There are a few problems with this:

Performance. Each setting of “opacity” will have to recursively loop through all child elements.
Performance * Animation fps. Yep, animating opacity in IE is plain screwed.
Pure messiness.
Long-term issues: what about when (if) Microsoft fixes the issue?

Don’t you just love Internet Explorer?

I keep thinking I must be wrong, and there must be some nice little workaround, CSS hack, or even the simple explanation of me just being plain wrong (I actually hope I am).

Anyone know?

Making SproutCore Objects Show in Chromium’s Heap Profiler

Alex Iskander — Tue, 15 Dec 2009 16:38:08 +0000

It’s a hack. If Chromium+v8 would just support displayName, it could all be nice and clean and part of SproutCore (please go tell the Chromium folks how helpful this would be).

As it is, this hack could have some SEVERE side-effects, but chances are, if you are trying to debug a memory leak, you won’t care about these side-effects because everything else will be driving you MAD (and, you’ll probably be able to find these side-effects much more easily than the leak itself).

It is very simple; here is how to do it for ListItemView:

SC.mixin(SC.ListItemView, { create: function() { return new SC._ListItemView(this, arguments); } });
 
SC._ListItemView = function(base_type, args){
  base_type.call(this, args);
};
 
// for extra safety (should get around most potential side-effects):
SC.mixin(SC._ListItemView, SC.ListItemView);
 
SC._ListItemView.prototype = SC.ListItemView.prototype;

Now, ListItemViews will show up as SC._ListItemView in the heap profile. Hooray!

P.S. This revealed for me that the objects leaking were not SC.ListItemViews. I still have no idea what is leaking, and am giving up for now—but at least I’ll have better tools when I come back to it. :)

JavaScript Animation Benchmarks

Alex Iskander — Wed, 21 Oct 2009 23:18:49 +0000

I’ve been working on a mixin to add animation to SproutCore views.

The current version only works for layout properties, and does not yet work for centerX and centerY properties (they used to work, but some of the performance optimizations have made it slightly more tricky—I’ll be adding it back soon, though).

I decided to see how fast the code was in different browsers. The tests were done using a test application which generated a specified number of views, and then, once per second, updated a “frames per second” display. The measuring is somewhat subjective, as I have to deduce, based on consistency (or lack thereof) in the numbers, what the frame rate actually is. For the most part, they were pretty consistent, but the WebKit browsers at really low numbers of views (and really high frame rates) could be quite inconsistent at times.

I ran the tests on two separate machines:

Mac OS X Snow Leopard on MacBook Pro 15″ Core 2 Duo 2.33Gz w/2GB RAM. Lots of open programs, including iTunes. Not as scientific as would be optimal, but it is my work machine.
Windows XP on Pentium 4 3.20GHz w/2GB RAM. Only the browser was open.

Browsers tested (note that my Firefox was OLD):

Safari 4.0.3 for Mac
Firefox 3.5.3 for Mac
Chromium 4.0.223.3 (29380) for Mac
Internet Explorer 8
Firefox 3.0.1 for Windows (oops. Should have upgraded first. Remind me to update numbers on Friday)
Safari 4.0.3 for Windows
Chrome 3.0.195.27 for Windows

Here is the data:

Data collected during testing of the different browsers

That’s boring. Where are the pictures?

All Browsers

Even here, you can tell that Chrome and Chromium are the best performers for small numbers of views. However, for larger numbers of views, Safari on Mac was a very clear winner, running at 12 frames per second for 2000 LabelViews..

Predictably, Internet Explorer was slowest. I didn’t bother to run all the tests for it.

Windows Browsers

This chart shows only the Windows browsers. Again, you can tell IE is very slow. Chrome is very very fast. Safari is pretty good. My out-of-date Firefox is not always that much faster than IE; it will be interesting to see how Firefox 3.5 fares.

Notice the dip in performance for Safari at around 300? It briefly does worse than Firefox. It quickly levels off, though, and is second only to Chrome.

Mac Browsers

And, of course, the best for last. Note how we see the same dip in performance for Safari—though it does not dip below Firefox; it would have dipped below Chromium, were it not already below it. Again, though, it leveled out faster than Chromium.

I always wanted to do a performance chart, but now that I have… it’s a lot of work (about 57 or so data points there… each measured separately).

Question Is the difference between Safari and Chromium completely caused by Google’s V8 JavaScript engine, or are there a few other differences?

Sprouting—Automated Spriting for SproutCore

Alex Iskander — Sat, 17 Oct 2009 23:09:09 +0000

Well. Spriting. Do you know what it is? While I’ve included a very brief summary, you may want to just skip that part if you already know about it, or find a better description of the technique. Also, while this article is specifically aimed at SproutCore, the concepts (and the script download) can easily be applied elsewhere.

Well, here are your options:

Read the whole article. Could be amusing, if I’m having a bad writing day.
Skip to the download and “how to use.” No, I’m not going to explain the code. Well, not here, at any rate. I’ll explain it in the code itself. Yes, of course there are comments.

What is Spriting?

Example Sprited Images

Spriting is a technique which combines multiple small images into one large image. This can be very beneficial, because it means that, instead of downloading, say, 100 small images for all the commonly used icons in your web application, the web browser downloads one bigger image (or, perhaps, two or three, but you get the picture).

The goal of spriting is to reduce the effect of latency on your web app’s performance.

To your left is an example set of sprited images.

Using a Script

You may thing that combining several images into one larger image might be a bit time-consuming and, overall, a tad challenging, especially if you are often updating the individual images.

You are not alone. I feel the same way.

However, I do know a little about writing scripts—even if I’d like to know a lot more; I’m still only learning Python. So, there is a simple answer:

Write a darn script!

If you just want the script and how to use it, you’re in luck. Because there are, in fact, some comments in the script, I’ll leave the explanation of how the code works for there.

You can download the code from GitHub.

The script is written in Python. It requires: Either: Python 2.6+ or Python 2.5 with simplejson installed PIL (Python Imaging Library)

Very briefly, here is what it does:

Takes a “sets” folder, which should have sub-folder sets of icons to be sprited.
Takes an output folder, under which it will create new set folders containing CSS and image files.
Combines images under a maximum size (128×128 by default—see Configuration) into a sprites.png file in the output set folder.
Creates CSS referencing these images using a URL template (SproutCore oriented at first—see URL)

Sets

You often might have more than one set of icons. For instance, you could have a main set, and then an alternate set that you only want to load on demand. You still want that additional set to be sprited.

Or, perhaps, you may have images meant for repeating backgrounds, but that you still want to sprite.

To facilitate this, the spriting script takes, as its first argument, the location to a sets folder. You put your sets as subfolders into this folder. For instance, this could be your folder layout:

my-sets
	common
		delete-32.png
		delete-64.png
		refresh-32.png
		refresh-64.png
		refresh-512.png
	repeat-x
		config.js       — see Configuration
		toolbar.png
		footer.png
		header.png
	odd-feature
		some-icon.png
		other-icon.png

URLs

The CSS has to reference the images. Only problem: it doesn’t know where those images are.

Usually, you’d just be able to use paths relative to the StyleSheet. However, if you are using SproutCore, this won’t work because it moves around all the CSS and JavaScript.

With SproutCore, you need something more like:

static_url('images/set/sprites.png')

This is what the script does by default. Just like above. It uses a URL template of

static_url('images/{set}/{image}')

It is a string which will be passed through the Python format function, with set and image as arguments.

Configuration

There are a few things that are configurable. Configuration is done per-set.

Configuration files are JSON files, and are very simple. They are optional, and take at most four parameters:

max-size: Default: 128.
The maximum size for sprited images. Images above this size will be put as separate images. Useful if you have some really high-res icons you don’t want to sprite, but want to keep with the set. 
repeat: Default: “none”; can also be “x” or “y”.
Determines the repeat mode. If repeat is “x,” the sprites will be laid out in rows; if “y,” the sprites will be laid out in columns. The generated CSS will include repeat-x or repeat-y, respectively.  Note that all images in a repeat-x set must be the same width, and all images in a repeat-y set must be the same height. The width or height of these images should be set using repeat-width and repeat height, which are discussed below. 
repeat-width: Default: 32.
If repeat=“x,” the width of the repeat images. If your repetition does not require a pattern, you could set this to 1. 
repeat-height: See that lovely summary of repeat-width above? Well, this is the same, but for y/height.

The CSS

As you may know, SproutCore automatically includes any CSS files. So, since this generates CSS files (and, if you provide the right argument, puts them directly into a SproutCore-watched folder), the CSS is automatically added. So, how do you use it?

Very simply. The script generates CSS rules in this format:

.set-name .icon-name.icon, .set-name .icon-name.icon { /* CSS */ }

For example, to show “refresh-64″ icon in set “common”, you’d do something like this:

ImageView.design({
  layout: { left: 100, top: 100, width: 64, height: 64 },
  value: "common refresh-64 icon" // using SproutCore's built-in className support for spriting
}

Or, as you may have noticed from the rule above, you can accomplish some basic theming very easily by treating set names as themes:

View.design({
  layout: {left: 100, top:100, width:256, height: 256},
  classNames: ["common"], // the theme
  childViews: ["styledView"],
  styledView: ImageView.design({
    layout: { left: 100, top: 100, width: 64, height: 64 },
    value: "refresh-64 icon" // using SproutCore's built-in className support for spriting
  })
})

Nice and simple, eh?

Running Orbited (or other Twisted Services) as a Windows Service

Alex Iskander — Tue, 04 Aug 2009 14:47:15 +0000

It took me awhile to figure out how to run Orbited and Dolores as Windows Services. Unfortunately, while I love Mac, almost every other computer at my workplace is Windows, including the servers. When deploying my web application, I can’t really just leave instructions like: “When you restart the server, please run these two scripts.”

They need to be Windows Services.

Thankfully, it isn’t actually too hard. You just need a couple of Python packages (which I actually don’t remember installing, so they may have come with Twisted or Python), and then all you have to do is write a couple of very simple scripts. Here are mine:

Orbited Service

import OrbitedManager # a copy of orbited.start with some very minor modifications, which I'll include after the break
import time
 
# Service Utilities
import win32serviceutil
import win32service
import win32event
 
class WindowsService(win32serviceutil.ServiceFramework):
	_svc_name_ = "Orbited"
	_svc_display_name_ = "Orbited Server"
 
	def __init__(self, args):
		win32serviceutil.ServiceFramework.__init__(self, args)
		self.hWaitStop = win32event.CreateEvent(None, 0, 0, None)
 
	def SvcStop(self):
		self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
		win32event.SetEvent(self.hWaitStop)
 
	def SvcDoRun(self):
		import servicemanager
		OrbitedManager.start()
		while True:
			retval = win32event.WaitForSingleObject(self.hWaitStop, 10)
			if not retval == win32event.WAIT_TIMEOUT:
				OrbitedManager.stop()
				break
				time.sleep(5.0)
 
 
if __name__=='__main__':
	win32serviceutil.HandleCommandLine(WindowsService)

Dolores Service

import Dolores
import time
 
# Service Utilities
import win32serviceutil
import win32service
import win32event
 
class WindowsService(win32serviceutil.ServiceFramework):
	_svc_name_ = "Dolores"
	_svc_display_name_ = "Dolores Server"
 
	def __init__(self, args):
		win32serviceutil.ServiceFramework.__init__(self, args)
		self.hWaitStop = win32event.CreateEvent(None, 0, 0, None)
 
	def SvcStop(self):
		self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
		win32event.SetEvent(self.hWaitStop)
 
	def SvcDoRun(self):
		import servicemanager
		Dolores.start()
		while True:
			retval = win32event.WaitForSingleObject(self.hWaitStop, 10)
			if not retval == win32event.WAIT_TIMEOUT:
				Dolores.stop()
				break
				time.sleep(5.0)
 
 
if __name__=='__main__':
	win32serviceutil.HandleCommandLine(WindowsService)

That’s the basics, but as you may have noticed, I also changed Orbited’s start script. I did this because the original did not play nicely with threads.

Instead of changing it in its original location in the filesystem, I created a copy of it in the local directory (as you can see, I’m not importing Orbited.OrbitedManager).

Before overwhelming you with a wall of code, I’ll explain what I changed. It is actually quite simple:

First, some imports at the top:

import orbited
import threading
import time

Since the file is no longer “start.py” in the Orbited package directory, the static files path needs changing (just find the line and change it):
```
static_files = static.File(os.path.join(os.path.dirname(orbited.__path__[0]), 'orbited/static'))
```

And, since we’re going to start the reactor manually, we need to remove the auto-start portion:

# REMOVE THIS PART:
    if options.profile:
        import hotshot
        prof = hotshot.Profile("orbited.profile")
        logger.info("running Orbited in profile mode")
        logger.info("for information on profiler, see http://orbited.org/wiki/Profiler")
        prof.runcall(reactor.run)
        prof.close()
    else:
        reactor.run()

Finally, I added a bit to the bottom, replacing the lines that call “main()”. It arranges to run Orbited (or any Twisted service) in a separate thread:

def thread_start():
	main() # initializes the reactor
	from twisted.internet import reactor
	reactor.run(installSignalHandlers=0)
	print "Crashed into Earth's Surface."
 
orbited_thread = None
def start():
	print "Starting Orbited..."
	orbited_thread = threading.Thread(target=thread_start)
	orbited_thread.start()
	print "In Orbit."
 
def stop_reactor():
	reactor.stop()
 
def stop():
	reactor.callFromThread(stop_reactor)
 
if __name__ == "__main__":
	start()
	run = True
	while run:
		try:
			time.sleep(5)
		except KeyboardInterrupt:
			from twisted.internet import reactor
			stop()
			run = False

The Raw OrbitedManager.py File

import os
import sys
import urlparse
from orbited import __version__ as version
from orbited import config
from orbited import logging
import orbited
import threading
import time
 
# NB: this is set after we load the configuration at "main".
logger = None
 
def _import(name):
    module_import = name.rsplit('.', 1)[0]
    return reduce(getattr, name.split('.')[1:], __import__(module_import))
 
def _setup_protocols(root):
    from twisted.internet import reactor
    protocols = [
        #child_path config_key  port_class_import,              factory_class_import
        ('tcp',     'proxy',    'orbited.cometsession.Port',    'orbited.proxy.ProxyFactory'),
    ]
    for child_path, config_key, port_class_import, factory_class_import in protocols:
        if config.map['[global]'].get('%s.enabled' % config_key) == '1':
            port_class = _import(port_class_import)
            factory_class = _import(factory_class_import)
            reactor.listenWith(port_class, factory=factory_class(), resource=root, childName=child_path)
            logger.info('%s protocol active' % config_key)
 
def _setup_static(root, config):
    from twisted.web import static
    for key, val in config['[static]'].items():
        if key == 'INDEX':
            key = ''
        if root.getStaticEntity(key):
            logger.error("cannot mount static directory with reserved name %s" % key)
            sys.exit(1)
        root.putChild(key, static.File(val))
 
def main():
    try:
        import twisted
    except ImportError:
        print "Orbited requires Twisted, which is not installed. See http://twistedmatrix.com/trac/ for installation instructions."
        sys.exit(1)
 
    #################
    # This corrects a bug in Twisted 8.2.0 for certain Python 2.6 builds on Windows
    #   Twisted ticket: http://twistedmatrix.com/trac/ticket/3868
    #     -mario
    try:
        from twisted.python import lockfile
    except ImportError:
        from orbited import __path__ as orbited_path
        sys.path.append(os.path.join(orbited_path[0],"hotfixes","win32api"))
        from twisted.python import lockfile
        lockfile.kill = None
    #################
 
    from optparse import OptionParser
    parser = OptionParser()
    parser.add_option(
        "-c",
        "--config",
        dest="config",
        default=None,
        help="path to configuration file"
    )
    parser.add_option(
        "-v",
        "--version",
        dest="version",
        action="store_true",
        default=False,
        help="print Orbited version"
    )
    parser.add_option(
        "-p",
        "--profile",
        dest="profile",
        action="store_true",
        default=False,
        help="run Orbited with a profiler"
    )
    parser.add_option(
        "-q",
        "--quickstart",
        dest="quickstart",
        action="store_true",
        default=False,
        help="run Orbited on port 8000 and MorbidQ on port 61613"
    )
 
    (options, args) = parser.parse_args()
 
    if args:
        print 'the "orbited" command does not accept positional arguments. type "orbited -h" for options.'
        sys.exit(1)
 
    if options.version:
        print "Orbited version: %s" % (version,)
        sys.exit(0)
 
    if options.quickstart:
        config.map['[listen]'].append('http://:8000')
        config.map['[listen]'].append('stomp://:61613')
        config.map['[access]'][('localhost',61613)] = ['*']
        print "Quickstarting Orbited"
    else:
        # load configuration from configuration
        # file and from command line arguments.
        config.setup(options=options)
 
    logging.setup(config.map)
 
    # we can now safely get loggers.
    global logger; logger = logging.get_logger('orbited.start')
 
 
    # NB: we need to install the reactor before using twisted.
    reactor_name = config.map['[global]'].get('reactor')
    if reactor_name:
        install = _import('twisted.internet.%sreactor.install' % reactor_name)
        install()
        logger.info('using %s reactor' % reactor_name)
 
    ############
    # This crude garbage corrects a bug in twisted
    #   Orbited ticket: http://orbited.org/ticket/111
    #   Twisted ticket: http://twistedmatrix.com/trac/ticket/2447
    import twisted.web.http
    twisted.web.http.HTTPChannel.setTimeout = lambda self, arg: None
    twisted.web.http.HTTPChannel.resetTimeout = lambda self: None
    ############
 
    from twisted.internet import reactor
    from twisted.web import resource
    from twisted.web import server
    from twisted.web import static
    import orbited.system
 
    root = resource.Resource()
    static_files = static.File(os.path.join(os.path.dirname(orbited.__path__[0]), 'orbited/static'))
    root.putChild('static', static_files)
    root.putChild('system', orbited.system.SystemResource())
 
    if config.map['[test]']['stompdispatcher.enabled'] == '1':
        logger.info('stompdispatcher enabled')
 
    #static_files.putChild('orbited.swf', static.File(os.path.join(os.path.dirname(__file__), 'flash', 'orbited.swf')))
    site = server.Site(root)
 
    _setup_protocols(root)
    _setup_static(root, config.map)
    start_listening(site, config.map, logger)
 
    # switch uid and gid to configured user and group.
    if os.name == 'posix' and os.getuid() == 0:
        user = config.map['[global]'].get('user')
        group = config.map['[global]'].get('group')
        if user:
            import pwd
            import grp
            try:
                pw = pwd.getpwnam(user)
                uid = pw.pw_uid
                if group:
                    gr = grp.getgrnam(group)
                    gid = gr.gr_gid
                else:
                    gid = pw.pw_gid
                    gr = grp.getgrgid(gid)
                    group = gr.gr_name
            except Exception, e:
                logger.error('Aborting; Unknown user or group: %s' % e)
                sys.exit(1)
            logger.info('switching to user %s (uid=%d) and group %s (gid=%d)' % (user, uid, group, gid))
            os.setgid(gid)
            os.setuid(uid)
        else:
            logger.error('Aborting; You must define a user (and optionally a group) in the configuration file.')
            sys.exit(1)
 
 
def start_listening(site, config, logger):
    from twisted.internet import reactor
    from twisted.internet import protocol as protocol_module
    # allow stomp:// URIs to be parsed by urlparse
    urlparse.uses_netloc.append("stomp")
    # allow test server URIs to be parsed by urlparse
    from orbited.servers import test_servers
    for protocol in test_servers:
        urlparse.uses_netloc.append(protocol)
 
    for addr in config['[listen]']:
        if addr.startswith("stomp"):
            stompConfig = ""
            if " " in addr:
                addr, stompConfig = addr.split(" ",1)
        url = urlparse.urlparse(addr)
        hostname = url.hostname or ''
        if url.scheme == 'stomp':
            logger.info('Listening stomp@%s' % url.port)
            from morbid import get_stomp_factory
            morbid_instance = get_stomp_factory(stompConfig)
            config['morbid_instance'] = morbid_instance
            reactor.listenTCP(url.port, morbid_instance, interface=hostname)
        elif url.scheme == 'http':
            logger.info('Listening http@%s' % url.port)
            reactor.listenTCP(url.port, site, interface=hostname)
        elif url.scheme == 'https':
            from twisted.internet import ssl
            crt = config['[ssl]']['crt']
            key = config['[ssl]']['key']
            try:
                ssl_context = ssl.DefaultOpenSSLContextFactory(key, crt)
            except ImportError:
                raise
            except:
                logger.error("Error opening key or crt file: %s, %s" % (key, crt))
                sys.exit(1)
            logger.info('Listening https@%s (%s, %s)' % (url.port, key, crt))
            reactor.listenSSL(url.port, site, ssl_context, interface=hostname)
        elif url.scheme in test_servers:
            test_factory = protocol_module.ServerFactory()
            test_factory.protocol = test_servers[url.scheme]
            logger.info("Listening %s@%s"%(url.scheme, url.port))
            reactor.listenTCP(url.port, test_factory)
            if url.scheme == 'monitor':
                config['globalVars']['monitoring'] = url.port
        else:
            logger.error("Invalid Listen URI: %s" % addr)
            sys.exit(1)
 
 
def thread_start():
	main()
	from twisted.internet import reactor
	reactor.run(installSignalHandlers=0)
	print "Crashed into Earth's Surface."
 
 
orbited_thread = None
def start():
	print "Starting Orbited..."
	dolores_thread = threading.Thread(target=thread_start)
	dolores_thread.start()
	print "In Orbit."
 
def stop_reactor():
	reactor.stop()
 
def stop():
	reactor.callFromThread(stop_reactor)
 
if __name__ == "__main__":
	start()
	run = True
	while run:
		try:
			time.sleep(5)
		except KeyboardInterrupt:
			from twisted.internet import reactor
			stop()
			run = False

So, What Do I Do?

You should just need to put the windows service script and the OrbitedManager.py script in the same directory, and then run the windows service script from the command line. If you run it with no arguments, I believe it will give you an overview of available options, including the option to install.

I hope someone can find this helpful.

Update: I added some clarification in the middle, discussing high-level about the replacement of Orbited’s start module, and a bit at the end about what to do.

Dolores & Cornelius, Sitting in a Tree. Of Comet, Orbited and Python.

Alex Iskander — Tue, 04 Aug 2009 14:27:59 +0000

I’ll give you a moment to allow you to get that disturbing image out of your head.

I have been working on a web application. I’ll hopefully be able to post the application soon.

I needed the server to be able to send updates to the client via Comet. I’d never done any Comet before, so I wasn’t sure what to do.

A lot of articles had examples making use of message queues and who knows what else. They were a tad overwhelming. While a message queue of some sort may eventually be a good idea, for now, all I needed to do was push a non-vital progress indicator for a job (15%, 20%, 30%, and so on) to the client so the user could see what was happening. I wanted more than one job to be supported, and the overall system should eventually allow sending updates when elements in the user’s filesystem are updated.

So, I’d like to be able to send:

UPDATE process/at/my/process/path; {progress:.5}
UPDATE file/at/my/file/path.txt

In short, I’d like to send paths, along (sometimes) with little tiny messages.

Also, each instance of the web app should only receive updates to paths they are listening to, and they should only be allowed to listen to paths they actually have access to. Or, at least, that should be the case once I implement permission management in the app &emdash; right now, anything goes, which is fine for the moment because it is limited to test use on our local network.

So, I made a little script, called Dolores, which handles all of this, and another script, named Cornelius, which connects to it (incidentally, there are two Corneliuses: a Python one, and a JavaScript one).

I have posted it here in case anyone is interested. I used Orbited in JavaScript to connect to the server. In a separate post in just a few moments, I’ll write about something a tad more useful than the Dolores & Cornelius code I’m writing about here: how to run the Orbited (and, actually, Dolores) as a Windows service.

Disclaimer: This is for interest only. I’m proud of it because I managed to make it, not because I actually think it is wonderful.

Before getting to the code, let me explain the API, which, if you have the server running, you can access manually using telnet:

You connect, and immediately receive:

I, Dolores, High Inquisitor, Hogwarts.
{A-THREAD-ID-HERE}

That thread ID can then be used by a controller thread to connect you to paths.

Speaking of controllers, to control, you’d just make a file named “control-{THREAD-ID}” in a folder named “threads” relative to Dolores.py. Then, you’d send:

CONTROL

Now, to connect or disconnect a thread to a path:

CONNECT {THREAD-ID} some/path/or/another
DISCONNECT {THREAD-ID} some/path/or/another

In the web application, a django view handles the connection of a client (who sends its thread id along with the connection request). By only allowing controllers to create the connections, the client is not able to listen in to places it isn’t allowed.

And to send an update to a path:

UPDATE some/path/or/another; {some: message, may: go, right: here, format: "any"}

Any listening thread will immediately receive:

UPDATE some/path/or/another; {some: message, may: go, right: here, format: "any"}

Just as you sent it.

And yes, part of the reason it is named Dolores (the name of a particularly nasty character in the Harry Potter universe) is because I believe it to be a somewhat ugly solution. But unlike the HP character, it works. The HP character&emdash;let’s not go there.

Now, without further ado:

from twisted.internet.protocol import Protocol, Factory
from twisted.internet import reactor
from twisted.protocols.basic import LineReceiver
import random, string
import os, os.path
import threading
import time
 
# This is Delores the High Inquisitor.
 
# Dolores is a series of tubes. To prevent clogging, there are separate tubes,
# rather than one big pipe everyone might get stuck in.
 
# A user can be connected to a tube. This allows the user to receive notifications
# through said tubes.
 
# The pipe server is not made for actual COMET, believe it or not. There is no
# persistence or anything. It is as simple as possible. Instead, we will try to
# use Orbited to make a nice connection between Dolores and the client.
 
# I don't know how this will hold up performance-wise, but it is simple enough
# that I think it will work for now.
 
# API:
# There are two kinds of connections: control and client. The api 
# Creating a connection immediately results in a response with a thread id. Thread
# ids consist of a few main parts: IPADDRESS-THREAD#-SECRET
# Where thread# is a sequential thread number, IPADDRESS is the address of the client,
# and SECRET is a random string of bytes made for the thread.
 
# Clients connecting should check first for a string starting with "I,", which means they
# successfully connected. They should then read another string with their thread id.
 
#	CONTROL
#			Attempts to make the thread a controller. This only works if a file
#			named "control-" and the thread id is in the "threads" folder (relative
#			to the Dolores.py file -- messy, I know)
#
#	CONNECT UID PATH 			(controller only)
#			Connects the thread with id UID to the specified path.
#
#	DISCONNECT UID PATH			(controller only)
#			Disconnects the thread with id UID from the specified path.
#
#	UPDATE PATH MESSAGE 		(controller only)
#			Sens a message regarding the path. Please keep the message twitter length.
#
#	CRASH UID					(controller only)
#			Aborts a thread. Could be used for testing whole (un-optimized/cometized) reload.
#
 
 
class Comet(LineReceiver):
	def __init__(self):
		self.listening = set()
		self.controls = False
 
	def connectionMade(self):
		self.transport.write("I, Dolores, High Inquisitor, Hogwarts.\n")
 
		secret = ""
		for i in range(32):
			secret += random.choice(string.ascii_letters + string.digits)
 
		self.id = str(self.transport.getPeer().host) + "-" + str(self.factory.nextThreadNumber()) + "-" + secret
		print "New Thread " + self.id
 
		self.transport.write(self.id + "\n")
		self.factory.comets[self.id] = self;
 
	def control(self):
		# if the file exists, do it.
		if os.path.exists(os.path.join(os.path.dirname(__file__), "threads", "control-" + self.id)):
			self.controls = True
			os.remove(os.path.join(os.path.dirname(__file__), "threads", "control-" + self.id))
		else:
			self.controls = False
 
	def connectionLost(self, reason):
		print "Closed Thread " + self.id
		del self.factory.comets[self.id]
		for listen in self.listening:
			self.factory.deregister(listen, self)
 
	def listen(self, path):
		self.factory.register(path.strip(), self)
		self.listening.add(path.strip())
 
	def ignore(self, path):
		self.factory.deregister(path.strip(), self)
		if path.strip() in self.listening:
				self.listening.remove(path.strip())
 
	def lineReceived(self, data):
		if data.startswith("CONNECT"):
			if not self.controls:
				self.transport.write("NOALLOW\n")
				return
			try:
				uid, path = data[7:].strip().split(" ", 1)
				uid = uid.strip()
				path = path.strip()
				print "Connect " + uid + " to " + path
 
				self.factory.comets[uid].listen(path)
				self.listening.add(data[6:].strip())
				self.transport.write("SUCCESS\n")
			except:
				print "Failed a connect attempt."
				self.transport.write("FAIL\n")
 
		elif data.startswith("DISCONNECT"):
			if not self.controls:
				self.transport.write("NOALLOW\n")
				return
			try:
				uid, path = data[10:].strip().split(" ", 1)
				uid = uid.strip()
				path = path.strip()
				if uid in self.factory.comets:
					self.factory.comets[uid].ignore(path)
					self.transport.write("SUCCESS")
					return
			except:
				pass
			self.transport.write("FAIL")
 
		elif data.startswith("CRASH"):
			if not self.controls:
				self.transport.write("NOALLOW\n")
				return
			try:
				uid = data[5:].strip()
				if uid in self.factory.comets:
					self.factory.comets[uid].transport.loseConnection()
				self.transport.write("SUCCESS")
			except:
				self.transport.write("FAIL")
 
		elif data.startswith("EXIT"):
			self.transport.loseConnection()
 
		elif data.startswith("UPDATE"):
			if not self.controls:
				self.transport.write("NOALLOW\n")
				return
			try:
				pieces = data[6:].split(";",1)
				path = pieces[0]
				message = ""
				if len(pieces) > 1:
					message = pieces[1]
				print "UPDATE " + path.strip() + ": " + message.strip()
				self.factory.message(path.strip(), message.strip())
				self.transport.write("SUCCESS")
			except:
				self.transport.write("FAIL\n")
 
		elif data.startswith("ALL"):
			if not self.controls:
				self.transport.write("NOALLOW\n")
				return
			try:
				path = data[3:]
				for i in self.factory.comets:
					self.factory.comets[i].sendMessage(path)
				self.transport.write("SUCCESS")
			except:
				self.transport.write("FAIL")
 
		elif data.startswith("CONTROL"):
			self.control()
			if not self.controls:
				self.transport.write("NOALLOW\n")
				return
 
	def sendMessage(self, path, message):
		self.transport.write("UPDATE " + path + ";" + message + "\n")
 
# Factory
class Manager(Factory):
	def __init__(self):
		self.protocol = Comet
		self.comets = {}
		self.lookup = {}
		self.threadCount = 0
 
	def startFactory(self):
		pass
 
	def stopFactory(self):
		pass
 
	def register(self, path, comet):
		if path not in self.lookup:
			self.lookup[path] = set()
 
		self.lookup[path].add(comet)
 
	def deregister(self, path, comet):
		if path not in self.lookup:
			return
 
		if comet not in self.lookup[path]:
			return
 
		self.lookup[path].remove(comet)
 
		if len(self.lookup[path]) == 0:
			del self.lookup[path]
 
	def message(self, path, message):
		if path not in self.lookup:
			return
 
		for i in self.lookup[path]:
			i.sendMessage(path, message)
 
	def nextThreadNumber(self):
		self.threadCount = self.threadCount + 1
		return self.threadCount
 
def thread_start():
	global dolores_reactor
	factory = Manager()
	reactor.listenTCP(8007, factory)
	reactor.run(installSignalHandlers=0)
	print "Dolores has been taken by the Centaurs. Clip clop."
 
 
dolores_thread = None
def start():
	print "Starting Dolores..."
	global dolores_thread
	dolores_thread = threading.Thread(target=thread_start)
	dolores_thread.start()
	print "Dolores has  now been appointed High Inquisitor."
 
def stop_reactor():
	reactor.stop()
 
def stop():
	reactor.callFromThread(stop_reactor)
 
if __name__ == "__main__":
    start()
    run = True
    while run:
    	try:
    		time.sleep(5)
    	except KeyboardInterrupt:
    		stop()
    		run = False

And, the arguably even uglier Cornelius, with raw socket communication:

cornelius_connection = None
 
import socket
import time
import os, os.path
 
class CorneliusFault(Exception):
	def __init__(self, value):
		self.value = value
	def __str__(self):
		return repr(self.value)
 
 
def _connect(reconnect = False):
	global cornelius_connection
	if cornelius_connection and reconnect:
		try:
			cornelius_connection.shutdown()
		except:
			pass
		cornelius_connection = None
 
	if not cornelius_connection:
		cornelius_connection = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
		cornelius_connection.connect(("localhost", 8007))
		cornelius_connection.settimeout(1) # wait one second for response if needed. Hopefully won't be that long.
 
		buffer = read()
		if len(buffer) < 2:
			raise CorneliusFault("He's Not Back!")
 
		verify_dolores = buffer[0].strip()
		if verify_dolores.strip() != "I, Dolores, High Inquisitor, Hogwarts.":
			raise CorneliusFault("Lord Thingy! Lord Thingy! Dolores, why have you turned into Lord Thingy?")
 
		cornelius_thread_id = buffer[1].strip()
		f = open(os.path.dirname(os.path.abspath(__file__)) + "/threads/control-" + cornelius_thread_id, "w")
		f.write("you big dummy.")
		f.close()
		send("CONTROL\n\n", True)
		read()
 
def read():
	try:
		buffer = cornelius_connection.recv(4096)
		return buffer.split("\n")
	except:
		pass
 
def send(what, noconnect = False):
	if not noconnect:
		_connect()
	global cornelius_connection
	try:
		cornelius_connection.sendall(what + "\r\n")
	except:
		if not noconnect:
			_connect(True)
		cornelius_connection.sendall(what + "\r\n")
 
	try:
		read() # don't do anythin' with output yet. Just get rid of it... so it doesn't sit in some buffer
	except:
		pass
 
def update(path, message = ""):
	send("UPDATE " + path + "; " + message)
 
 
def connect(uid, path):
	send("CONNECT " + uid + " " + path)
 
def disconnect(uid, path):
	send("DISCONNECT " + uid + " " + path)
 
 
def crash(uid):
	send("CRASH " + uid)
 
def log():
	global cornelius_connection
	print cornelius_connection.read_very_eager()

Well, again, at least they work. And they are quite simple.

Dojo, Profiling, and Function Names

Alex Iskander — Tue, 23 Jun 2009 18:06:59 +0000

Since we are using Bespin for a new web project we are working on, it is only natural that we use the Dojo Toolkit as well. But we have had some issues with dojo.declare().

In short, when using dojo.declare(), all functions defined in a class are anonymous functions. It looks something like this:

Uh... What?

Nasty. Too many “anonymous” functions.

I recalled a write-up about how a new feature had been added to WebKit (and, by extension, Safari 4): functions can have display names just for this purpose. That would be perfect, except that I’d have to go to each function and add a "this.functionName.displayName = 'somename'", or something like that.

Then I realized that all of the classes are declared through one function: dojo.declare(). So, with a quick modification, I added these lines near the beginning of the function (you can find it in dojo.js.uncompressed.js, if you search for dojo.declare = function):

// preprocess props, updating functions
	for (var i in props)
		if (typeof props[i] == "function")
			props[i].displayName = i;

And now, that profiler output looks much better:

Much nicer output with displayName

And that’s it.

Side Note

I tried first to implement it by wrapping dojo.declare kind of like this:

dojo.olddeclare = dojo.declare;
dojo.declare = function(name, base, props) {
    //stuff here
    dojo.olddeclare(name, base props);
};

It didn’t work, I think because of something to do with the arguments.callee and a few other things. So, I settled for modifying the original function.

Update

A thought just occurred to me: you can display the class name quite easily too. Just change the line that sets the display name in the above example to:

props[i].displayName = className + "::" + i;

Notes on Bespin’s Model and Cursor Positions

Alex Iskander — Wed, 17 Jun 2009 19:43:40 +0000

I am writing this so that I will remember (as I have forgotten some of the things here before). However, since others may find it useful, and since I’d really like any double-checking I can get, I am publishing it.

In a web-based Create Framework related project, we are using Mozilla’s wonderful Bespin text editor. In the process of using it, we sometimes need to modify the editor in some way, perhaps by adding new features, perhaps by finding and fixing bugs.

The following is technical in nature.

In Bespin, characters are arranged in rows and columns. Any character may be referred to with a row number and a column number. This set of two numbers is referred to as a position.

Bespin’s editor has two significant parts which each have their own kind of position: the editor, and the model. This may seem silly at first: why should the editor’s position at row 5 column 4 be any different than the model’s?

There are a few different reasons for this, and the reasons could grow. It boils down to one concept: the editor, in order to enhance the usability, does not show characters exactly as they are in the model. For instance, a single tab in the model will usually be transformed into more than one space in the editor (for instance, four spaces). In this case, column 4 is the column immediately following a tab at the beginning of the line in the editor — the model position would be (5, 1), pointing to the column immediately following the tab character.

Also, code folding hides some lines of code, so row 4 could actually have been lines 4 through 30, but collapsed to take just one line. In this case, the model position for (5, 4) would be (31, 1).

The positions in the editor are called cursor positions (as they are possible positions of the cursor in the editor), and the positions in the model are called model positions.

As detailed in Gordon Hemsley’s guide on working with tabs, there are a few helper functions. There are two that are most important:

cursorManager.getCursorPosition(modelPosition);
cursorManager.getModelPosition(cursorPosition);

As you might expect by looking at their names and arguments, they convert cursor positions to model positions and vice-versa. There are two proxies to these functions in the editor component as well:

editor.getCursorPos(modelPosition); //note the lack of "ition"
editor.getModelPos(cursorPosition);

So what positions are used where? The model, naturally, always uses model positions. The editor always uses cursor positions. Actions (where most of the actual editing takes place) have convert between the two.

For instance, the action insertCharacter (in actions.js), used whenever you type in a letter (with extra comments inserted marking the conversions):

    insertCharacter: function(args) {
        if (this.editor.readonly) return;
 
        if (this.editor.selection) {
            this.deleteSelectionAndInsertCharacter(args);
        } else {
            // SEE HERE:
            // the "pos" argument given to insertCharacter is, of course, the current cursor position.
            // so, we convert it to a model position and hand it off to the model.
            this.model.insertCharacters(this.cursorManager.getModelPosition(args.pos), args.newchar);
            this.cursorManager.moveRight(true);
            this.repaint();
 
            // undo/redo
            args.action = "insertCharacter";
            var redoOperation = args;
            var undoArgs = { action: "deleteCharacter", pos: bespin.editor.utils.copyPos(args.pos), queued: args.queued };
            var undoOperation = undoArgs;
            this.editor.undoManager.addUndoOperation(new bespin.editor.UndoItem(undoOperation, redoOperation));
        }
    },