Create Blog » Comet http://create.tpsitulsa.com/blog the create framework blog Sat, 30 Jan 2010 20:08:19 +0000 en hourly 1 http://wordpress.org/?v=3.0 Roots Deliver Nutrients http://create.tpsitulsa.com/blog/2009/12/07/roots-deliver-nutrients/ http://create.tpsitulsa.com/blog/2009/12/07/roots-deliver-nutrients/#comments Mon, 07 Dec 2009 21:14:06 +0000 Alex Iskander http://create.tpsitulsa.com/blog/?p=245 Push is complex. I tried to figure it out with ActiveMQ, Orbited, and friends awhile back, but it just seemed too complicated. It made me feel like I was not very smart—obviously, some people understand this thing, but I don’t. I am a bit too prideful, and do not like feeling unintelligent.

But I need messaging.

I don’t care about queuing or persistence. For web applications (what I work on) messages should be delivered as quickly as possible. Any “queues” can be in-memory, and any message should not remain in a queue for more than a minute—if the client was disconnected for any duration, it would likely be better off re-downloading that receiving incremental updates.

So, whatever these systems do, they seem too complicated (I could be wrong—as I said, I have trouble understanding them).

My requirements:

  • Normal Authentication. Clients should go through the “normal” web app to subscribe to events.
  • Easy. It should be very simple and easy to understand.
  • Quick. I should be able to add Comet to an app in ten to fifteen minutes, maximum.

I found out that it is not only possible to write messaging servers on one’s own—it is relatively easy. So I wrote one.

I’ll stop the backstory here, though there is plenty more—tell me if you want any of the boring details about Dolores and the rest.

Roots

Roots—codenamed Dobby after the Harry Potter character—is a messaging server that has a built-in long-polling server (among several other things).

Roots is made up of many small message-passing pieces. They plug together, just like Lego pieces. Each piece is simple, and the connection between the pieces are even simpler: a single function, called “update.”

If you are interested, I’ll go over that more thoroughly in “Implementation.”

How Roots is Used

Above All, Roots is Easy.

It takes me about fifteen minutes to use Roots to add push to a SproutCore application (starting with a fresh clone of Roots—which, by the way, does not need any configuration for testing). It should only take me five minutes, but I am often a bit slow of a coder.

There is a brief semi-tutorial on GitHub. The following is based on that tutorial, but has some extra background information.

So, a bird’s-eye view: unless you implement the server-side in Roots itself, you will have three components in any application: back-end (Rails, Django, etc.), front-end (SproutCore), and Roots.

  • Client—such as Pomona, the SproutCore framework for Roots) connects to a Roots server.
  • Roots sends the client (Pomona) an id.
  • The client calls predefined URL to connect to events.
    • The URLs are part of the back-end (Django, etc.)
    • The ID is sent with these requests.
    • Clients (Pomona) usually will keep track of what server-side events you are listening to and can reconnect automatically if the Dobby connection is interrupted—you just provide the client with the URLs and the events to listen to.
  • Back-end tells Roots to connect client to events—and can perform any authorization it wishes.
  • Back-end sends updates to Roots.
  • Client receives notifications—everyone’s Happy!!!

What You Do (for SproutCore)

You don’t have to do much:

  1. Add Pomona to your SproutCore project.
  2. Add a Roots connector to your server-side project:
    • Cornelius for Python
    • Others as I add them (I plan Ruby and PHP)
    • Implement your own (the Python version of the better protocol—Dudley—is 45 lines counting whitespace and comments).
  3. “Update” Roots (call update(path, message) from your back-end—Django has handy model post-save signals you can use).
  4. Add some URLs to your back-end that connect clients.
    • URLs accept client ID in them (/my/connect/AN_ID_HERE or /my/connect.src?id=AN_ID_HERE)
    • URLs should accept POST of a simple JSON array of path/event names.
    • For each path/event/whatever you want to call it, call connect(id, path) on your Roots connector.
    • Initialize Pomona and connect to Roots in your SproutCore Data Source.
    • Accept incoming data!

A more detailed tutorial is here on GitHub with the rest of Dobby (Roots).

Open Question: Would anyone like a tutorial on setting up a Django back-end for the SproutCore Todos tutorial, with an extra step of adding Comet at the end? Done!

Implementation

Roots/Dobby is currently written in Python. I am working on a rewrite in node.js, because node.js is super cool—and, secondarily, it means that some of the same code used in the server (for instance, the message dispatch code) could be used on the client.

Roots has many small parts that hook together in a very simple way. At Roots’s core is Thestral, a simple interface that requires implementors to do two things: have an id and implement a single method: update(sender, path, message)—where sender is an originator, path is an event name, and message is some data to send.

Here are some of the components:

  • Dolores: This is the main controller. It assigns IDs to all Thestral instances. It itself is a Thestral instance, and sends its updates to “delegates”.
  • Pig: An “Owl,” or, really, a dispatcher; it knows who receives what message. It is often one of Dolores’s delegates. Specially-formed messages, tell it to connect certain Thestral ids with certain events—for instance, the event “::connect” with message “12345->my/data” would connect Thestral “12345″ to “my/data”.
  • Imperio. A somewhat pathetic text-based protocol, with accompanying receiver server (a bit like Dudley, which should be used instead). In addition to reading from a server, Imperio supports writing to anything with a write() method, and as such, is used for logging as a delegate of Dolores.
  • Dudley. Dudley is not a delegate of Dolores. Rather, Dolores can be its “delegate” (or, in this case, “receiver”). Dudley is an HTTP server, and receives messages over HTTP and funnels them to a receivers (Thestral implementation), and Dolores is its default receiver.
  • Firenze. Firenze, like Dudley, is an HTTP server; but instead of being a receiving server, it is a long-polling push server. It is not usually Dolores’s delegate—you don’t want every client to get every message. Instead, Firenze merely registers itself with Dolores to get an ID, and lets others register it (for instance, a client could send Dudley a message, which, through Dolores, would be received by Pig to make the connection).

Above all, it is simple. The longest of these pieces is Firenze, at 280 lines of code (50 of which are intro comments I used to explain it to myself and spec it while planning).

That’s All!

I hope you find this helpful. I am excited about porting Roots to node.js; progress is somewhat slow, as I am quite busy (I am a student, and finals for all four classes tomorrow!)

What I am most looking forward to is how Pig (the dispatcher) could use the same code in both the client (Pomona) and the server—the routing is quite similar; in one case, you want to route to handlers/callbacks, in the other, to update functions on Thestral instances—pretty similar; even identical.

I am also looking forward to experimenting with two-way push, where all data is transferred through push—completely eradicating the expectation of a response to a request, and providing only a single receive/send code path on both client and server.

Anyway, I think this is all super cool. :)

Thanks to Endash for the nice title. :) Roots (called Dobby in development) is available on GitHub.

P.S. I like the name Dobby, but Roots is better P.R.

]]> http://create.tpsitulsa.com/blog/2009/12/07/roots-deliver-nutrients/feed/ 7 Running Orbited (or other Twisted Services) as a Windows Service http://create.tpsitulsa.com/blog/2009/08/04/running-orbited-or-other-twisted-services-as-a-service/ http://create.tpsitulsa.com/blog/2009/08/04/running-orbited-or-other-twisted-services-as-a-service/#comments Tue, 04 Aug 2009 14:47:15 +0000 Alex Iskander http://create.tpsitulsa.com/blog/?p=185 It took me awhile to figure out how to run Orbited and Dolores as Windows Services. Unfortunately, while I love Mac, almost every other computer at my workplace is Windows, including the servers. When deploying my web application, I can’t really just leave instructions like: “When you restart the server, please run these two scripts.”

They need to be Windows Services.

Thankfully, it isn’t actually too hard. You just need a couple of Python packages (which I actually don’t remember installing, so they may have come with Twisted or Python), and then all you have to do is write a couple of very simple scripts. Here are mine:

Orbited Service

import OrbitedManager # a copy of orbited.start with some very minor modifications, which I'll include after the break
import time
 
# Service Utilities
import win32serviceutil
import win32service
import win32event
 
class WindowsService(win32serviceutil.ServiceFramework):
	_svc_name_ = "Orbited"
	_svc_display_name_ = "Orbited Server"
 
	def __init__(self, args):
		win32serviceutil.ServiceFramework.__init__(self, args)
		self.hWaitStop = win32event.CreateEvent(None, 0, 0, None)
 
	def SvcStop(self):
		self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
		win32event.SetEvent(self.hWaitStop)
 
	def SvcDoRun(self):
		import servicemanager
		OrbitedManager.start()
		while True:
			retval = win32event.WaitForSingleObject(self.hWaitStop, 10)
			if not retval == win32event.WAIT_TIMEOUT:
				OrbitedManager.stop()
				break
				time.sleep(5.0)
 
 
if __name__=='__main__':
	win32serviceutil.HandleCommandLine(WindowsService)

Dolores Service

import Dolores
import time
 
# Service Utilities
import win32serviceutil
import win32service
import win32event
 
class WindowsService(win32serviceutil.ServiceFramework):
	_svc_name_ = "Dolores"
	_svc_display_name_ = "Dolores Server"
 
	def __init__(self, args):
		win32serviceutil.ServiceFramework.__init__(self, args)
		self.hWaitStop = win32event.CreateEvent(None, 0, 0, None)
 
	def SvcStop(self):
		self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
		win32event.SetEvent(self.hWaitStop)
 
	def SvcDoRun(self):
		import servicemanager
		Dolores.start()
		while True:
			retval = win32event.WaitForSingleObject(self.hWaitStop, 10)
			if not retval == win32event.WAIT_TIMEOUT:
				Dolores.stop()
				break
				time.sleep(5.0)
 
 
if __name__=='__main__':
	win32serviceutil.HandleCommandLine(WindowsService)

That’s the basics, but as you may have noticed, I also changed Orbited’s start script. I did this because the original did not play nicely with threads.

Instead of changing it in its original location in the filesystem, I created a copy of it in the local directory (as you can see, I’m not importing Orbited.OrbitedManager).

Before overwhelming you with a wall of code, I’ll explain what I changed. It is actually quite simple:

  1. First, some imports at the top: 
    import orbited
import threading
import time
  2. Since the file is no longer “start.py” in the Orbited package directory, the static files path needs changing (just find the line and change it): 
    static_files = static.File(os.path.join(os.path.dirname(orbited.__path__[0]), 'orbited/static'))
  3. And, since we’re going to start the reactor manually, we need to remove the auto-start portion: 
    # REMOVE THIS PART:
    if options.profile:
        import hotshot
        prof = hotshot.Profile("orbited.profile")
        logger.info("running Orbited in profile mode")
        logger.info("for information on profiler, see http://orbited.org/wiki/Profiler")
        prof.runcall(reactor.run)
        prof.close()
    else:
        reactor.run()
  4. Finally, I added a bit to the bottom, replacing the lines that call “main()”. It arranges to run Orbited (or any Twisted service) in a separate thread: 
    def thread_start():
	main() # initializes the reactor
	from twisted.internet import reactor
	reactor.run(installSignalHandlers=0)
	print "Crashed into Earth's Surface."
 
orbited_thread = None
def start():
	print "Starting Orbited..."
	orbited_thread = threading.Thread(target=thread_start)
	orbited_thread.start()
	print "In Orbit."
 
def stop_reactor():
	reactor.stop()
 
def stop():
	reactor.callFromThread(stop_reactor)
 
if __name__ == "__main__":
	start()
	run = True
	while run:
		try:
			time.sleep(5)
		except KeyboardInterrupt:
			from twisted.internet import reactor
			stop()
			run = False

The Raw OrbitedManager.py File

import os
import sys
import urlparse
from orbited import __version__ as version
from orbited import config
from orbited import logging
import orbited
import threading
import time
 
# NB: this is set after we load the configuration at "main".
logger = None
 
def _import(name):
    module_import = name.rsplit('.', 1)[0]
    return reduce(getattr, name.split('.')[1:], __import__(module_import))
 
def _setup_protocols(root):
    from twisted.internet import reactor
    protocols = [
        #child_path config_key  port_class_import,              factory_class_import
        ('tcp',     'proxy',    'orbited.cometsession.Port',    'orbited.proxy.ProxyFactory'),
    ]
    for child_path, config_key, port_class_import, factory_class_import in protocols:
        if config.map['[global]'].get('%s.enabled' % config_key) == '1':
            port_class = _import(port_class_import)
            factory_class = _import(factory_class_import)
            reactor.listenWith(port_class, factory=factory_class(), resource=root, childName=child_path)
            logger.info('%s protocol active' % config_key)
 
def _setup_static(root, config):
    from twisted.web import static
    for key, val in config['[static]'].items():
        if key == 'INDEX':
            key = ''
        if root.getStaticEntity(key):
            logger.error("cannot mount static directory with reserved name %s" % key)
            sys.exit(1)
        root.putChild(key, static.File(val))
 
def main():
    try:
        import twisted
    except ImportError:
        print "Orbited requires Twisted, which is not installed. See http://twistedmatrix.com/trac/ for installation instructions."
        sys.exit(1)
 
    #################
    # This corrects a bug in Twisted 8.2.0 for certain Python 2.6 builds on Windows
    #   Twisted ticket: http://twistedmatrix.com/trac/ticket/3868
    #     -mario
    try:
        from twisted.python import lockfile
    except ImportError:
        from orbited import __path__ as orbited_path
        sys.path.append(os.path.join(orbited_path[0],"hotfixes","win32api"))
        from twisted.python import lockfile
        lockfile.kill = None
    #################
 
    from optparse import OptionParser
    parser = OptionParser()
    parser.add_option(
        "-c",
        "--config",
        dest="config",
        default=None,
        help="path to configuration file"
    )
    parser.add_option(
        "-v",
        "--version",
        dest="version",
        action="store_true",
        default=False,
        help="print Orbited version"
    )
    parser.add_option(
        "-p",
        "--profile",
        dest="profile",
        action="store_true",
        default=False,
        help="run Orbited with a profiler"
    )
    parser.add_option(
        "-q",
        "--quickstart",
        dest="quickstart",
        action="store_true",
        default=False,
        help="run Orbited on port 8000 and MorbidQ on port 61613"
    )
 
    (options, args) = parser.parse_args()
 
    if args:
        print 'the "orbited" command does not accept positional arguments. type "orbited -h" for options.'
        sys.exit(1)
 
    if options.version:
        print "Orbited version: %s" % (version,)
        sys.exit(0)
 
    if options.quickstart:
        config.map['[listen]'].append('http://:8000')
        config.map['[listen]'].append('stomp://:61613')
        config.map['[access]'][('localhost',61613)] = ['*']
        print "Quickstarting Orbited"
    else:
        # load configuration from configuration
        # file and from command line arguments.
        config.setup(options=options)
 
    logging.setup(config.map)
 
    # we can now safely get loggers.
    global logger; logger = logging.get_logger('orbited.start')
 
 
    # NB: we need to install the reactor before using twisted.
    reactor_name = config.map['[global]'].get('reactor')
    if reactor_name:
        install = _import('twisted.internet.%sreactor.install' % reactor_name)
        install()
        logger.info('using %s reactor' % reactor_name)
 
    ############
    # This crude garbage corrects a bug in twisted
    #   Orbited ticket: http://orbited.org/ticket/111
    #   Twisted ticket: http://twistedmatrix.com/trac/ticket/2447
    import twisted.web.http
    twisted.web.http.HTTPChannel.setTimeout = lambda self, arg: None
    twisted.web.http.HTTPChannel.resetTimeout = lambda self: None
    ############
 
    from twisted.internet import reactor
    from twisted.web import resource
    from twisted.web import server
    from twisted.web import static
    import orbited.system
 
    root = resource.Resource()
    static_files = static.File(os.path.join(os.path.dirname(orbited.__path__[0]), 'orbited/static'))
    root.putChild('static', static_files)
    root.putChild('system', orbited.system.SystemResource())
 
    if config.map['[test]']['stompdispatcher.enabled'] == '1':
        logger.info('stompdispatcher enabled')
 
    #static_files.putChild('orbited.swf', static.File(os.path.join(os.path.dirname(__file__), 'flash', 'orbited.swf')))
    site = server.Site(root)
 
    _setup_protocols(root)
    _setup_static(root, config.map)
    start_listening(site, config.map, logger)
 
    # switch uid and gid to configured user and group.
    if os.name == 'posix' and os.getuid() == 0:
        user = config.map['[global]'].get('user')
        group = config.map['[global]'].get('group')
        if user:
            import pwd
            import grp
            try:
                pw = pwd.getpwnam(user)
                uid = pw.pw_uid
                if group:
                    gr = grp.getgrnam(group)
                    gid = gr.gr_gid
                else:
                    gid = pw.pw_gid
                    gr = grp.getgrgid(gid)
                    group = gr.gr_name
            except Exception, e:
                logger.error('Aborting; Unknown user or group: %s' % e)
                sys.exit(1)
            logger.info('switching to user %s (uid=%d) and group %s (gid=%d)' % (user, uid, group, gid))
            os.setgid(gid)
            os.setuid(uid)
        else:
            logger.error('Aborting; You must define a user (and optionally a group) in the configuration file.')
            sys.exit(1)
 
 
def start_listening(site, config, logger):
    from twisted.internet import reactor
    from twisted.internet import protocol as protocol_module
    # allow stomp:// URIs to be parsed by urlparse
    urlparse.uses_netloc.append("stomp")
    # allow test server URIs to be parsed by urlparse
    from orbited.servers import test_servers
    for protocol in test_servers:
        urlparse.uses_netloc.append(protocol)
 
    for addr in config['[listen]']:
        if addr.startswith("stomp"):
            stompConfig = ""
            if " " in addr:
                addr, stompConfig = addr.split(" ",1)
        url = urlparse.urlparse(addr)
        hostname = url.hostname or ''
        if url.scheme == 'stomp':
            logger.info('Listening stomp@%s' % url.port)
            from morbid import get_stomp_factory
            morbid_instance = get_stomp_factory(stompConfig)
            config['morbid_instance'] = morbid_instance
            reactor.listenTCP(url.port, morbid_instance, interface=hostname)
        elif url.scheme == 'http':
            logger.info('Listening http@%s' % url.port)
            reactor.listenTCP(url.port, site, interface=hostname)
        elif url.scheme == 'https':
            from twisted.internet import ssl
            crt = config['[ssl]']['crt']
            key = config['[ssl]']['key']
            try:
                ssl_context = ssl.DefaultOpenSSLContextFactory(key, crt)
            except ImportError:
                raise
            except:
                logger.error("Error opening key or crt file: %s, %s" % (key, crt))
                sys.exit(1)
            logger.info('Listening https@%s (%s, %s)' % (url.port, key, crt))
            reactor.listenSSL(url.port, site, ssl_context, interface=hostname)
        elif url.scheme in test_servers:
            test_factory = protocol_module.ServerFactory()
            test_factory.protocol = test_servers[url.scheme]
            logger.info("Listening %s@%s"%(url.scheme, url.port))
            reactor.listenTCP(url.port, test_factory)
            if url.scheme == 'monitor':
                config['globalVars']['monitoring'] = url.port
        else:
            logger.error("Invalid Listen URI: %s" % addr)
            sys.exit(1)
 
 
def thread_start():
	main()
	from twisted.internet import reactor
	reactor.run(installSignalHandlers=0)
	print "Crashed into Earth's Surface."
 
 
orbited_thread = None
def start():
	print "Starting Orbited..."
	dolores_thread = threading.Thread(target=thread_start)
	dolores_thread.start()
	print "In Orbit."
 
def stop_reactor():
	reactor.stop()
 
def stop():
	reactor.callFromThread(stop_reactor)
 
if __name__ == "__main__":
	start()
	run = True
	while run:
		try:
			time.sleep(5)
		except KeyboardInterrupt:
			from twisted.internet import reactor
			stop()
			run = False

So, What Do I Do?

You should just need to put the windows service script and the OrbitedManager.py script in the same directory, and then run the windows service script from the command line. If you run it with no arguments, I believe it will give you an overview of available options, including the option to install.

I hope someone can find this helpful.

Update: I added some clarification in the middle, discussing high-level about the replacement of Orbited’s start module, and a bit at the end about what to do.

]]> http://create.tpsitulsa.com/blog/2009/08/04/running-orbited-or-other-twisted-services-as-a-service/feed/ 0 Dolores & Cornelius, Sitting in a Tree. Of Comet, Orbited and Python. http://create.tpsitulsa.com/blog/2009/08/04/dolores/ http://create.tpsitulsa.com/blog/2009/08/04/dolores/#comments Tue, 04 Aug 2009 14:27:59 +0000 Alex Iskander http://create.tpsitulsa.com/blog/?p=175 I’ll give you a moment to allow you to get that disturbing image out of your head.

I have been working on a web application. I’ll hopefully be able to post the application soon.

I needed the server to be able to send updates to the client via Comet. I’d never done any Comet before, so I wasn’t sure what to do.

A lot of articles had examples making use of message queues and who knows what else. They were a tad overwhelming. While a message queue of some sort may eventually be a good idea, for now, all I needed to do was push a non-vital progress indicator for a job (15%, 20%, 30%, and so on) to the client so the user could see what was happening. I wanted more than one job to be supported, and the overall system should eventually allow sending updates when elements in the user’s filesystem are updated.

So, I’d like to be able to send:

UPDATE process/at/my/process/path; {progress:.5}
UPDATE file/at/my/file/path.txt

In short, I’d like to send paths, along (sometimes) with little tiny messages.

Also, each instance of the web app should only receive updates to paths they are listening to, and they should only be allowed to listen to paths they actually have access to. Or, at least, that should be the case once I implement permission management in the app &emdash; right now, anything goes, which is fine for the moment because it is limited to test use on our local network.

So, I made a little script, called Dolores, which handles all of this, and another script, named Cornelius, which connects to it (incidentally, there are two Corneliuses: a Python one, and a JavaScript one).

I have posted it here in case anyone is interested. I used Orbited in JavaScript to connect to the server. In a separate post in just a few moments, I’ll write about something a tad more useful than the Dolores & Cornelius code I’m writing about here: how to run the Orbited (and, actually, Dolores) as a Windows service.

Disclaimer: This is for interest only. I’m proud of it because I managed to make it, not because I actually think it is wonderful.

Before getting to the code, let me explain the API, which, if you have the server running, you can access manually using telnet:

You connect, and immediately receive:

I, Dolores, High Inquisitor, Hogwarts.
{A-THREAD-ID-HERE}

That thread ID can then be used by a controller thread to connect you to paths.

Speaking of controllers, to control, you’d just make a file named “control-{THREAD-ID}” in a folder named “threads” relative to Dolores.py. Then, you’d send:

CONTROL

Now, to connect or disconnect a thread to a path:

CONNECT {THREAD-ID} some/path/or/another
DISCONNECT {THREAD-ID} some/path/or/another

In the web application, a django view handles the connection of a client (who sends its thread id along with the connection request). By only allowing controllers to create the connections, the client is not able to listen in to places it isn’t allowed.

And to send an update to a path:

UPDATE some/path/or/another; {some: message, may: go, right: here, format: "any"}

Any listening thread will immediately receive:

UPDATE some/path/or/another; {some: message, may: go, right: here, format: "any"}

Just as you sent it.

And yes, part of the reason it is named Dolores (the name of a particularly nasty character in the Harry Potter universe) is because I believe it to be a somewhat ugly solution. But unlike the HP character, it works. The HP character&emdash;let’s not go there.

Now, without further ado:

from twisted.internet.protocol import Protocol, Factory
from twisted.internet import reactor
from twisted.protocols.basic import LineReceiver
import random, string
import os, os.path
import threading
import time
 
# This is Delores the High Inquisitor.
 
# Dolores is a series of tubes. To prevent clogging, there are separate tubes,
# rather than one big pipe everyone might get stuck in.
 
# A user can be connected to a tube. This allows the user to receive notifications
# through said tubes.
 
# The pipe server is not made for actual COMET, believe it or not. There is no
# persistence or anything. It is as simple as possible. Instead, we will try to
# use Orbited to make a nice connection between Dolores and the client.
 
# I don't know how this will hold up performance-wise, but it is simple enough
# that I think it will work for now.
 
# API:
# There are two kinds of connections: control and client. The api 
# Creating a connection immediately results in a response with a thread id. Thread
# ids consist of a few main parts: IPADDRESS-THREAD#-SECRET
# Where thread# is a sequential thread number, IPADDRESS is the address of the client,
# and SECRET is a random string of bytes made for the thread.
 
# Clients connecting should check first for a string starting with "I,", which means they
# successfully connected. They should then read another string with their thread id.
 
#	CONTROL
#			Attempts to make the thread a controller. This only works if a file
#			named "control-" and the thread id is in the "threads" folder (relative
#			to the Dolores.py file -- messy, I know)
#
#	CONNECT UID PATH 			(controller only)
#			Connects the thread with id UID to the specified path.
#
#	DISCONNECT UID PATH			(controller only)
#			Disconnects the thread with id UID from the specified path.
#
#	UPDATE PATH MESSAGE 		(controller only)
#			Sens a message regarding the path. Please keep the message twitter length.
#
#	CRASH UID					(controller only)
#			Aborts a thread. Could be used for testing whole (un-optimized/cometized) reload.
#
 
 
class Comet(LineReceiver):
	def __init__(self):
		self.listening = set()
		self.controls = False
 
	def connectionMade(self):
		self.transport.write("I, Dolores, High Inquisitor, Hogwarts.\n")
 
		secret = ""
		for i in range(32):
			secret += random.choice(string.ascii_letters + string.digits)
 
		self.id = str(self.transport.getPeer().host) + "-" + str(self.factory.nextThreadNumber()) + "-" + secret
		print "New Thread " + self.id
 
		self.transport.write(self.id + "\n")
		self.factory.comets[self.id] = self;
 
	def control(self):
		# if the file exists, do it.
		if os.path.exists(os.path.join(os.path.dirname(__file__), "threads", "control-" + self.id)):
			self.controls = True
			os.remove(os.path.join(os.path.dirname(__file__), "threads", "control-" + self.id))
		else:
			self.controls = False
 
	def connectionLost(self, reason):
		print "Closed Thread " + self.id
		del self.factory.comets[self.id]
		for listen in self.listening:
			self.factory.deregister(listen, self)
 
	def listen(self, path):
		self.factory.register(path.strip(), self)
		self.listening.add(path.strip())
 
	def ignore(self, path):
		self.factory.deregister(path.strip(), self)
		if path.strip() in self.listening:
				self.listening.remove(path.strip())
 
	def lineReceived(self, data):
		if data.startswith("CONNECT"):
			if not self.controls:
				self.transport.write("NOALLOW\n")
				return
			try:
				uid, path = data[7:].strip().split(" ", 1)
				uid = uid.strip()
				path = path.strip()
				print "Connect " + uid + " to " + path
 
				self.factory.comets[uid].listen(path)
				self.listening.add(data[6:].strip())
				self.transport.write("SUCCESS\n")
			except:
				print "Failed a connect attempt."
				self.transport.write("FAIL\n")
 
		elif data.startswith("DISCONNECT"):
			if not self.controls:
				self.transport.write("NOALLOW\n")
				return
			try:
				uid, path = data[10:].strip().split(" ", 1)
				uid = uid.strip()
				path = path.strip()
				if uid in self.factory.comets:
					self.factory.comets[uid].ignore(path)
					self.transport.write("SUCCESS")
					return
			except:
				pass
			self.transport.write("FAIL")
 
		elif data.startswith("CRASH"):
			if not self.controls:
				self.transport.write("NOALLOW\n")
				return
			try:
				uid = data[5:].strip()
				if uid in self.factory.comets:
					self.factory.comets[uid].transport.loseConnection()
				self.transport.write("SUCCESS")
			except:
				self.transport.write("FAIL")
 
		elif data.startswith("EXIT"):
			self.transport.loseConnection()
 
		elif data.startswith("UPDATE"):
			if not self.controls:
				self.transport.write("NOALLOW\n")
				return
			try:
				pieces = data[6:].split(";",1)
				path = pieces[0]
				message = ""
				if len(pieces) > 1:
					message = pieces[1]
				print "UPDATE " + path.strip() + ": " + message.strip()
				self.factory.message(path.strip(), message.strip())
				self.transport.write("SUCCESS")
			except:
				self.transport.write("FAIL\n")
 
		elif data.startswith("ALL"):
			if not self.controls:
				self.transport.write("NOALLOW\n")
				return
			try:
				path = data[3:]
				for i in self.factory.comets:
					self.factory.comets[i].sendMessage(path)
				self.transport.write("SUCCESS")
			except:
				self.transport.write("FAIL")
 
		elif data.startswith("CONTROL"):
			self.control()
			if not self.controls:
				self.transport.write("NOALLOW\n")
				return
 
	def sendMessage(self, path, message):
		self.transport.write("UPDATE " + path + ";" + message + "\n")
 
# Factory
class Manager(Factory):
	def __init__(self):
		self.protocol = Comet
		self.comets = {}
		self.lookup = {}
		self.threadCount = 0
 
	def startFactory(self):
		pass
 
	def stopFactory(self):
		pass
 
	def register(self, path, comet):
		if path not in self.lookup:
			self.lookup[path] = set()
 
		self.lookup[path].add(comet)
 
	def deregister(self, path, comet):
		if path not in self.lookup:
			return
 
		if comet not in self.lookup[path]:
			return
 
		self.lookup[path].remove(comet)
 
		if len(self.lookup[path]) == 0:
			del self.lookup[path]
 
	def message(self, path, message):
		if path not in self.lookup:
			return
 
		for i in self.lookup[path]:
			i.sendMessage(path, message)
 
	def nextThreadNumber(self):
		self.threadCount = self.threadCount + 1
		return self.threadCount
 
def thread_start():
	global dolores_reactor
	factory = Manager()
	reactor.listenTCP(8007, factory)
	reactor.run(installSignalHandlers=0)
	print "Dolores has been taken by the Centaurs. Clip clop."
 
 
dolores_thread = None
def start():
	print "Starting Dolores..."
	global dolores_thread
	dolores_thread = threading.Thread(target=thread_start)
	dolores_thread.start()
	print "Dolores has  now been appointed High Inquisitor."
 
def stop_reactor():
	reactor.stop()
 
def stop():
	reactor.callFromThread(stop_reactor)
 
if __name__ == "__main__":
    start()
    run = True
    while run:
    	try:
    		time.sleep(5)
    	except KeyboardInterrupt:
    		stop()
    		run = False

And, the arguably even uglier Cornelius, with raw socket communication:

cornelius_connection = None
 
import socket
import time
import os, os.path
 
class CorneliusFault(Exception):
	def __init__(self, value):
		self.value = value
	def __str__(self):
		return repr(self.value)
 
 
def _connect(reconnect = False):
	global cornelius_connection
	if cornelius_connection and reconnect:
		try:
			cornelius_connection.shutdown()
		except:
			pass
		cornelius_connection = None
 
	if not cornelius_connection:
		cornelius_connection = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
		cornelius_connection.connect(("localhost", 8007))
		cornelius_connection.settimeout(1) # wait one second for response if needed. Hopefully won't be that long.
 
		buffer = read()
		if len(buffer) < 2:
			raise CorneliusFault("He's Not Back!")
 
		verify_dolores = buffer[0].strip()
		if verify_dolores.strip() != "I, Dolores, High Inquisitor, Hogwarts.":
			raise CorneliusFault("Lord Thingy! Lord Thingy! Dolores, why have you turned into Lord Thingy?")
 
		cornelius_thread_id = buffer[1].strip()
		f = open(os.path.dirname(os.path.abspath(__file__)) + "/threads/control-" + cornelius_thread_id, "w")
		f.write("you big dummy.")
		f.close()
		send("CONTROL\n\n", True)
		read()
 
def read():
	try:
		buffer = cornelius_connection.recv(4096)
		return buffer.split("\n")
	except:
		pass
 
def send(what, noconnect = False):
	if not noconnect:
		_connect()
	global cornelius_connection
	try:
		cornelius_connection.sendall(what + "\r\n")
	except:
		if not noconnect:
			_connect(True)
		cornelius_connection.sendall(what + "\r\n")
 
	try:
		read() # don't do anythin' with output yet. Just get rid of it... so it doesn't sit in some buffer
	except:
		pass
 
def update(path, message = ""):
	send("UPDATE " + path + "; " + message)
 
 
def connect(uid, path):
	send("CONNECT " + uid + " " + path)
 
def disconnect(uid, path):
	send("DISCONNECT " + uid + " " + path)
 
 
def crash(uid):
	send("CRASH " + uid)
 
def log():
	global cornelius_connection
	print cornelius_connection.read_very_eager()

Well, again, at least they work. And they are quite simple.

]]> http://create.tpsitulsa.com/blog/2009/08/04/dolores/feed/ 1