create blog

go home go home
  1. about
  2. code
  3. wiki
  4. blog

Archive for the ‘Python’ Category

Roots Deliver Nutrients

Monday, December 7th, 2009

Push is complex. I tried to figure it out with ActiveMQ, Orbited, and friends awhile back, but it just seemed too complicated. It made me feel like I was not very smart—obviously, some people understand this thing, but I don’t. I am a bit too prideful, and do not like feeling unintelligent.

But I need messaging.

I don’t care about queuing or persistence. For web applications (what I work on) messages should be delivered as quickly as possible. Any “queues” can be in-memory, and any message should not remain in a queue for more than a minute—if the client was disconnected for any duration, it would likely be better off re-downloading that receiving incremental updates.

So, whatever these systems do, they seem too complicated (I could be wrong—as I said, I have trouble understanding them).

My requirements:

  • Normal Authentication. Clients should go through the “normal” web app to subscribe to events.
  • Easy. It should be very simple and easy to understand.
  • Quick. I should be able to add Comet to an app in ten to fifteen minutes, maximum.

I found out that it is not only possible to write messaging servers on one’s own—it is relatively easy. So I wrote one.

I’ll stop the backstory here, though there is plenty more—tell me if you want any of the boring details about Dolores and the rest.

Roots

Roots—codenamed Dobby after the Harry Potter character—is a messaging server that has a built-in long-polling server (among several other things).

Roots is made up of many small message-passing pieces. They plug together, just like Lego pieces. Each piece is simple, and the connection between the pieces are even simpler: a single function, called “update.”

If you are interested, I’ll go over that more thoroughly in “Implementation.”

How Roots is Used

Above All, Roots is Easy.

It takes me about fifteen minutes to use Roots to add push to a SproutCore application (starting with a fresh clone of Roots—which, by the way, does not need any configuration for testing). It should only take me five minutes, but I am often a bit slow of a coder.

There is a brief semi-tutorial on GitHub. The following is based on that tutorial, but has some extra background information.

So, a bird’s-eye view: unless you implement the server-side in Roots itself, you will have three components in any application: back-end (Rails, Django, etc.), front-end (SproutCore), and Roots.

  • Client—such as Pomona, the SproutCore framework for Roots) connects to a Roots server.
  • Roots sends the client (Pomona) an id.
  • The client calls predefined URL to connect to events.
    • The URLs are part of the back-end (Django, etc.)
    • The ID is sent with these requests.
    • Clients (Pomona) usually will keep track of what server-side events you are listening to and can reconnect automatically if the Dobby connection is interrupted—you just provide the client with the URLs and the events to listen to.
  • Back-end tells Roots to connect client to events—and can perform any authorization it wishes.
  • Back-end sends updates to Roots.
  • Client receives notifications—everyone’s Happy!!!

What You Do (for SproutCore)

You don’t have to do much:

  1. Add Pomona to your SproutCore project.
  2. Add a Roots connector to your server-side project:
    • Cornelius for Python
    • Others as I add them (I plan Ruby and PHP)
    • Implement your own (the Python version of the better protocol—Dudley—is 45 lines counting whitespace and comments).
  3. “Update” Roots (call update(path, message) from your back-end—Django has handy model post-save signals you can use).
  4. Add some URLs to your back-end that connect clients.
    • URLs accept client ID in them (/my/connect/AN_ID_HERE or /my/connect.src?id=AN_ID_HERE)
    • URLs should accept POST of a simple JSON array of path/event names.
    • For each path/event/whatever you want to call it, call connect(id, path) on your Roots connector.
    • Initialize Pomona and connect to Roots in your SproutCore Data Source.
    • Accept incoming data!

A more detailed tutorial is here on GitHub with the rest of Dobby (Roots).

Open Question: Would anyone like a tutorial on setting up a Django back-end for the SproutCore Todos tutorial, with an extra step of adding Comet at the end? Done!

Implementation

Roots/Dobby is currently written in Python. I am working on a rewrite in node.js, because node.js is super cool—and, secondarily, it means that some of the same code used in the server (for instance, the message dispatch code) could be used on the client.

Roots has many small parts that hook together in a very simple way. At Roots’s core is Thestral, a simple interface that requires implementors to do two things: have an id and implement a single method: update(sender, path, message)—where sender is an originator, path is an event name, and message is some data to send.

Here are some of the components:

  • Dolores: This is the main controller. It assigns IDs to all Thestral instances. It itself is a Thestral instance, and sends its updates to “delegates”.
  • Pig: An “Owl,” or, really, a dispatcher; it knows who receives what message. It is often one of Dolores’s delegates. Specially-formed messages, tell it to connect certain Thestral ids with certain events—for instance, the event “::connect” with message “12345->my/data” would connect Thestral “12345″ to “my/data”.
  • Imperio. A somewhat pathetic text-based protocol, with accompanying receiver server (a bit like Dudley, which should be used instead). In addition to reading from a server, Imperio supports writing to anything with a write() method, and as such, is used for logging as a delegate of Dolores.
  • Dudley. Dudley is not a delegate of Dolores. Rather, Dolores can be its “delegate” (or, in this case, “receiver”). Dudley is an HTTP server, and receives messages over HTTP and funnels them to a receivers (Thestral implementation), and Dolores is its default receiver.
  • Firenze. Firenze, like Dudley, is an HTTP server; but instead of being a receiving server, it is a long-polling push server. It is not usually Dolores’s delegate—you don’t want every client to get every message. Instead, Firenze merely registers itself with Dolores to get an ID, and lets others register it (for instance, a client could send Dudley a message, which, through Dolores, would be received by Pig to make the connection).

Above all, it is simple. The longest of these pieces is Firenze, at 280 lines of code (50 of which are intro comments I used to explain it to myself and spec it while planning).

That’s All!

I hope you find this helpful. I am excited about porting Roots to node.js; progress is somewhat slow, as I am quite busy (I am a student, and finals for all four classes tomorrow!)

What I am most looking forward to is how Pig (the dispatcher) could use the same code in both the client (Pomona) and the server—the routing is quite similar; in one case, you want to route to handlers/callbacks, in the other, to update functions on Thestral instances—pretty similar; even identical.

I am also looking forward to experimenting with two-way push, where all data is transferred through push—completely eradicating the expectation of a response to a request, and providing only a single receive/send code path on both client and server.

Anyway, I think this is all super cool. :)

Thanks to Endash for the nice title. :) Roots (called Dobby in development) is available on GitHub.

P.S. I like the name Dobby, but Roots is better P.R.

Posted in Code, Comet, Python, Script, SproutCore | 7 Comments »

Sprouting—Automated Spriting for SproutCore

Saturday, October 17th, 2009

Well. Spriting. Do you know what it is? While I’ve included a very brief summary, you may want to just skip that part if you already know about it, or find a better description of the technique. Also, while this article is specifically aimed at SproutCore, the concepts (and the script download) can easily be applied elsewhere.

Well, here are your options:

  1. Read the whole article. Could be amusing, if I’m having a bad writing day.
  2. Skip to the download and “how to use.” No, I’m not going to explain the code. Well, not here, at any rate. I’ll explain it in the code itself. Yes, of course there are comments.

What is Spriting?

Example Sprited Images

Example Sprited Images

Spriting is a technique which combines multiple small images into one large image. This can be very beneficial, because it means that, instead of downloading, say, 100 small images for all the commonly used icons in your web application, the web browser downloads one bigger image (or, perhaps, two or three, but you get the picture).

The goal of spriting is to reduce the effect of latency on your web app’s performance.

To your left is an example set of sprited images.

Using a Script

You may thing that combining several images into one larger image might be a bit time-consuming and, overall, a tad challenging, especially if you are often updating the individual images.

You are not alone. I feel the same way.

However, I do know a little about writing scripts—even if I’d like to know a lot more; I’m still only learning Python. So, there is a simple answer:

Write a darn script!

The Product

If you just want the script and how to use it, you’re in luck. Because there are, in fact, some comments in the script, I’ll leave the explanation of how the code works for there.

You can download the code from GitHub.

The script is written in Python. It requires: Either: Python 2.6+ or Python 2.5 with simplejson installed PIL (Python Imaging Library)

Very briefly, here is what it does:

  • Takes a “sets” folder, which should have sub-folder sets of icons to be sprited.
  • Takes an output folder, under which it will create new set folders containing CSS and image files.
  • Combines images under a maximum size (128×128 by default—see Configuration) into a sprites.png file in the output set folder.
  • Creates CSS referencing these images using a URL template (SproutCore oriented at first—see URL)

Sets

You often might have more than one set of icons. For instance, you could have a main set, and then an alternate set that you only want to load on demand. You still want that additional set to be sprited.

Or, perhaps, you may have images meant for repeating backgrounds, but that you still want to sprite.

To facilitate this, the spriting script takes, as its first argument, the location to a sets folder. You put your sets as subfolders into this folder. For instance, this could be your folder layout:

my-sets
	common
		delete-32.png
		delete-64.png
		refresh-32.png
		refresh-64.png
		refresh-512.png
	repeat-x
		config.js       — see Configuration
		toolbar.png
		footer.png
		header.png
	odd-feature
		some-icon.png
		other-icon.png

URLs

The CSS has to reference the images. Only problem: it doesn’t know where those images are.

Usually, you’d just be able to use paths relative to the StyleSheet. However, if you are using SproutCore, this won’t work because it moves around all the CSS and JavaScript.

With SproutCore, you need something more like:

static_url('images/set/sprites.png')

This is what the script does by default. Just like above. It uses a URL template of

static_url('images/{set}/{image}')

It is a string which will be passed through the Python format function, with set and image as arguments.

Configuration

There are a few things that are configurable. Configuration is done per-set.

Configuration files are JSON files, and are very simple. They are optional, and take at most four parameters:

  • max-size: Default: 128.
    The maximum size for sprited images. Images above this size will be put as separate images. Useful if you have some really high-res icons you don’t want to sprite, but want to keep with the set.

  • repeat: Default: “none”; can also be “x” or “y”.
    Determines the repeat mode. If repeat is “x,” the sprites will be laid out in rows; if “y,” the sprites will be laid out in columns. The generated CSS will include repeat-x or repeat-y, respectively.

Note that all images in a repeat-x set must be the same width, and all images in a repeat-y set must be the same height. The width or height of these images should be set using repeat-width and repeat height, which are discussed below.

  • repeat-width: Default: 32.
    If repeat=“x,” the width of the repeat images. If your repetition does not require a pattern, you could set this to 1.

  • repeat-height: See that lovely summary of repeat-width above? Well, this is the same, but for y/height.

The CSS

As you may know, SproutCore automatically includes any CSS files. So, since this generates CSS files (and, if you provide the right argument, puts them directly into a SproutCore-watched folder), the CSS is automatically added. So, how do you use it?

Very simply. The script generates CSS rules in this format:

.set-name .icon-name.icon, .set-name .icon-name.icon { /* CSS */ }

For example, to show “refresh-64″ icon in set “common”, you’d do something like this:

ImageView.design({
  layout: { left: 100, top: 100, width: 64, height: 64 },
  value: "common refresh-64 icon" // using SproutCore's built-in className support for spriting
}

Or, as you may have noticed from the rule above, you can accomplish some basic theming very easily by treating set names as themes:

View.design({
  layout: {left: 100, top:100, width:256, height: 256},
  classNames: ["common"], // the theme
  childViews: ["styledView"],
  styledView: ImageView.design({
    layout: { left: 100, top: 100, width: 64, height: 64 },
    value: "refresh-64 icon" // using SproutCore's built-in className support for spriting
  })
})

Nice and simple, eh?

Tags: , , , , ,
Posted in Code, Python, Script, SproutCore, Web | 2 Comments »

Running Orbited (or other Twisted Services) as a Windows Service

Tuesday, August 4th, 2009

It took me awhile to figure out how to run Orbited and Dolores as Windows Services. Unfortunately, while I love Mac, almost every other computer at my workplace is Windows, including the servers. When deploying my web application, I can’t really just leave instructions like: “When you restart the server, please run these two scripts.”

They need to be Windows Services.

Thankfully, it isn’t actually too hard. You just need a couple of Python packages (which I actually don’t remember installing, so they may have come with Twisted or Python), and then all you have to do is write a couple of very simple scripts. Here are mine:

Orbited Service

import OrbitedManager # a copy of orbited.start with some very minor modifications, which I'll include after the break
import time
 
# Service Utilities
import win32serviceutil
import win32service
import win32event
 
class WindowsService(win32serviceutil.ServiceFramework):
	_svc_name_ = "Orbited"
	_svc_display_name_ = "Orbited Server"
 
	def __init__(self, args):
		win32serviceutil.ServiceFramework.__init__(self, args)
		self.hWaitStop = win32event.CreateEvent(None, 0, 0, None)
 
	def SvcStop(self):
		self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
		win32event.SetEvent(self.hWaitStop)
 
	def SvcDoRun(self):
		import servicemanager
		OrbitedManager.start()
		while True:
			retval = win32event.WaitForSingleObject(self.hWaitStop, 10)
			if not retval == win32event.WAIT_TIMEOUT:
				OrbitedManager.stop()
				break
				time.sleep(5.0)
 
 
if __name__=='__main__':
	win32serviceutil.HandleCommandLine(WindowsService)

Dolores Service

import Dolores
import time
 
# Service Utilities
import win32serviceutil
import win32service
import win32event
 
class WindowsService(win32serviceutil.ServiceFramework):
	_svc_name_ = "Dolores"
	_svc_display_name_ = "Dolores Server"
 
	def __init__(self, args):
		win32serviceutil.ServiceFramework.__init__(self, args)
		self.hWaitStop = win32event.CreateEvent(None, 0, 0, None)
 
	def SvcStop(self):
		self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
		win32event.SetEvent(self.hWaitStop)
 
	def SvcDoRun(self):
		import servicemanager
		Dolores.start()
		while True:
			retval = win32event.WaitForSingleObject(self.hWaitStop, 10)
			if not retval == win32event.WAIT_TIMEOUT:
				Dolores.stop()
				break
				time.sleep(5.0)
 
 
if __name__=='__main__':
	win32serviceutil.HandleCommandLine(WindowsService)

That’s the basics, but as you may have noticed, I also changed Orbited’s start script. I did this because the original did not play nicely with threads.

Instead of changing it in its original location in the filesystem, I created a copy of it in the local directory (as you can see, I’m not importing Orbited.OrbitedManager).

(more…)

Posted in Code, Comet, Python, Script, Web | No Comments »

Python, Windows, and Network

Thursday, April 16th, 2009

We have several machines. All of them (except for mine, a MacBook Pro which I use for almost everything) run Windows.

I’ve been learning Python. It is a really useful language. It runs on the command line, and that is really important. If I could, I’d use JavaScript instead, but some things — such as connecting to MySQL servers — are a bit much for me to write JavaScript shells for at the moment (especially because it has been a year or three since I’ve dealt with the C++ MySQL APIs).

So, for a recent diversion which required MySQL database access, I decided to use Python. If I can remember, I’ll write about the project later. It was a simple script to keep track of sequence numbers for the new Intelligent Mail Barcode, so that we could ensure each piece had a unique number in any 45-day period.

Windows does not come with Python pre-installed. This is not usually a problem, as Python is not at all difficult to install. But in addition to having to install Python, I’d have to install the Python package for MySQL. This Python script could be used from several computers, so I’d have to do in on each computer. Even if I posted an article on our intra-office wiki (which it still seems like nobody reads), I would probably have to end up helping everyone install it.

And, what if I make a mistake? Having so many installations (at least 3 or 4, possibly more, and prone to change whenever we replace or add a computer) could make me very busy with things I don’t really want to be busy with.

What would be best is if I could install Python on a central server. Well, I couldn’t do that, but I could install Python on my Windows computer (which, alas, I still have to use sometimes), and figure out what I needed to copy to the server.

It is really quite simple in principle: just install Python on Windows. Copy C:\Python25 (or whatever your Python install directory is) to your central server.

Except… it still won’t run on the other computers.

It complains about missing a Python25.dll. It would seem that the Python installer was stupid enough to not install its DLLs in the install folder along with Python itself, but instead litter the already-overpopulated C:\Windows\System32 directory with them.

So, I found all of the DLLs that looked like they belonged to Python 2.5, and copied them into the Python25 directory on the central server:

  • python25.dll
  • pythoncom25.dll
  • pywintypes25.dll

Could I have missed some? Perhaps. But everything appeared to work.

I still had one complication: it would be nice if scripts could run without the command needing to say r:\path\to\Python25\python. For this, each computer would need to have Python’s path in their PATH system variable.

Instead of adding yet another item to PATH, however, I decided to make a simple command file that would “initialize” Python by adding it to the instance’s PATH. This would only last the duration of the Command window.

Here is the script, changed only slightly:

@echo off
 
IF "%INITIALIZED_PYTHON_25%"=="YES" GOTO END
 
SET PYTHONPATH=R:\path\to\python\python25\
SET PATH=R:\path\to\python\python25\;%PATH%
SET INITIALIZED_PYTHON_25=YES
 
:END

And that’s it. As the script I was working on (and likely any others in future) would be run from batch files anyway, they’d just need an extra line at the beginning to call this command. And that’s it.

Tags:
Posted in Python, Script, USPS | No Comments »