Archive for the 'Python' Category

Optimizing for the App Engine Datastore

Monday, July 21st, 2008

I’m finding that BlobProperties are a very handy way to optimize for the App Engine Datastore. The first iteration of my island map had a Map model, and a Tile model. It used entity groups, so one Map entity was the parent of 100 Tile entities. Nice and clean, and of course dog slow. You don’t want to be loading 101 entities with every request if you can avoid it.

I added some quick-and-dirty benchmarking to time the datastore calls and then tried several different approaches. The one I settled on was BlobProperties and pickle. Basically, my Map model has a BlobProperty which stores all the tiles in pickled format. When I load the map out of the datastore, I use pickle.loads() and before I save the map back, I use pickle.dumps(). This means that I only have to load a single entity out of the datastore to display a map.

Nice side benefit: My core game logic is now pure python, so it is really easy to work on it outside of app engine (or even django). This is handy when writing benchmarking tools, testing, and prototyping new algorithms.

Possible drawback: I think I will hit compatibility problems if I try to load old pickled objects after renaming python modules. If this becomes a problem, I plan to start pickling an intermediate format like a dict instead of full objects.

Django on App Engine

Monday, June 30th, 2008

screenshot of island game

Some of the folks at Google have released a sweet Google App Engine Helper for Django. It consists of a skeleton Django project with all the little tweaks necessary to run on top of App Engine, and it makes it really easy to get started.

I used the helper to start a simple Django app that shows tiled maps, using the images I previously created. Making a map generator is way, way down on the list of things that might possibly make this game fun, so I just quickly made three 10×10 islands manually. Here’s the code which generates the island in the screenshot:

island1 = _Expand("""O O L L O O L L L O
O L C_road_s L L L L F L O
O L L_road_en L_road_ew L_road_ew F_road_esw C_road_w F O O
O L L L L F_road_ns F L L O
O O O L L L_road_ns L L L O
O L L L M L_road_ns L L L L
L L L L L L_road_ns M L L L
L L L M M L_road_ns L L L O
O L L L L L_road_en C_road_w O L O
O O O L L L L O O L""")

The app picks one of the three islands randomly and spits out an HTML table containing the image tiles. A little bit of CSS makes it fit together without borders/gaps:

table {
  border-collapse: collapse;
}

td, tr {
  line-height: 0;
  padding: 0;
}

.island {
  border: solid 1px black;
  float: left;
  margin: 1em;
}

If you want to see it live, the latest version is running here.

A Tiled Island Map

Sunday, June 22nd, 2008
island with coastline

Smooth coastline!

For the island game I’m creating using App Engine, I need a tiled map. I wasn’t expecting this to be difficult, but the sheer number of different tiles required is daunting. If I limit roads to only entering a tile from the north, south, east, or west, there are still 15 ways for roads to cross a tile. Since I don’t want blocky, square islands with abrupt land/ocean transitions, I need a set of 46 coastline tiles. Start multiplying by different types of base tile (forests, fields, mountains, tiny villages, big cities, etc.) and the situation quickly gets out of hand.

island without coastline

Blocky coastline

The only sane solution appears to be transparent overlays which are composited on demand. I don’t see a way to do server-side compositing with App Engine (plus I don’t think it would scale well), so that leaves client-side compositing. I did a simple mock-up with transparent PNG images, using absolute positioning to stack them on top of each other. It seems workable. Traditionally, transparent PNG images weren’t supported well by IE6, but since only 10% of the visitors to my site are using IE6, I might just forget about supporting it.

So that’s the long-term plan. In the spirit of getting a working prototype as quickly as possible, though, I’m going to start with a really simple tileset where I can pre-generate all the combinations and just serve static images. Here are the 6 tiles I’m starting with:

ocean tile

Ocean

land tile

Land

mountains tile

Mountains

city tile

City

field tile

Field

road tile

Road



I’m not going to allow roads over mountains for now, so there are 50 possible tiles (16 road combinations * (city, field, land) + ocean + mountains). Pre-generating all the combinations is a snap using PIL.

#!/usr/bin/env python2.5

import os
from PIL import Image

def composite(images):
  """Composite a stack of images, [0] on top, [-1] on bottom."""
  top = images[0]
  for lower in images[1:]:
    top = Image.composite(top, lower, top)
  return top

def combinations(list):
  """Generate all combinations of items in list."""
  if list:
    for c in combinations(list[:-1]):
      yield c
      yield c + [list[-1]]
  else:
    yield []

def load(name):
  return Image.open(os.path.join('sources', '%s.png' % name))

def save(name, image):
  image.save(os.path.join('img', '%s.png' % name))

def make_roads(name, above, below):
  """Save tiles with roads.  Images in above will be composited above
  the roads.  Images in below will be composited below the roads."""
  roads = dict(w=load('road_overlay'),
               s=load('road_overlay').transpose(Image.ROTATE_90),
               e=load('road_overlay').transpose(Image.ROTATE_180),
               n=load('road_overlay').transpose(Image.ROTATE_270))
  above = [load(filename) for filename in above]
  below = [load(filename) for filename in below]
  for directions in combinations(sorted(roads.keys())):
    out = composite(above + [roads[x] for x in directions] + below)
    if directions:
      save('%s_road_%s' % (name, ''.join(directions)), out)
    else:
      save(name, out)

def main():
  save('mountains', load('mountains'))
  save('ocean', load('ocean'))
  make_roads('land', [], ['land'])
  make_roads('city', ['city_overlay'], ['land'])
  make_roads('field', [], ['field_overlay', 'land'])

if __name__ == '__main__':
  main()

PIL also comes in handy to paste together the first proof-of-concept map using the new tiles:

#!/usr/bin/env python2.5

import os
import sys
from PIL import Image

TILE_SIZE = 50  # Width/height of a tile in pixels.

in_file, out_file = sys.argv[1:3]
data = [line.split() for line in open(in_file).readlines()]

map = Image.new('RGB', (TILE_SIZE * len(data[0]), TILE_SIZE * len(data)))
for y, row in enumerate(data):
  for x, name in enumerate(row):
    image = Image.open(os.path.join('img', '%s.png' % name))
    map.paste(image, (x * TILE_SIZE, y * TILE_SIZE))
map.save(out_file)

And here it is: The first example map!
Example Map

Next on the agenda: Getting Django running on App Engine and serving this example map using HTML.

Drive: a simple scrolling demo in pygame

Saturday, March 17th, 2007


A couple weekends ago I wanted to play around with some game ideas, to see if they were super-awesome or boring. I needed a simple framework to prototype them on, so I whipped one out using pygame. Then I sketched up some art. And made an installer.

And totally forgot to play around my original game ideas.

Damn. Maybe next time.

Anyway, here it is: a simple scrolling demo made with python and pygame. It has no real purpose (unless you want to do scrolling in pygame).

OSX
Windows
Source (Linux)

Delicious Python

Monday, October 30th, 2006

Organizing bookmarks is too hard. Does Backcountry Maps go under Maps or Backpacking? Does Hot Library Smut go under Libraries or Pornography? And how am I supposed to keep my bookmarks synched across 5 computers? I’m a busy guy, I don’t have time to figure this stuff out.

That’s why I’ve wanted to use del.icio.us for a long time. I was really excited when they announced their import tool. A few minutes later I was staring at 1000 poorly tagged bookmarks. Oops, should’ve cleaned them up before importing. After failing to find any way to batch process the mess, I dejectedly went back to my hierarchy.

I checked again today and there’s still no “Delete All” (the FAQ helpfully suggests canceling & remaking my account). Time to do it myself. I looked at the API docs (oh, I see, I’m not supposed to flood the server…) and grabbed pydelicious. Fired up ipython and had my bookmarks deleted in no time. Sharing all of my bookmarks turned out to be simple too: Since pydelicious doesn’t seem to understand private bookmarks [1] it shares everything. Changing my mind and deciding that I really should have tagged all my new bookmarks “import” (oops, again) was similarly easy. Hooray!

[1] It is unfortunate that the del.icio.us team doesn’t provide up-to-date libraries in a few languages. In a lot of ways I think that officially maintained libraries are an ideal answer to the SOAP vs. REST arguments

steal the mouse back from SDL

Monday, August 21st, 2006

I hate it when a poorly-behaved SDL app (usually mine) grabs the mouse, crashes, and doesn’t give the mouse back. Great, now I’m stuck in X with no mouse. Here’s a handy python script to get it back:

#!/usr/bin/env python
# get the mouse back after an SDL app crashes

import pygame
pygame.init()
pygame.mouse.set_visible(False)
pygame.event.set_grab(True)
pygame.quit()

XCode & Python Resources

Thursday, July 13th, 2006

Some resources for using XCode with Python. A few introductory sites plus the ones I find most useful on a day-to-day basis:

XCode & Python Hints

Thursday, July 13th, 2006

Here are a few hints about how I set up XCode for Python development.

Run your program from XCode

Some versions of PyObjC have a bug so when you make new projects, XCode doesn’t know how to run them. Make a new custom executable & point it build/Development/myProgram.app.

No, don’t quit!

Having the keyboard shortcut to quit XCode be the same shortcut as quitting your application (cmd-Q) is a recipe for annoyance when you accidently hit it twice. Instead, go into XCode’s preferences and change the key binding to quit XCode to something like cmd-option-Q.

Faster builds

You can shave a second or two off the build-test-fix cycle if you set up a dummy “No Build” target. The Development build target will run every time but behind the scenes it isn’t copying files, it is just making aliases (py2app’s –alias option). Set up a new shell script target which runs /usr/bin/true and you can save the effort.

You have no debugger, but at least use PDB

If you set the USE_PDB environment variable, PyObjC will dump you at a PDB prompt when there is an unhandled exception. You can set this on the “Arguments” tab of the custom executable.

XCode & Python

Thursday, July 13th, 2006

Introduction

XCode is the IDE that Apple ships with OS X. Although primarily targeted towards C++, Objective C, and Java it also plays well with Python. It is excellent for writing OS X applications in Python thanks to good integration with Apple’s Interface Builder and py2app (the OS X distutils packager).

If you want to write OSX applications using python, XCode is the tool you want. This is thanks to solid integration with Apple’s Cocoa application framework and Interface Builder.

Debugger

First, the bad news: no debugger. XCode has a debugger but it doesn’t work with Python. You’ll have to use another tool like PDB instead.

Environment

xcode.jpg

Except for the debugger, the rest of the environment is nice. The UI feels like a proper OSX app, with all the standard keyboard shortcuts and controls, so you’ll feel right at home.

The UI is very flexible, and can accommodate a single fullscreen window or multiple separate windows for each file.

Editor

editor.jpg

Standard features here. Syntax highlighting, split panes, menus to jump straight to a class or function. Content assist exists, but leaves something to be desired. I have yet to see a content assist for python that is able to keep up with the dynamic nature of Python classes, however.

Source Control

diff.jpg

XCode knows about Subversion, Perforce, and CVS. File comparisons are done using Apple’s File Merge tool so you get nice side-by-side views.

Finding Stuff

“Find in Project” lets you search your entire project. Bookmarks let you quickly jump to common sections of code

Making OSX Apps

XCode is a great environment for Cocoa development. PyObjC provides the Cocoa bindings for Python, and once it is installed XCode knows how to make a new Python project. The distutils setup.py script is handled for you and you can create your .app bundle by clicking Build

newproject.jpg

Drag-and-Drop your interface…

guides.jpg

Interface Builder lets you set up your widgets. Helpful guides pop up to help you position your widgets according to the Aqua guidelines.

…and then Drag-and-Connect your buttons

buttons.jpg

Hooking up buttons to actions in your Python objects is just a matter of ctrl-dragging from the button to the object.

Conclusion

If you want a Python IDE for your Mac, XCode is a good choice. If you also want to develop OSX applications in Python, XCode is a very good choice.

Super Happy Dev House

Tuesday, May 2nd, 2006

The new work

Attended my first Super Happy Dev House on Saturday night and had a great time. The house was packed with energetic people devoted to getting stuff done. Even though most of the people were strangers working on their own projects, it was really motivating to be surrounded by so much activity.

My project was getting boost.python working on my iBook. I wasn’t able to stay all night and didn’t reach my goal*, but I did learn about iPython from the people sitting next to me. Someone (not sure who) had designed a great poster for the event and they had printed up stickers & pins so there was even some schwag. I’m looking forward to the next one.

* My problem was I was trying to use the boost packages from fink. Turns out they delete a bunch of the files you need to use bjam. The answer? Download the source directly from boost.org (not sure why I didn’t think of that right away, actually). Mission accomplished.