newsprint fray

more hip visualizations, part 0_1

pam — Fri, 05 Feb 2010 00:21:57 +0000

A while back, I started messing around with data visualization stuff, and came up with a chloropleth North American map that attempted to show the places the Tragically Hip played most often. I now have a slightly shinier and more granular map that shows cities, which is step 0.1 on the way to glory. Not that I have any idea what constitutes ‘glory’ in this instance, but I’m told that life is about the journey, not the destination.

Once again, I started with the [admittedly unreliable] show archive on the Hip’s site. (Actually, I started with the .csv I generated last time, but whatever. That’s the data source. It’s still only North American shows, but I think it’d be easy enough to extend to the rest of them.) To get the shiny googlemap with markers for places they’ve played, I came up with the following steps:

Pull a list of cities & states/provinces out of the csv.
Ask google for the lat/lng coordinates of those cities.
Turn said coordinates into xml tags.
Feed the xml into googlemaps and get a marked map.

This went way better than my experiments last time, but there’s a lot I still want to do with this map to make it more ~~useful~~ interesting. It’s still some lazy, sloppy code, but here it is anyway.

Step one: get a list of cities to look up.

queries = []

reader = csv.reader(open('concerts.csv'), delimiter=",")
for row in reader:
    if row[4].strip() in ['United States', 'Canada']:
        city = row[2] + "+" + row[3]
        queries.append(city)

locales = set(queries)

So that gives us a list of unique cities. (I may want to know that they’ve played Toronto 1938459 times or whatever, but I don’t need to look up Toronto’s coordinates more than once.) The way the geocode api works is that you pass in a URL with a bunch of parameters in the query string, and then it returns the coordinates in whatever format you’ve requested. So the next step is to set up all the junk to build those URLs.

scheme = 'http'
netloc = 'maps.google.com'
path = '/maps/geo'
params = ''
fragment = ''
query_dict =   {'output': 'csv',
                'sensor': 'false',
                'key': 'REDACTED',
                'q': '',}

all_query_strings = []
for city in locales:
    query_dict['q'] = city
    newqs = urllib.urlencode(query_dict)
    all_query_strings.append(newqs)

all_urls = []
for place_qs in all_query_strings:
    url = urlunparse((scheme, netloc, path, params, place_qs, fragment))
    all_urls.append(url)

Yeah, I probably should have used list comprehensions there, but I tend to write those when I’m refactoring stuff. The first pass at something tends to be step by rudimentary step. At any rate, all_urls is now a list of URLs to give to google, which is step two.

f = open('coords.csv', 'wb')

# using regular file handling instead of the csv module because i am lazy
# and google sends back strings.
h = httplib2.Http()
for url in all_urls:
    resp, content = h.request(url)
    f.write(content)
    f.write('\n')
    time.sleep(1)
f.close()

There’s a limit to how many requests you can send in a given time period, but I don’t know what that limit is and am not in any hurry. Also, I should only have to run this once, so I’m not uptight about that one-second sleep in there. At any rate, I now have a csv called coords.csv that looks like this:

200,4,28.3936186,-81.5386842
200,4,32.9911550,-117.2711481

Etc. The format is: response code, accuracy, latitude, longitude. Now I want to turn all of that into xml, which is just straight-up string interpolation.

all_xml = open('coords.xml', 'wb')
reader = csv.reader(open('coords.csv'), delimited=",")
for row in reader:
    latitude = row[2]
    longitude = row[3]
    xml_tag = "\n" % (latitude, longitude, )
    all_xml.write(xml_tag)
all_xml.close()

And that’s pretty much that. Add in the other xml stuff to make sure it’s a valid xml document, and then call the whole thing from an html file with some canned javascript.

Et voilá. An unuseful map of unlabeled markers, some of which are incorrect (like, why is there one in the Cayman Islands?). Having the coordinates, though, opens up a lot of possibilities for future awesomeness, which I’m sure someone else will come up with. My own current thoughts involve things like grouping the markers by year (or possibly by tour) and let people turn them on and off; and one marker per show would be better than one marker per city (although, in that case, do I stick with city markers and just say they’ve played Toronto a lot, or do I do the data massaging necessary to get coordinates for and show Lee’s Palace vs the Horseshoe vs the ACC?); and some way to make it collaborative and interactive would be completely amazing. Like, there would be one giant hipmap and if someone were to be logged into their google account, they could click on a marker for a show they’ve attended and add their info and their two cents. And who knows what I’m going to do about setlists. Something, something, something, someday, maybe.

there’s music out there laying in wait

pam — Tue, 22 Dec 2009 18:40:18 +0000

A few months ago, my external hard drive died. It had all my music on it, nearly ten years of accumulated mp3s from various sources: ripping my own collection, music from my friends, buying downloads from various sources, those glorious months in 2002 when eMusic gave you unlimited downloads, and almost certainly a few things that were torrented. It was a pretty devastating loss, really; much of it is easy enough to find again, but some of my favorite things were lost to the ether and I haven’t been able to get them back.

I tried to look on the bright side, though; organizing my itunes is always this horrible ordeal and it’s never finished and there were a few months that I didn’t really listen to music because the mere IDEA of opening my itunes gave me anxiety attacks. (You know the ones: HOW DO WE TAG THE TAGS?!) So declaring itunes bankruptcy seemed like an okay idea! I could pull in the album I wanted to listen to right then, make sure the metadata was okay, listen, move on. That had actually been going fairly well, even if I couldn’t necessarily put my hands on some rare b-side from back when there were actualfax b-sides.

…and then I started listening to Hip shows. I have maybe 170 right now, which is not a lot compared to the hardcore people who have been doing this for a long time, but seems ludicrous to people who don’t do it. And I was digging through my itunes the other night, looking for a particular show, and realized that I have once again got to the point where I don’t recognize a lot of the stuff in my itunes, and the rest of it is the Hip. I joked with a friend that my itunes has started spawning new music again, and he laughed at me. [We have a running joke that my itunes is some kind of musical font, because somehow it is always full of stuff I do not recognize and have never listened to, and sometimes I find it and am like, okay, what the hell is THIS and where did it even come from? I mean, I download stuff, yeah, but it's not like I have told my computer to just go download every mp3 on the internet, but sometimes I feel like that's exactly what it's done.] Then he related a conversation he had with some of his buddies about how most people have around 600 songs digitally available and then probably a handful of albums that haven’t been ripped in some manner. He has around 7,000, and his friends felt this was a LOT of music. I felt it was probably about average, and so I decided to ask the people I know and tally results.

I’m pretty much ready to declare the experiment over, because the average has now been holding steady for the last 20 people I’ve entered, even when their answer consists of a screenshot of a music folder, which is on its own 1-tb drive and contains 52,000+ songs and includes no boots or shows. Commercial releases only!

After that long-winded and probably boring explanation, here are the results:

total respondents: 56
total songs: 527,168
low answer: 0
high answer: 52,760
average song count: 9,413
median song count: 5,679
throwing out the freaks: 5,391 (this is the average of respondents who have fewer than 20,000 songs; it’s close to the median, which a math teacher informed me was the better number to use as an indicator anyway)
<10,000 songs: 43 people
10-20,000 songs: 5 people
>20,000 songs: 8 people

Responses collected by checking out all the shared itunes folders at work, asking twitter, asking LJ/DW, and asking on the hipbase (the Tragically Hip fan forum, which I expected to skew the results more than it actually did — I figured people there were likely to have a lot of music, and unlikely to have it all digitized; in the end, it was a wash).

The caveats here are, of course, huge. Some people answered with the size of their collection, and so I divided it by 6 megs to get a song count that is probably wrong. Many people estimated their answers. Many people told me that they have a lot of stuff that isn’t ripped; a few have literal rooms full of vinyl that has never been digitized. I got answers about ipod vs hard drive; work vs home; hard drive vs music server. In those cases, I used the highest number, because I figured that’s the one that more accurately represented the answer to my [extremely poorly worded] question about how much digital music is available to people for listening. Some people gave range estimates; I took the middle number in that case.

For a while it was looking as if most people either had fewer than 10,000 songs or more than 20,000 songs; there weren’t a lot of people in between. That settled out a bit, but it’s the reason for the breakdown in the number of people with n songs. Many of the people I would consider music geeks didn’t have too many songs digitized, but made a point to say, “this isn’t even close to my entire music collection.”

So! I have drawn the following conclusions:

I was right.
My collection is totally and completely reasonable.

Nothing like bad stats and terrible science to validate my position! \o/

you’ll be serving the song

pam — Wed, 02 Dec 2009 15:16:24 +0000

This isn’t a personal blog, so I don’t really say much about what I’m up to, but I found a few pictures from the Amsterdam concert I went to last week. And I am actually in one of them! I always appreciate photographic evidence that I’m not making up my entire life.

These photos on flickr, by Henk Ritskes, are all really good, but here are the two I actually care about.

This is early in the show, and nothing is going on right there, so it’s pretty calm. Just over the watermark, you can see my face right at Gord’s feet, my arms on the stage, looking up. I’m not kneeling; I’m on my toes. I look like I’m about eight years old. I didn’t feel like it at the time, but I look ridiculously tiny in that photograph. Note that I’m surrounded by guys who are all much, much bigger than I am.

This is the closing song (‘Blow at High Dough,’ for those of you who care about such things), a shot of the same area of the pit, and it’s pretty easy to imagine me curled against the stage, trying to protect my head but mostly just getting kicked around as the crowd presses in. It wasn’t a 90s-style NIN pit or anything (I was in those, too, and came out hurting), but it got pretty rough for me. But note that I am not complaining! I’m small and by myself and deserve what I get for standing there. I wouldn’t change a thing. Well, okay, maybe I would have had a little less beer dumped down my back.

There was a moment in this show, during ‘Locked in the Trunk of a Car,’ which starts off a little slow, a little quiet, and the room was mostly dark. And then the lights flashed on, bright glaring white shot through with smoke, and the drums kicked in, low and heavy and driving, and I looked to my left and the pit was this huge writhing mass, and people were hanging over the rail of the balconies, and it went up and back and on forever, alive. I thought, “yes,” and then I didn’t think anymore for a long time.

And that is pretty much what I have to say.

data visualizations and the tragically hip, part 0

pam — Mon, 23 Nov 2009 19:16:12 +0000

The very short background to this is that I love data visualizations, and I love the Tragically Hip. So when I saw this tutorial on making chloropleth maps with python, I thought, hey, I can totally do that. I’ve been wanting to mess around with data visualization stuff for a while now, and this seemed like a pretty good place to start. I decided I wanted to generate a map of North America showing me where the Hip has played most often. Seemed like a pretty simple and straightforward place to start.

First, I needed data, which I got from the Hip’s show archive. I just pasted it into a text editor and did some find-and-replace to make it a .csv. I really thought that would be the hard part. Ha!

Next, I needed a map. The tutorial uses a US county map, but that wouldn’t work for this. First, Canada is not actually in the United States! Also, they do not have counties. They have ridings, but reliable Canadian sources tell me they only care about ridings during elections. So anyway, this North American map is the one I ended up with.

If you read the first post I linked to, you will learn valuable information, such as the fact that the map has to be an .svg, which is a Scalable Vector Graphic, which is really XML. I hope you are now thinking, “oh god, will we have to parse XML?” Sadly, the answer is yes. Also, this particular map has shitty XML, which will make it all the more exciting.

But whatever! Just ignore that impending sense of dread and look at the map in a text editor. Each state/province/territory (henceforth SPT) has an ISO-3166 code assigned to it as an ID. So Illinois is US-IL, and Ontario is CA-ON. The csv I made lists city, SPT, and country, but not in handy ISO-3166 form, so I had to do something about that. I am not sure I really did the right thing, but here’s the code I wrote to do it.

# read in the concert csv
concerts = [r for r in csv.reader(open('concerts.csv', 'r'))]

for show in concerts:
    if show[4].strip() == "Canada":
        show[3] = "CA-" + "%s" % (show[3])
        # quebec's code is QC in iso-world
        if show[3] == "CA-PQ":
            show[3] = "CA-QC"
    elif show[4].strip() == "United States":
        show[3] = "US-" + "%s" % (show[3].upper())

This changes my csv from this:

11/07/2009,Landmark Theatre, Syracuse,NY,United States,TTH_SMT_20091107_1328

to this:

11/07/2009,Landmark Theatre, Syracuse,US-NY,United States,TTH_SMT_20091107_1328

I feel like there is a better way to do it, but that was the first solution that popped into my head, and it worked just fine. From there, I needed to figure out how often they played in a given SPT. I KNOW there is a better way to do this, but I stared at it for a few minutes before the laziness won out and I moved on.

    north_american_shows = {}
    for show in concerts:
        # if the state's not already there, add it. the first time through,
        # it will never be there, so everything will be set to 0.
        if show[3] not in north_american_shows.keys():
            north_american_shows[show[3]] = 0

        # look, i know this is stupid. i don't care. just count, okay.
        if show[3] in north_american_shows.keys():
            frequency = north_american_shows[show[3]]
            north_american_shows[show[3]] = frequency + 1

As you can see, I first made an empty dictionary called north_american_shows, and then I ripped through the csv and gave my dict keys corresponding to all the ISO codes. I set all their initial values to zero. Then I went through AGAIN, and just incremented the values by one for every show they played in a given SPT. Printing out north_american_shows at this point gives something like this:

{'': 11, 'BE': 1, 'US-NY': 83, 'US-PA': 33, 'US-TN': 3, ...}

I think I probably should have tried to make a dict that only used valid ISO codes as keys, instead of everything in that SPT column, and I probably should have just gone through the csv once, rather than twice. At any rate, this got me a usable dictionary that told me shocking things like mostly the Hip plays in Ontario. That piece of information, by the way, is also nicely conveyed by this:

That is a word cloud I made in wordle in approximately 30 seconds by pasting in the ‘city’ column from my csv. But let us not be deterred by the fundamental uselessness of our exercise! Probably I would never do anything at all if I let that stop me.

Anyway! We now have our data. A quick glance through it seemed to say I should break up the distribution like this:

There is only once place they’ve played more than 100 shows (Ontario), and only a few in the 76-100 range, so more granularity than that didn’t make much sense to me. Of course, I know exactly nothing about statistics, so I could be wrong.

Time to pick out some colors. Here, I mostly just did what Flowing Data told me to and used ColorBrewer; I needed eight colors. (Actually, I ended up needing nine because the last color was white, so you couldn’t see the SPT delineations for places they’ve never played.)

After that, the fun part. I expected this to be pretty easy; I’m familiar with Beautiful Soup, the parser used in the tutorial. All I needed to do was go through the XML, find a tag that carried one of my ISO codes as an ID, and then append a fill attribute to color the area based on my chosen color scheme/distribution. Unfortunately, Beautiful Soup did not deal at all well with the poor markup of the map; even just reading it in and then printing it out did all sorts of terrible things. I ended up having to use lxml, which I don’t really like (mostly because it’s hard to build, although that isn’t an issue on Snow Leopard, thank god). Here’s what I ended up with for this part of it:

    colors = ['#EFEDF5', '#DADAEB', '#BCBDDC', '#9E9AC8', '#807DBA', '#6A51A3',
             '#54278F', '#3F007D']

    tree = etree.parse('map.svg')
    tags = tree.find('{http://www.w3.org/2000/svg}g')

    for tag in tags.iterdescendants():
        for a in tag.attrib.values():
            if re.match(r'(CA|US)\-.{2}', a):
                if a in north_american_shows.keys():
                    if north_american_shows[a] > 100:
                        color_class = 7
                    elif north_american_shows[a] > 75:
                        color_class = 6
                    elif north_american_shows[a] > 50:
                        color_class = 5
                    elif north_american_shows[a] > 20:
                        color_class = 4
                    elif north_american_shows[a] > 10:
                        color_class = 3
                    elif north_american_shows[a] > 5:
                        color_class = 2
                    else:
                        color_class = 1
                else:
                    color_class = 0

                color = colors[color_class]
                tag.set('fill', color)

    print etree.tostring(tree)

For whatever reason, I had to pass in the link and then the tag I was looking for. I have no idea why; a coworker had to tell me to do that (thanks, Nat!). Once I had the root tag, I iterated through all its descendants, looking for IDs that looked like an ISO code. I initially looked for ones that matched something in north_american_shows, but then I realized that wouldn’t quite get it done; because there are places they have never played, those places are not in the dict. Hence the regex and the if/else; I needed to find those, too, and set them to the color I chose for 0.

Once that was done, I ran:

$ python colourize.py > hipmap.svg

And, amazingly enough, I have a map that shows me things I already knew.

The darker the purple, the more often they’ve played in that SPT. I’m pretty pleased with myself, even though this is the simplest possible visualization I could do and it took me a lot longer than I feel it should have. Seriously, I spent many days fighting with XML parsers, trying to get it figured out. But I’m definitely curious to know how the code could be better, because it’s obvious to me that it could be.

In the meantime, I think I’m going to try doing something with google maps, incorporating more granularity in terms of where they’ve played and on what tour. I’d also like to do something with setlists, but that data is a little harder to get into a useful format. It’s out there, for sure, thanks to fans more obsessive than I am, but it’s going to take me a while to figure out what I want to do and how I want to do it.

Before I do any of that, though, I’m going to go see some Hip concerts.

Oh, right. The full script, for completeness’ sake:

import csv
import re
from lxml import etree

def do_stuff():
    # okay. step one is to dump the csv into a list of lists.
    concerts = [r for r in csv.reader(open('concerts.csv', 'r'))]

    # so we have a list of lists. now we need the strings to be useful. while
    # you're in there, strip out any whitespace, because we hate whitespace
    # here in python land.
    for show in concerts:
        if show[4].strip() == "Canada":
            show[3] = "CA-" + "%s" % (show[3])
            # fix the PQ/QC thing in the data
            if show[3] == "CA-PQ":
                show[3] = "CA-QC"
        elif show[4].strip() == "United States":
            show[3] = "US-" + "%s" % (show[3].upper())

    # okay, now we need to find out how often they play in a given place. so,
    # uh, let's build a dictionary.
    north_american_shows = {}
    for show in concerts:
        # if the state's not already there, add it. the first time through,
        # it will never be there, so everything will be set to 0.
        if show[3] not in north_american_shows.keys():
            north_american_shows[show[3]] = 0

        # look, i know this is stupid. i don't care. just count, okay.
        if show[3] in north_american_shows.keys():
            frequency = north_american_shows[show[3]]
            north_american_shows[show[3]] = frequency + 1

    colors = ['#EFEDF5', '#DADAEB', '#BCBDDC', '#9E9AC8', '#807DBA', '#6A51A3',
              '#54278F', '#3F007D']

    tree = etree.parse('map.svg')
    tags = tree.find('{http://www.w3.org/2000/svg}g')

    for tag in tags.iterdescendants():
        for a in tag.attrib.values():
            if re.match(r'(CA|US)\-.{2}', a):
                if a in north_american_shows.keys():
                    if north_american_shows[a] > 100:
                        color_class = 7
                    elif north_american_shows[a] > 75:
                        color_class = 6
                    elif north_american_shows[a] > 50:
                        color_class = 5
                    elif north_american_shows[a] > 20:
                        color_class = 4
                    elif north_american_shows[a] > 10:
                        color_class = 3
                    elif north_american_shows[a] > 5:
                        color_class = 2
                    else:
                        color_class = 1
                else:
                    color_class = 0

                color = colors[color_class]
                tag.set('fill', color)

    print etree.tostring(tree)

if __name__ == "__main__":
    do_stuff()

job opening and recruitment efforts

pam — Thu, 17 Sep 2009 21:16:48 +0000

My company, Leapfrog Online, is looking to hire a Python web developer. There’s more about the company and the tech team here, and details of the job itself here. I think our tech team is pretty cool, and we try hard (with high-level support) not to be dicks about using open-source tools. Which is to say: we use them and we try to give back to the community by sending our engineers to conferences (as attendees and presenters), sponsoring said conferences (we’ve sponsored PyCon and Windy City Rails in the past), submitting patches to the tools we use, and releasing our code when possible. So I think, tech-wise, it is a pretty good place to work.

Among the software and test engineers, though, I’m the only woman. We’re also a pretty white bunch of people. I often hear things like, “well, I’d hire a woman, but none apply!” And I raise my eyebrows and think, “well, where are you looking?” Our recruiting is poor, and we all admit it; we have one HR person who does almost all of it, and she knows very little about technology. Some of us will post a link on our blogs, if we have them, or on twitter, and that’s about it. We tend to have a problem finding qualified devs in general, let alone qualified devs from under-represented groups. But today I asked my VP about it, and he is totally behind the idea, and asked me to come up with some ideas about where we might focus our recruiting efforts to attract more female and minority applicants.

Any ideas? Please feel free to let me know in comments, or email me (zerbie at gmail), or send this around to any groups or lists you know about.

[ETA: There are some ideas in the comments to this post on geekfeminism that I plan to follow up on.]

someone get me a command line

pam — Fri, 11 Sep 2009 21:22:05 +0000

Over on the Diversity in Python list (on which I am largely lurking, for reasons both various AND sundry), the question was raised: What brought you to Python? When you got to the fork in the road, what made the The Python Way more attractive than all those other ways? The people? The projects? The Django pony at the end of the rainbow? The assumption was that there was, somewhere, at some point, a decision that had to be made, and we pythonistas at least have that decision in common. After all, here we are.

Me, though, I never really made that decision. I went to journalism school (for what I believe is one of the only reasons people go to journalism school, which is to save the world). Yeah, while I was there, I built some websites and enjoyed it, but it was always a means to an end, always just another thing I could put on my resume for when I had to go out and get a real job. One of the almost-real jobs I had was at Playboy. I fell in with the programmer types on the pay site, and we stayed in touch, and when that internship was over, I went off to save the world.

Except that I didn’t have the stomach for world-saving. Working as a reporter in a courthouse on Chicago’s south side was grindingly, soul-crushingly depressing, and sometimes I still marvel that I came out of that alive. Illinois has the death penalty, and so I sat through murder trials that I still can’t talk about, and then sat through them all over again as people took the stand to explain why more death would make everything better. I realized that I could do it, if I wanted to; I could shut down the part of myself that threw up in the courthouse bathroom before calling the newsroom with my story. But I didn’t really like the person I would have had to become to do that, and so I walked away from hard-news journalism.

So then I thought, okay! Maybe science! Science can totally save the world! And anyway, mainstream reporting of the sciences often isn’t very good. So I did that for a while, working as a writer and editor for various scientific institutions and publications. It was fun and it was interesting and I loved going to work every day and learning new things and talking to amazing people and looking at experiments.

But in the end, I realized that I can’t write for a living. I can’t do it. It makes me insane, hollows out all the parts of myself I need to get through my day, leaves me empty and aching and scraped raw. Other people can do it — thank god — but I can’t. I can’t do deadlines, and I can’t do regular contributions to anything, and if I am to write, I have to do it on my own time and my own terms. I do, however, have to do it.

So, there I was, 20-something and lost, and those guys I’d fallen in with at Playboy (remember them?) had gone to a different company, and there was a part-time job opening for a contractor to come in and do straight black-box manual testing of websites. Okay, sure. I can totally click links, and I can blow my paycheck on ice cream and comic books, and everything will be super. Two things became clear quite quickly: One, they needed someone full-time, and two, clicking links is stupid. There must be a way to automate this. And there is! It’s called twill, and it’s written in python; what else do I need to know? Quick! Someone get me a command line.

And that gets me back to the beginning of this post: There wasn’t really a decision. I got a job, and I needed this tool to do my job, and here I am. It’s pretty nice here. I have the good fortune to work with many great pythonistas at my job, people to keep me interested and motivated and tell me when my ideas or code (or both) are full of shit. Many of them are active and involved with The Python Community.

When I put my roll-call for women in python, I really wasn’t sure what to expect. But the comments on that post, and conversations I’ve had about it have led me to believe that there are a lot more women using python than people think. But are we in python? Are we active and involved members of the community? Known and respected and liked (or, I suppose, loathed)? A few, yes, but not many. Why is that?

Me, I’m on the fringes. I could certainly become more involved if I were so inclined (and sometimes I am). But there are a few things keeping me where I am.

Communities are hard work. Participating is emotionally exhausting, and dev communities are largely based in mediums I don’t really enjoy all that much. I hate mailing lists. I swore off IRC sometime in the 90s. It drives me crazy to have to follow 270 different blogs and subscribe to RSS feeds for comments so I can follow conversations, or constantly refresh pages having discussions that I care about. Being in The Python Community (or, I assume, any dev community) is incredibly time-consuming, and I don’t have a lot of time. The Second Shift phenomenon is well-known, but for me, it’s more like my Fifth Shift.

And so, if I’m going to get involved, there need to be rewards. And there totally are. Support and encouragement for my projects. Answers to my questions. New friends. New ideas. New, better technologies. Cool software. Exciting travel opportunities and informative conferences. Possible job opportunities. The sheer joy of the thing itself, of a single, elegant line of code, of starting from nothing and building something cool, building something that works and that people can use and love and share.

I don’t know what other people get out of it. I assume others have similar reasons. (My straw poll on twitter largely confirmed this for me.)

But because I’m a woman, I get other stuff, too: Constant questions about where the other women are, sexist comments on this blog any time I post, demands on my patience and my time to answer questions about what I’m doing, how and why I’m doing it, what is this whole “feminism” thing all about anyway, what about the mens, and on and on and on. For me, wading into the python community ends up being only a little about python, and a lot about women — whether I want it that way or not.

Usually, I don’t. I announced at the last pycon that the first dude who asked me where were all the women was going to get punched in the face. I managed to largely avoid the conversation, but it was hard work, and it kept me pretty cut off. I stuck with the guys I already knew, and I didn’t go to any parties or meet many new people or do much of the social stuff that makes pycon so awesome. I still had a good time, but it was a very controlled good time.

To be an active part of any community, python included, you have to want it. You have to love it and sometimes you have to hate it, but mostly, you have to want whatever it is you get out of it. And if you’re part of an under-represented group and you want to play, you’d better also want to save the world, because that’s the assumption everyone is going to make. You’re going to be the token, the anomaly, the exception to the rule, and people are going to ask you about it, and heaven help you if you don’t want to discuss it. They’ll call you names under their breath (affectionately) and roll their eyes (fondly) and pat you on the head (sweetly) and ask why you’re so emotional (seriously).

It’s hard, is what I’m saying.

Is it worth it? I don’t know. I cannot and will not speak for other women. For me, sometimes, yes, it’s worth it. I’m not opposed to work, and I neither expect nor want everything to be easy. But sometimes it’s not worth it. Sometimes I tell all my mailing lists to stop sending me mail and I quit out of my RSS reader and I pause twitter until I can breathe again. Maybe I knit a hat or read a book or study French or write a story or go to a play or hang out with my friends. Sometimes I pull back for a day, sometimes for a month. There’s no shortage of activity or community in my life, just a shortage of sleep.

I keep losing track of the point of this post. What am I even trying to say? I’m not sure I know. Communities are difficult. Being actively involved with python is a lot of work, all those lists and feeds and conferences and blog posts to write and conversations to have. It’s not remotely surprising to me that there are so many women using python without engaging with the community, awesome as it is. Maybe they use it at work, like I do, and they can get support and encouragement and help from coworkers. They can read everyone else’s blog posts for fresh ideas and new technologies. They use python like they use Firefox, as just another tool to get the job done, whatever the job may be. Maybe they have all the friends they need. Maybe they’d rather spend their time writing software than answering questions about feminism (or any other *ism). Maybe they don’t want it badly enough. I can’t hold that against anyone; I’m often in that boat. I’m sure everyone is, about something.

So if my point is (and I am pretty sure this is my point!) that dev communities are difficult and time-consuming, and moreso for under-represented groups, what can we do about it? [Please note that I am completely uninterested in explaining why we should care, and I'm not going to engage with anyone who thinks under-represented groups should just stay under-represented.] And that’s where, I’m afraid, I’m out of words. The diversity list is kicking around some pretty good ideas about this, like Python Open Mike for people who don’t want to (or can’t) maintain their own blogs. So that’s awesome. Is the wiki useful enough? Should there be better newbie guides? Is there some kind of This Week In Python thing that people can subscribe to for an idea of what’s going on in the community without drowning in Planet Python? Would that even be useful? Are there community-building and maintaining technologies out there that are better than mailing lists and irc channels and far-flung blogs? What else can we do to make engaging less of a pain in the ass?

pre-post-mortem

pam — Sun, 16 Aug 2009 20:19:38 +0000

When I put up my last post asking for a show of hands from women in python, I wasn’t really sure what to expect. I’m not exactly well-known in python outside of some of the testing guys and the guys I work with, and I did a poor job of publicizing the post. In fact, all I did was link it on twitter and on my livejournal; other people took care of the rest for me. (Thanks, other people!)

It’s turned out rather well, though! I knew it was more than eight, and I know it’s more than have commented, but it’s really cool to see all these women showing up and talking a little bit about how they use python. It’s equally cool how diverse their experiences and projects are: everything from “sometimes I use it at home to get links into delicious” to “I use it as part of my high-energy physics research.” Comments continue to trickle in, and I smile every time.

… well, almost every time. There have been, unsurprisingly, a number of trollish comments that I’ve deleted (and mocked, of course). There was also the dude who came by to give me helpful advice on how to ask a question in order to get a more definitive answer. “Are you a woman in python,” he claimed, was too broad, too open to interpretation.

And that, I say, is part of the point: I want it to be open to interpretation. I think the ways in which we interpret questions can be telling. What right do I have to make a list of things you have to do to be “in python”? I wondered how many people would comment to say they use python, but they’re not a “real” programmer. Why do we say that? How common is it for a woman to say, “well, I write a few scripts in my spare time,” and have that be brushed off because, in some way, “it doesn’t count”? I don’t know, but I’m not going to do it. If you’re a woman, and you consider yourself to be “in python,” whatever that means to you, then that’s good enough for me.

More later. This has stirred up a lot of thinky thoughts for me.

[By the way, if you feel like asking, "why does this matter?" or "who cares about gender when we are so awesomely post-feminism?" (as a few people already have) then I will direct you to this site and delete your comment.]

Roll call: women in python

pam — Wed, 12 Aug 2009 21:52:45 +0000

Lately I’ve been wanting to talk more about women in python, which I see as a subset of the women in open source conversation that’s been taking place. I really wanted to start by talking to other women, though, to see who they are and what their experiences have been and how mine compare.

Except… who ARE the women in python? I can name eight, including myself. There must be more, right? There must be women using python who don’t participate in the larger community. There must be women who ARE active and whom I’m just not aware of. It can’t possibly just be eight. My Planet Python RSS feed shows me no regular female contributors (that can’t be right, can it?); three PSF members are female (out of 112) but none of them are officers or are on the board; nobody on python core is female.

My view is fairly limited in scope, so what I want to know is: are there really so few women using python in the first place? Or are there women using it who keep their heads down or don’t engage with the larger python community? It seems to me that those are separate issues, to be addressed in different ways.

So I am proposing a roll-call. Are you a woman using python? At work, at home, at school, in any capacity whatsoever? Raise your hand. Be counted.

(Men, please do not raise your hand on behalf of your female colleagues or friends. Send ‘em on over. Let them speak for themselves.)

adventures in app development

pam — Fri, 15 May 2009 17:31:23 +0000

There are many important pieces to The Puzzle Of Pam, and one of the biggest is my unreasonable and undying love for delicious. I think I have four — sorry, five — accounts there, plus access to quite a few more. I am often asked how I spent my weekend, and it’s not unusual for me to say something like, “re-tagged everything in delicious,” or something that might sound benign except for the fact that I have multiple thousands of bookmarks and only some of my management tasks have been scripted.

Anyway. One of the things about del that makes me a sad pamda is their bundle management interface. I know that many people don’t bundle at all (just yesterday I was asked, “what’s a bundle?”), but I like it. It keeps my tags organized and readable and useful, both to me and to other people. Depending on what you’re doing with the account, bundles can be used to impose some hierarchical structure on an otherwise flat setup. So I like bundling, and it’s important to me and to my likeminded friends. And I feel that bundling should be full of fun, but actually it is full of nails, sorrow, and repetitive stress injuries.

I was going to talk about WHY I hate the bundle management interface, and about the greasemonkey scripts and user styles and crazy tagging hacks that make it slightly less painful, but that is also not the point of this post. [This is one of the things that's difficult for me about this blog. How much background do I give? My usual MO is to just start waving my hands in the air and shouting, with the expectation everyone will be able to follow along. I'm not sure that's the case here, but... I'm also not sure I care! I'm awesome that way.]

The point of this post is that I have, after a few years of threatening to do so, started working on my Glorious Bundle Management App Of Magnificent Amazingness. It needs a name that does not consist entirely of adjectives, and probably it also needs unicorns and sparkles. The thing is, though, that I’m not an app developer. I am a test developer. And so it’s a strange and interesting learning experience to try to build something for other people to use. I’m not used to considering other people! I design for my own needs, and then the second I show it to someone else, it breaks. And my needs are… specific and strange, especially when it comes to delicious bundle management.

I’m still not sure what’s going to work and what isn’t, and the design has already changed a little bit as I’ve worked on it, but the basic idea is an interface that pulls in all your bundles, shows you the tags in those bundles, and allows you to add or remove tags via drag-and-drop. There will be a list down one side of the page that will show you your unbundled tags or your bundled ones; if it’s showing you all tags, you can hover over them and see which bundles those tags are currently in. None of this is built into the del management interface. You have to use a GM script to see what bundles a tag is in. You have to go back and forth between a lot of screens to see what tags are in which bundles. Etc etc. So I feel my app will transform bundling from something that is soulsucking to something that is soulenriching.

I’m not very far along, for a few reasons. One, I just started. Two, I’m writing it in PHP and jQuery, neither of which I know all that much about. I find it a little sadmaking that I would rather dig through the increasingly foggy recesses of my mind to try to remember where the fuck to put semicolons (EVERYWHERE) than try to figure out how to deploy a Python app. I just don’t want to mess with it. I feel like, even for someone who knows Python fairly well, and who deploys applications to production servers on a regular basis, the deployment (and, to some extent, development of) Python web apps sucks. [Yeah, yeah, appengine. It's STILL more difficult than it needs to be, IMO. I am lazy.] I just want to write code for a fairly simple web app, and I don’t want to have to do much else. And PHP makes that really fast and easy, even if the combination of PHP and javascript means I’m having nightmares about curly braces.

At my job, we’re big on iterative development, and I’ve watched it work for years, so that’s what I’ve been trying to do. I have a nice step-by-step list that does not include any items like, “5. ??????” or “17. Make it work.” I even have user stories. I’ve noticed, though, that it’s really easy to get distracted. Granted, I am an easily distracted person. But when it’s just me working on something that is largely for myself, when there’s no one to say, “um, Pam, probably you do not need to spend the next seven hours writing CSS for a login form, because the login form does not currently log anyone in to anything,” I tend to get a little lost.

And, like I said, some of that is me being me, and some of it is that it’s my project and I have to be my own PM, but I also feel like I’m falling into a rabbit hole I see other people going down. I think it’s a danger in user-focused javascripty apps in general, to spend a lot of time up front focusing on making it pretty “for the user,” when the user would probably like to have something to use. I like to think that users would rather have something that works than something that looks good sucking.

So anyway, I have to keep reminding myself of that, and reining myself in, and this process has been slow and a little painful, but also very illuminating. Updates as warranted.

inaugural post

pam — Tue, 12 May 2009 05:57:37 +0000

Hello! I have no real clue what I plan to do with this site, but I figured I should have one. It took me a while to get this wordpress theme to do what I wanted it to do (and, in fact, it still doesn’t, but whatever! I have moved on); I think it’ll do for now. I even think it looks pretty okay, although I haven’t looked at it on a PC. I will be singularly unsurprised if it breaks in IE.

And because I know you’re all dying to know what I’ve been up to, I spent a fair amount of time editing, writing and reorganizing the nose documentation for the .11 release, which came out last week. With that out of the way, I plan to get back to work on twill, which I have been avoiding. I don’t know when .9.2 will come out, but if I have my way (and haven’t forgotten how to code), it should be in the relatively near future.

Other than that, I’ve been doing my usual thing: reading too much Shakespeare, watching too much television, and eating too many Dunkin’ Donuts breakfast sandwiches.