Even week! Must be a win right! Wrong.

I go into the Sunday night game with over 85 points, a 7 point lead, and 2 players left: Miles Austin and Jason Witten. Going against the Eagles no less. Sure, my opponent had LeSean McCoy who I knew could bust out, but 2 prime players against 1 should be advantage Rangers.

Sadly, I would only go on to add 9 more points (jeez I hate the Cowboys) while McCoy hit for 33. The Plexus Rangers lost by 17 points. And just to rub the salt in my wounds, I didn’t start the Rams’ Steven Jackson. 33 points sitting on my bench.

We’ve had one go round through the league plus one. Every game I’ve had 100 points scored against me. Makes for a 3 and 5 record, 7th place in an 8 team league, and in bad playoff position. At least I’m only 2 games out in the loss column.

No apologies for going back to back on Python posts. I’m slowly easing back into exercising and spent a 1/2 hour on the stationary bike today. Great opportunity to put the Kindle to use, and I did so knocking out about half of Niall O’Higgins’ Mongo DB & Python. Very productive use of my time.

At 50 printed pages, it’s a slim tome, but O’Higgins so far has done two things really well. First, in about two or three pages he succinctly summarizes the differences between MongoDB and typical RDBMS systems. Frankly I found 10Gen’s documentation to be pretty poor in this regard. Second, some time is spent on the MongoDB operators that can atomically mutate embedded documents. This is somewhat the equivalent of Redis’ atomic data structures, although not quite as clean in my opinion. Take that with a boulder of salt as I’ve used neither MongoDB nor Redis for even minor experiments much less in production. The key point is that I learned MongoDB is somewhat closer to Redis than I thought.

At $19.99 MongoDB & Python seems a bit pricey for the page count, but I bought the e-book version, which is cheaper, and got a half off deal. So I only wound up paying $8 which is quite is reasonable.

Apropos of nothing. My Kindle is showing some cracks in the casing. It’s less than a year old! Then again, My Little Guy(™) has gotten a hold of it a few times.

Link parkin’: Mitch Garnaat, the author of the boto toolkit, has finished an O’Reilly book Python & AWS Cookbook. Given my tweet collection hacking, looks like it might be useful with its focus on EC2 and S3.

I recently bought a bunch of O’Reilly e-books on Python and MongoDB. What’s one more on the stack?

Garnaat’s blog is probably also a good catch and worth a subscription. For example, he’s got a tutorial on multi-part S3 uploads, which I was having trouble with previously. So I wound up using Cyberduck. Now I can see how to do it right.

There’s no sense in being precise when you don’t even know what you’re talking about. — John Von Neumann

I liked Ben Horowitz’s post on Lead Bullets. But I really enjoyed the above quote that was attached.

Finally treated myself to some new music purchases. Just a quick listing along with some initial reactions:

• DJ Marky, FabricLive.55. An interesting mix of DnB and other stuff from a Brazilian prodigy. Probably worth repeated listening
• Noisia, FabricLive.40. On first listen, not danceable enough for me. Every time the beat gets going, the mix turns all bleep, bloopy. Probably not due for heavy rotation.
• Marcus Intalex, FabricLive.35. DnB with a somber bent. Likely to get more listens.
• Mastercuts, Classic Salsoul, Volume 1. An oldie, but expensive, goodie that I just sprung for used. Worth it just for First Choice’s Let No Man Put Asunder.
• Pete Tong and Felix Da Housecat, All Gone Ibiza, 11. Only listened to the Felix Da Housecat disc, which had a solid, old time Chicago House feel.
• DJ Heather, Tangerine. On deck. Can’t wait to hear one of my favorite dj’s earliest commercial releases.

I bought half of these as used CDs from the Amazon Marketplace since there was no digital media purchase option. Worked out pretty well. Despite having to pay standard shipping, since these aren’t covered by my Amazon Prime, they were all still pretty cheap. And it all arrived within 5 business days of my order.

Okay, my kvetching about Chrome causing machine slowdowns might have been slightly misguided. Disabling Flash has clearly improved things, but I’m still noticing bouts of poor machine behavior. In some way, shape, or form the 1Password Chrome extension is part of the problem. Often times when I go to it to fill in a password it’ll take minutes (yes minutes) to pop up the selection dialog. And the Chrome Task Manager shows the 1Password extension process taking up a couple hundred MBs of memory.

Note that other folks are having similar issues. At least AgileBits is aware of the issue. But if this keeps up I’ll have to push all of my 1Password usage to Firefox. Not horrible, but a bit of a pain.

On the tweet collection front, the file in which I’m stashing my real-time notifications from Twitter has reached over 2 million lines. The requested notification format is length delimited, a line with the tweet length followed by the tweet text. So by my best estimation, I probably have over 1 million tweets harvested.

I actually need to do some validation processing, but I think I’m pretty close to declaring victory on this front. Once again, I’m impressed at how little handholding was necessary. My only worry is that it looks like my script restarted its curl process a couple of more times (yeah), but I know I didn’t add any code to cleanly mark those restart points (boo). So there may be some lurking data corruption.

I’ll let it run overnight just to provide a little safety margin, then get to some data crunching. A 2.5 Gb file is a heaping plate. 100 10K tweet files sounds like a job for Elastic Map Reduce!

I took the time to finally pick up Matthew Pearson’s Generative Art. Since, I’ve got the honest-to-gosh physical book, I’m trying actually read all of it end to end. Have to admit, the long preface and first chapter were a bit of a slog. But then I didn’t really need any convincing that generative art is art and is worth exploring. Nice color plates though. Now I’m working my way through chapter 2 even though it’s terminally remedial for me.

Pearson’s AbandonedArt.org was an interesting find though, given my 100Hours project that I eventually abandoned. Pearson had a little more focus and determination, completing his goal of 100 Processing sketches.

He also had the foresight to put most of his code under a Creative Commons license. I had a notion that taking each piece, shoving it into a GitHub repository, and tweaking from there would be a worthwhile effort. A combination of working on generative art skills and DVCS skills. Plus you could see various artistic paths preserved for others to fork.

Sounds like a plan.

Link parkin’: StrongSteam, an app store of Artificial Intelligence APIs. Buzzword compliant but interesting concept of making AI a service.

My first taste of writing code was Fortran or BASIC, can’t remember which led, but I’d seen both before graduating high school. Either way I can’t say I’d really done any programming.

I got a brutal, 6 week introduction to C (not Mark Randolph’s fault, just a tough intro language (inadvertent DMR reference)) in a summer prep program before matriculating at MIT. Still, not really a programmer.

Then I had the great fortune of having 6.001 with Jerry Sussman. The whole course was taught in Scheme, a highly elegant, beautifully designed, descendant of John McCarthy’s original Lisp. For whatever the reason, I took to the language like a fish and I’ve been programming, sometimes in dribs and drabs, ever since. I got Lisp well enough to do a UROP under Steve Ward, hacking on Lisp machines. My One True Editor, Emacs, is inseparable from Lisp. The key concept of my dissertation was stuffing Lisp, as an extension language, in various platforms like a Web browser.

Paul Graham has one of the better descriptions of the essence of Lisp, but it’s safe to say that John McCarthy had a pretty profound, indirect influence on my life. Sad to see the end of McCarthy’s life arrive.

Oh and Lisp was just a sidebar for all of the AI work he really wanted to do.

Godspeed, kind sir.

No need to wait until tomorrow to report on this. Both sides have no players in tonight’s tilt.

Odd week? Must be a loss for the Plexus Rangers. 79 fantasy points for. 146 against. And I thought Man U had it bad this weekend.

I picked up Earnest Graham, who continued the injury streak by going down with a season ending knee/ankle injury after 16 yards gained. At least I dumped Santana Moss for Graham. Moss is now out 4-5 weeks with a broken hand. And here I was seriously considering picking up DeMarco Murray and his 251 yard performance.

Miles Austin chipped in a lousy 2.5 points. Rob Bironas scored a single lonely extra point. Larry Fitzgerald failed to make it to the end zone. At least Aaron Rodgers hit for his regular 25+ points.

My opponent must be pissed he wasted such a great outing on such a feeble opponent. He played both Drew Brees for 40 fantasy points, and Arian Foster for 43 more. That’s right. Two of his players outscored my whole entire lineup.

Onward to next week.

I’m glad the Barclay’s Premier League returned last weekend from a round of international friendlies and qualifying. Since the live matches shown here in the states are on Saturday and Sunday morning, gives me something to watch instead of the US football pre-game yakfests.

And jeez did Manchester City deliver an interesting display this morning, opening a can of whoop-ass on Manchester United (yeah that Manchester United) to the tune of a 6-1 beating. This was administered on the Man U home pitch of Old Trafford. By the end it sounded like the Red Devils were just mailing it in. Now Man City is five points clear of the pack.

Then of course, my horse in the race, Chelsea FC, spit the bit, losing to Queens Park Rangers. The Blues could have jumped over Man U to take second in the tables. I feel like Chelsea has a solid team, with a lot of quality, but not quite the character to take the trophy.

Still, there’s a long way to go. By my calculation the league hasn’t even reached the quarter pole in terms of games played, and there’s 7 months left in the campaign.

Being in Loudoun County Virginia, Leesburg nestles in one of the wealthiest parts of the country. But I wouldn’t exactly call it cosmopolitan. Not the boonies, but definitely exurb.

Which makes the notion of building a relationship with a good homey coffeeshop a bit tricky. One’s lucky to find a “cafe” a notch above Starbucks, and then it’s usually placed in a scenic mall setting with a stunning view of a parking lot.

When I first got to Leesburg, there was a franchise of the regional Greenberry’s chain that exactly fit that description. Not Nefeli Caffe, but it did the trick. In fact, the still standing Greenberry’s in Maclean actually has a good cafe feel. Then the Leesburg one changed ownership a couple of times, ditched the Greenberry’s affiliation (name change to Dolce Coffee House), and just developed an overall incompatible vibe with my tastes. Couldn’t put my finger on it, but some different commute patterns over the summer, and general indifference led to a steep dropoff in visits. (Unfortunately this led to a lot of Starbucks visits, but that’s for another post.)

Checked back in to Dolce twice this weekend and things feel a bit different. I think they might have changed ownership again, which may or may not be a good thing for the baked products. But the coffee’s still good. The same youthful staff is there on all of my recent visits. They’ve got a little of that urban flair. They may eventually get around to fixing the floorboards, but at this point it’s sort of a cute affectation.

But best of all, they’ll give you a hot drink in a ceramic mug. If I had to put my finger on the singular difference between Starbucks, along with its clones, and real cafes is that one. Mugs are for folks who are going to sit for a leisurely bit, relax and enjoy the atmosphere, and maybe commiserate with like minded customers. Paper cups are for people who are leaving soon.

Bravo Dolce!

Memo to self, check back in with processing. Looks like there are some big upcoming changes. Time to give up on the “processing in Python” dream and learn to love the best possible tool. I pulled the trigger on Matt Pearson’s Generative Art which should kickstart the process.

So Gowalla and Instagram have real-time streaming apis. Not quite as easy as Twitter since those services use PubSubHubbub to stream their updates. It also means running a notification receiver that accepts incoming HTTP requests on my Linode. So for my next trick, I’d like to see if I can’t get a million updates out of each of those services.

Passed the 500K mark in tweets collected. Halfway home. I’m not so much amazed that I’ve gotten this far, but that there haven’t been more bumps. Basically, I’ve had a single script running for well over a week. That script has only needed to launch two child processes and only recover once from a child failure. The same script is a paltry 51 lines of code including empty and comment lines. Meanwhile the data file is now well over 1GB.

That’s a testament to 1) Python’s concision, 2) Linux/UNIX’s simple clean design, 3) Twitter’s clean streaming API and 4) curl‘s robustness. I’ve just glued a bunch of really solid parts together to good effect.

Now that I’m on the downhill, I’m starting to think of datastores to hold, index, and query the data with. MongoDB seems like it might be the best fit of the NoSQL camp since the Twitter streaming API pumps data out in JSON. MongoDB also seems to have a nice query language, to have indexing for time along with geography, and to be production ready at scale.

Let’s see, even numbered week. Oh that must mean VICTORY! It was big scoring, for a bye week, but not a big victory. Then again, I was essentially a man down, what with Minnesota as my DEF delivering a -1 on the scorebook. Yes, I would have been better off with an empty spot.

Still won with a comfortable 9 point spread. I’ll give myself a GM pat on the back for picking up the Giant’s Ahmad Bradshaw. Loving his three TDs. Aaron Rodgers met expectations while Buffalo’s Fred Jackson doubled up his projection. After that it was a bunch of mediocre performances.

My opponent must be wondering how he lost. He had the second highest score in the league this week. And the opposing team spotted him a point, and then effectively played 7 guys versus his 8. We were both expecting a track meet between Dallas and New England, but he was hurt worse when a defensive struggle broke out. On my sidle Miles Austin and Jason Witten were solid, but on the other side Tom Brady and Wes Welker were significantly down. But enough of him, I’m enjoying a win.

Holding down the playoffs in fourth position. Now can I put together back to back victories?

Obmention: William Gibson’s Setup. Not all that interesting to be honest.

Attended the Big Data DC meetup this evening. The meeting was surprisingly conveniently located for me and I enjoyed both speakers, even though I only had surface knowledge of the night’s topic. The presentations were focused on the Cassandra NoSQL data store. We got an overview of the fresh off the press Cassandra 1.0 release, and some insights into the CQL query language for Cassandra.

Didn’t mind the dive into infrastructure, but I’m more of a straight-up data and analytics guy. Looking forward to more Big Data DC meetups though.

Actually, I’ve got over 335K tweets and counting captured on this collection. There were two keys to making progress over my last effort. First, I moved my collector to a personal Linode virtual private server (VPS). As opposed to my ancient home Linux workstation, the VPS can stay up and on the network for days and weeks at a time. No outages due to brief power shutdowns or arbitrary Verizon glitches. Second, my script actually recovered from a network hiccup that killed its child curl process. Picked up the right error code, forked off a new curl, and kept on trucking.

The back of the envelope rate of collection is about 60K+ tweets per 24 hours. At this rate, it’ll take me about another week and a half to hit the million mark. I’m not holding my breath, but glad to see a significant advance over the last milestone.

P.S. After a few months, can thoroughly recommend Linode’s services.

Belay that MacBook Air lust. My poor little white MacBook seemed to hit a severe thrashing wall. (Load avg 23!!) A little inspection and it turns out a bunch of Chrome processes were taking up all the CPU time. I could have gone tab killing, but I had a hunch about another culprit.

A little less than a year ago, John Gruber dumped Flash. I’ve always sympathized with that approach. My laptop’s fan starting up seems to correlate with playing Flash content. Now I suspected Flash was the real villain in my MacBook slowdown. What to do?

Turns out you can configure Chrome to not autoload Flash content. Subsequently you click on demand for playback. Works wonders.

Still got the same number of tabs open in Chrome, but now my machine is qualitatively much more useful.

Hat tip to Chris Kasten

Fingers crossed, someday I’ll meet my target of collecting 1 million geotagged tweets. So then what? Well I’m not sure, but I’m guessing some geospatial based analysis might be fun.

Enter SimpleGeo. I don’t quite know what their business model is, but I like their products/APIs:

• Context: gives you contextual information about a place.
• Places: allows you to search for further Points of Interest (POIs) near a given location
• Storage: lets you store, index, and query geospatially tagged data

Plus SimpleGeo’s pricing seems pretty reasonable for hobbyist tinkerer like me. Not to mention they make the PolyMaps library, which seems mighty handy.

Intermittently over the past few years, I’ve been writing various command line apps in Python. Things like split, but with a little more complexity and in a higher level language.

I’ve often felt I wasn’t quite writing these in a Pythonic fashion. I’d stashed away Guido van Rossum’s BDFL blessed idiom for main, but never really put it to use.

Steve Lott has a much better and simpler style of writing Python main functions, that seems more easy to ingest and adopt. He even includes a nice example of how to tie option parsing into logging.

Now it would be great if someone could come up with a cookbook for using the argparse module.

Need to chop a 7 million line file into 1 million line chunks? There’s a UNIX command line utility for that: split.

Does what it says on the tin and does it quite well.

In the spirit of Dennis Ritchie. Godspeed kind sir.

Sigh. The beat goes on. Lose, win big, lose, win big, lose. And of course Julio Jones adds to the injury streak, going down with, you guessed it, a bad hammy.

About the best that can be said for this past week is that I had my optimal lineup in. Of course my bench was fully parked with Must Start level guys on bye. Makes filling out the lineup card easy.

Did have a little hope, holding a lead going into Monday Night Football. But when the opponent has Matthew Stafford and Calvin Johnson… it was over mercifully quick.

Still 9 games left, and only a game out of the last playoff spot. A-Listers back in the lineup in a big way. Every streak starts with one win.

Link parkin’: GeoNames

The GeoNames geographical database covers all countries and contains over eight million placenames that are available for download free of charge.

To: Mark Zuckerberg

From: C. Ross Jam

Re: “You Have Notifications Pending”

Zuck dude, all I know is your unsubscribe link better work. Because I skipped out on those activities on purpose. Remind me again and I may dump you wholesale.

That is all.

I’m in violent agreement with Jason Snell’s thoughts regarding Tim McCarver’s recent illness:

I wish Mr. McCarver a speedy recovery and a healthy, long life. I also wish him a lengthy and joyous retirement.

By which I mean, Tim, this broadcasting game. It’s too stressful. You have just had a “minor” heart procedure—but how minor can anything regarding your heart be?

…So Tim. We’re your pals. Quit this broadcasting racket. …

Might be the best thing I’ve seen on American McCarver yet.

Despite my little tantrum, turns out one round of my tweet harvesting experiment spanned the evening of Oct 1st to the afternoon of Oct 6th. Geofocused on New York City and San Francisco, there’s a good chance I caught a solid burst of interesting tweets.

Link parkin’: Square’s Cube project.

Cube is an open-source system for visualizing time series data, built on MongoDB, Node and D3. If you send Cube timestamped events (with optional structured data), you can easily build realtime visualizations of aggregate metrics for internal dashboards.

Specialization is for insects. —Robert A. Heinlein in Time Enough For Love.

I really enjoyed Ted Leung’s recap of the Surge 2011 conference. I don’t know squat about scalability matters, the purview of the DevOps community, but there was one quote from Leung’s recap that really struck me:

specialization is an industrial age notion and needs to be discounted in spaces where we operate at the boundary of the known versus unknown.

This was from Ben Fried, Google’s CIO, commenting on how his organization’s structure failed to solve a problem with a very complex system. Turns out no one could really understand the system from a global, end-to-end perspective, which was the key to solving the problem.

The quote really resonated, because on the day job I’ve been working on a team trying to strategically address a particularly thorny innovation challenge. Many of the technology components are known, and pairwise compositions well understood. But the unknown solution feels like it needs to emerge from a global, end-to-end perspective. If you agree with Fried, and I think I do, we’ll need some generalists to rise to the task.

And of course our company is chock full of stovepiped specialists.

Today, I declare myself The Most Unsentimental Man In The World ™. Think Different my friends.

Moment of silence for Rev. Fred Shuttlesworth though. Godspeed kind sir.

About a week and a half ago I got possessed with the luny notion to collect 1 million tweets. I’ve worked with the Twitter streaming api in the past, so thought it would simply be a matter of keeping the connection up for an extended period of time.

I started with the simplest possible thing that might work, directly from the Twitter streaming API documentation:

curl -d @locations https://stream.twitter.com/1/statuses/filter.json -uAnyTwitterUser:Password.

First couple of cracks didn’t quite work. As documented, the HTTP connection is subject to various disruptions, so I wrapped the curl invocation in a Python script that would re-execute as needed.

So now the script has been running for about 5 days now. I’ve got about 647 Mb of data totaling north of 250,000 tweets. I can’t tell if the script has needed to do a restart, but I’m impressed it’s lived this long.

And 250K tweets, with metadata, should make for some interesting analysis.

A maxim to live by, from your’s truly:

There’s no need to sit in a parked car, listening to sports talk radio. It’s just sports blather. They’ll make more.

Occasionally I have to catch myself on this one.

All I have to say is, I’m glad Aaron Rodgers is on my fantasy football team. With the scoring rules in my league, he dropped 51 fantasy points, far and away the top scorer in the league.

Which was pretty handy, seeing as how the opposing fantasy QB was Cam Newton, who wasn’t too shabby at 30 points himself. Plus my opponent also played Greg Jennings for 19 points, Ray Rice for 20, and Darren McFadden for 14.

Besides Rodgers, the key support on my team came from WRs Santana Moss for 12 points, Julio Jones for 18, Fred Jackson for 18, and Jason Witten for 19. Between Rodgers and Witten, I cruised to an easy 26 point lead, although I had to wait out a potential Ray Rice explosion.

The league only has 8 teams in it, so there’s plenty of talent to go around. The key is getting one or two really outstanding performances, which is a combination of matchups and luck.

At this point I’m 2-2, fighting for the 4th playoff spot. Still plenty of games to go, but I have to put a little streak together. I think I’ve got a viable core, but I’m always scuffling on a spot or two. At least this week, I don’t think anyone got injured.

The NFL is heading into its stretch of bye weeks, which puts the bench and the waiver wire at a premium. I think I’m configured for success during these challenging weeks, but we’ll see.

Update: Title modified for the correct team in 2011. Yeesh! I’m slipping.

Update 2: Oops! Forgot about Rashard Mendenhall pulling a hammy, and possibly missing next week. Then again, he was due for a benching anyway.

Ben Lorica runs the @bigdata Twitter stream, which routinely provides good big data related nuggets. I don’t actually follow the stream because he occasionally summarizes the best tweets on his blog

Speaking of PostgreSQL, it’s also worth mentioning PostGIS, which is a fabulous geospatial extension for PostgreSQL. Nelson Minar recommends the book PostGIS in Action to supplement the PostGIS documentation. As an example, PostGIS is improving in the area of doing nearest neighbor searches.

Alternatively, one can try SpatiaLite which embeds GIS capabilities in the self-contained SQLite RDBMS. However, I found SpatiaLite had a lot of moving parts and didn’t get a lot of mileage out of it.

Sideways observation: Pinboard doesn’t make individual item URLs obviously visible in its HTML. A little digging in an RSS feed reveals their existence. Turns out, on the person’s web page, the dates are the permalinks for the bookmarks. Useful for giving credit.

Been noodling around with PostgreSQL at work. Great relational database system. Glad I got a chance to get back to it. Gets even better with age.

For the longest time, I’d never really used stored procedures in an RDBMS because the languages to write the procedures were unappealing. However, PostgreSQL has embedded scripting languages, including Python, as procedural languages. Makes things a lot more fun.

One caveat, it was pretty tricky to convince the PostgreSQL build process on MacOS Lion to not use the system Python. An important point if you don’t have root privileges to install modules, which comes in handy. I’ll have to look up the mods and post them since it could save someone a little bit of time.

Well the Fantasy Football gods giveth, and the Fantasy Football gods taketh away. The Rangers got beat by a little under 48 points. I was saved from the worst beating award by someone who got shellacked for a 60 point loss.

This is the third week in a row I’ve had a player incur an injury that put them out the next week. I lucked out in that Steven Jackson and Miles Austin had decent and great games. Kenny Britt wasn’t so lucky, done for the season with no production.

I’m patting myself on the back for picking up Buffalo’s Fred Jackson off the waiver wire. Only he and Aaron Rodgers scored in double digits for me. Meanwhile the opposing team had five 15+ point scorers, including Jermichael Finley going for 30. Note that it’s pretty difficult in fantasy football to gain ground when your quarterback is throwing TDs to the other guy’s tight end.

Biggest disappointment was Rashard Mendenhall, who couldn’t do anything against the Colts. The Pittsburgh running game is giving me the willies.

Anyhew, back to the waiver wire. Gotta scrounge another receiver.

Link parkin’: DJ Patil, who build LinkedIn’s data team from the ground up, expounds on Building Data Science Teams