Considering how much I enjoyed Old Man’s War, you’d figure I’d be a sucker for John Scalzi’s The Human Division.

What does “not strictly a novel” mean here? Well, The Human Division is probably best described as an “episodic narrative” — it’s a collection of individual episodes that each tell a complete story, arranged chronologically, so that if you read them all in sequence you get a larger narrative arc. The closest analogy would be a season of a television show, and indeed The Human Division is arranged into thirteen “episodes,” including a double-length “pilot episode,” entitled “The ‘B’ Team.”

However, I’ve only just now started reading/buying the “episodes”. The question is, can I catch up with the release schedule? Through two and half chapters, so far, so good, although I find it a touch heavy on the expository dialogue, enough to be uncomfortably noticeable. Also, the dialogue in Walk The Plank, was a little too clean for me. If you’re traumatized from a crash landing, injured from an animal bite, and infected with an explosive microbe, I don’t expect you to string together sentences that well.

But they’re well paced with Scalzi’s good old sense of humor, and intriguing from a sci-fi perspective. So I will forge on.

Vega is a new browser-side visualization toolkit from Jeff Heer. Why?

Vega provides a higher-level visualization specification language on top of D3. By design, D3 will maintain an “expressivity advantage” and in many cases will be better suited for novel, highly interactive graphics. On the other hand, we hope that Vega will be convenient for a wide range of common yet customizable visualizations. Vega’s design builds on concepts we developed in both Protovis and D3, and is informed by multiple years of research at Stanford and our experiences building data-driven applications at Trifacta.

Still waiting for prefuse to rise from the grave and reclaim its rightful place in the visualization firmament.

Pandas contributor Adam Klein has released a kindred in spirit toolkit, Saddle, for the Scala language:

Saddle is a high-performance data manipulation library for Scala.

Saddle provides array-backed, indexed, one- and two-dimensional data structures; vectorized numerical calculations; automatic data alignment; and robustness to missing values.

Scala, Pandas, Spark. Are you thinking what I’m thinking?

I’ve been holding out on new traditional home computing gear (e.g. laptop or desktop) since I really haven’t had a good need, despite my lust for the finer things. I’ve even gone so far as to jack-up the hard drives in Ye Olde MacBook to good effect. Such good effect that it could easily be my primary personal laptop for at least another year and a half. I mean really, if MS Office 2011 runs decently on this old kit, what more can you ask for?

Except for that pesky desire to run virtual machines. I picked up a copy of Parallels Desktop in a recent MacUpdate promotion. Then I wanted to start kicking around some Hadoop virtual machines from vendors like Cloudera and Hortonworks. Guess what? You really need a free and clear 4GB of RAM to get started with these vms. My poor old machine only has 4GB total.

So Ye Olde MacBook is fine for the standard Ubuntu or CentOS distro, but the heavy duty data sciency type stuff is probably beyond it’s meager capabilities.

Something to chew on.

Another insightful gem from Rafe Colburn on taking a quick “hack” and getting the result into a production system:

Why write this up? It’s to point out that for most projects, getting something to work is just a small, small part of building a production service. Exporting a CSV file from a database query and uploading it to an FTP server takes just a few minutes. Converting that into a service that runs within the standard infrastructure, and handles failure conditions smoothly takes hours.

Echoes of the challenge of building capabilities.

Also, I’ve learned through hard won experience, don’t trust the environment cron provides your script at all. Make everything as explicit as possible. Even then it’s a difficult debugging process. Good luck!

Good personal development bit by Marc Hedlund on leaving behind career tools that have been overtaken by events:

Advancement is not a collection of skills. Advancement is an awesome ability to adapt to a new situation. Experience is a toolbox to help you do that – but any new job, any new role, any environmental change should make you question whether the tool you know and rely on is still the right tool for the job. Think of your skills as disposable, and actively work on unlearning the ones that were right once but aren’t right now.

This also seems like a good feature, maybe even interview question, to seek in candidates to join your team. “Talk about a skill you really relied on in one position but had to give up or adapt as you moved into a new position. Why?”

Sometimes going back to the basics is the best path. I just grabbed the python-mode.el 6.1.1 release from the python-mode.el launchpad site, which is maintained by some folks who use Python on a frequent basis. Fired up the mode and the current implementation seems much more consistent than some of the current new kids on the block, including Emacs 24.3’s built-in Python mode.

Even better 6.1.1 has support for virtualenvs.

Link parkin’: emacsredux, tagline: “Return to the Essence of Text Editing”. Subscribed.

Also of note, emacsrocks.

Keeping an eye on Titan, the Hadoop friendly graph data repository, they’ve version bumped to 0.3, with some interesting new features:

Titan 0.3.0 has been released and is ready for download. This release provides a complete performance-driven redesign of many core components. Furthermore, the primary outward facing feature is advanced indexing. The new indexing features are itemized below:

• Geo: Search for elements using shape primitives within a 2D plane.
• Full-text: Search elements for matching string and text properties.
• Numeric range: Search for elements with numeric property values using intervals.
• Edge: Edges can be indexed as well as vertices.

Related discussion over at Hacker News.

Via @gnat

Neat extension hack of iPython by Catherine Devlin:

RDBMS access via IPython

Introduces a %sql (or %%sql) magic.

Connect to a database, using SQLAlchemy connect strings, then issue SQL commands within IPython or IPython Notebook.

Granted there’s a lot of magic going on, but this speaks to another great feature of iPython, the ability to build up and embed domain specific languages within a nice interactive environment. Knock on effects easily follow, such as rapidly stuffing SQL query results into pandas for data analytics.

Link parkin’: The Flask Mega-Tutorial

This is the first article in a series where I will be documenting my experience writing web applications in Python using the Flask microframework.

Here is an index of all the articles in the series that have been published to date:

And there’s been 16 to date, which is a heck of a lot better than many of the grass roots tutorial series I’ve seen over the years. Each article looks quite substantive as well.

Interesting post-PyCon take on the state of matplotlib and Python and visualization from one Jake Vanderplas, a core matplotlib developer:

And where does that leave matplotlib? I would not, by any means, discount it just yet. Still, as John Hunter noted last summer, it faces some significant challenges, particularly in the area of client-rendered, dynamic visualizations. Any core matplotlib developers reading this should go back and re-watch John’s SciPy keynote: it was his last public outline of his vision for the project he started and led over the course of a decade. An IPython notebook-compatible client-side matplotlib viewer along the lines of the ideas John mentioned at the end of his talk would be the killer app that would, in all likelihood, allow matlotlib to maintain its position as the de facto standard visualization package for the Scientific Python community.

And all that being said, regardless of what the future brings, you can be assured that in the meantime I and many others will still be doing all our daily work and research using matplotlib. Despite its weaknesses and the challenges it faces, matplotlib is a powerful tool, and I don’t anticipate it withering away any time soon.

Of special interest to me was the callout to NodeBox OpenGL, harkening back to my old generative art in Python noodling days. NodeBox 1 wasn’t a bad analog to processing, but it looks like NodeBox OpenGL might be a major advance beyond pyprocessing, which seems to have stalled out as a project.

Still the stickler was the bit level manipulations that processing did well and I haven’t seen done quite right in any of the Python based parallels.

Very convenient. I’ve been on the lookout for an approach to integrating geospatial information processing into Hadoop. Looks like ESRI is open sourcing some GIS Tools for Hadoop:

Data that includes location, and that is enhanced with geographic information in a structured form, is often referred to as Spatial Data. Doing Analysis on Spatial data requires an understanding of geometry and operations that can be preformed on it. Enabling Hadoop to include spatial data and spatial analysis is the goal of this Esri Open Source effort.

GIS Tools for Hadoop is an open source toolkit intended for Big Spatial Data Analytics. The toolkit provides different libraries:

The libraries:

• Esri Geometry API for Java
• Spatial Framework for Hadoop
• Geoprocessing Tools for Hadoop

Via NoSQL Weekly

Link parkin’: Lily looks like an interesting mashup of various NoSQL projects:

Lily is a data management platform combining planet-sized data storage, indexing and search with on-line, real-time usage tracking, audience analytics and content recommendations. It’s a one-stop-platform for any organization confronted with Big Data challenges that seeks rapid implementation, rock-solid performance at scale, and efficiency at management.

Lily unifies Apache HBase, Hadoop and Solr into a comprehensively integrated, interactive data platform with easy-to-use access APIs, a high-level data model and schema language, flexible, real-time indexing and the expressive search power of Apache Solr. Best of all, Lily is open source - allowing anyone to explore and learn what Lily can do.

Speaking of Cloudera, I’m a dumbass for not subscribing to the Cloudera blog. Their posts tend to be quite meaty and well written. Over and above the Cloudera ML announcement I’ve enjoyed:

• One User’s Impala Experience at Data Hacking Day
• How-to: Create a CDH Cluster on Amazon EC2 via Cloudera Manager
• How-to: Use the Apache HBase REST Interface, Part 1
• Analyzing Twitter Data with Apache Hadoop

Problem fixed!

Bonus: TIL about the DEBS 2013 Challenge. Mmmmmmm, data.

[embed]https://twitter.com/josh_wills/status/315126912412438528[/embed]

Cloudera ML, at least making Apache Mahout a bit more usable

Today, I’m pleased to introduce Cloudera ML, an Apache licensed collection of Java libraries and command line tools to aid data scientists in performing common data preparation and model evaluation tasks. Cloudera ML is intended to be an educational resource and reference implementation for new data scientists that want to understand the most effective techniques for building robust and scalable machine learning models on top of Hadoop.

Good on ’ya Mr. Wills.

Sad news from Soda Hall:

Robert Wilensky, professor emeritus of computer science at the University of California, Berkeley, and one of the campus’s first faculty members in artificial intelligence when the field was just taking off, has died at age 61.

…

During Wilensky’s tenure at UC Berkeley, he served as chair of the Computer Science Division, director of the Berkeley Cognitive Science Program, director of the Berkeley Artificial Intelligence Research Project, and board member of the International Computer Science Institute.

Even though I wasn’t an AI guy, Prof. Wilensky was chair for a significant part of my graduate career. While I didn’t have a plethora of interactions with him, I always found them enjoyable. He was great for the Computer Science Division, presiding over the department during The Great Migration (TM) from Evans Hall, or shortly thereafter (need to confirm). Whatever. I just remember him being one of the earliest occupants of that office on the first floor, and suavely striding out, being one of the few faculty to pull off a suit on a daily basis. Not necessarily business, but definitely a notch up from elbow patches.

Godspeed, kind sir.

[embed]https://twitter.com/DidTheWizWin/status/316362659542990848[/embed]

I’ve always said truly great players often seem to have explosive nights, early in their careers, that really demonstrate their potential. Think Jordan lighting up the Celtics for 63 and juking Larry Bird in a first round playoff loss in Jordan’s rookie season. Not 100% true, took a while for Steve Nash to get going, but I bet there’s a high correlation.

John Wall’s not quite there yet, but 47 points, with 8 assists, in carrying a decimated Wizards to victory over a good Grizzlies team is a solid sign. And that so quickly after torching the Lakers at Staples.

There’s hope here in DC.

Ben Lorica posts some observations from the recent PyData conference:

It’s getting easier to use the Python data stack:

There are tools that facilitate the dissemination and sharing of code and programming environments. IPython2 notebooks allow Python code and markup in the same document. Notebooks are used to record and share complex workflows and are used heavily for (conference) tutorials. As the data stack grows, one of the major pain points is getting all the packages to work properly together (version compatibility is a common issue). In particular setting up environments were all the pieces work together can be a pain. There are now a few solutions that address this issue: Anaconda and cloud-based Wakari from Continuum Analytics, and cloud computing platform PiCloud.

One of the things I find most exciting is the quality incorporation of Python into the Hadoop ecosystem and varied spinoffs. While Java will always be on the top of the depth chart, since it’s the implementation language and Java is so enterprisey, Python is a reasonably first class citizen. I’ve gone a decent ways building Hadoop processing pipelines with essentially no Java programming.

Like me are you looking to become a more sophisticated Markdown user? You might want to checkout MacSparky’s Markdown Fieldguide. Only 10 bucks!

Markdown started as a clever way to write for the web but has become so much more. This book demystifies Markdown, making it easy for anybody to learn. This book includes 130 pages and 27 screencasts totaling more than one and a half hours of video. There is also an additional hour of audio interviews. This book will take you from zero knowledge of Markdown to being a Markdown pro and change the way you write for the better.

I’ll be checking out the PDF version since I’m a Kindle Store sucke… err, fan.

Update: Decided to go with the iBookstore version just for a little variety.

Via Daring Fireball

[embed]https://twitter.com/wesmckinn/status/314928646068531201[/embed]

I’m with you Wes. Python mode in Emacs 24.3 doesn’t feel very polished. May have to fall back to your solution as well.

[embed]https://twitter.com/wesmckinn/status/314932693752242177[/embed]

[embed]https://twitter.com/DidTheWizWin/status/315326026907533312[/embed]

C’mon @DidTheWizWin! You can do better than that. The Wiz came from double digits down, at the Staples Center, against a fully loaded Lakers with Pau and Kobe. Plus the Wizards went without Bradley Beal, Emeka Okafor, and AJ Price. Only nine guys got in the box score.

And John Wall showed something with 24 points and 16 assists in 44 minutes under the bright lights.

Weird box score observation. The Wiz totaled only 236 minutes, but by my math every regulation game should have 240. Guessing due to round down of fractional minutes.

Been looking around for a flexibly engineered batch scheduling tool that’s Hadoop friendly. Spotify open sourced its Luigi framework which looks like it might fit the bill:

Luigi is a Python package that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, handling failures, command line integration, and much more.

The purpose of Luigi is to address all the plumbing typically associated with long-running batch processes. You want to chain many tasks, automate them, and failures will happen. These tasks can be anything, but typically long running things like Hadoop jobs, dumping data to/from databases, running machine learning algorithms, or anything else.

There’s also an attendant post on other uses of Python at Spotify.

I enjoyed reading Ned Batchelder’s presentation on idiomatic iteration in Python, even though I knew of everything he presented. Made me feel like I actually knew something about Python! It’s good baseline knowledge any Pythonista should comprehend and internalize.

He also ingeniously made it available in multiple formats.

This is a presentation I gave at PyCon 2013. You can read the slides and text on this page, or open the actual presentation in your browser (use right and left arrows to advance the slides), or watch the video:

[embed]http://youtu.be/EnSu9hHGq5o[/embed]

Thank goodness I’ve retired from filling out NCAA Mens Basketball Tournament brackets. Georgetown, New Mexico, and Wisconsin would have given me headaches, and if the Hoyas don’t come back, whatever bracket I had would be busted.

Now I can just kick back, root for Cal, and keep my blood pressure low.

Link parkin’: rvm, it’s like virtualenv for Ruby.

RVM is a command-line tool which allows you to easily install, manage, and work with multiple ruby environments from interpreters to sets of gems.

I was amused by Brent Simmons’ republishing of Brian Reischl’s Stupid Feed Tricks

Brian is intimately acquainted with the different ways feeds can be screwed up. So he posted Stupid Feed Tricks on Google Docs.

I quote the entire thing below for people like me who don’t have Google accounts. The below is all by Brian:

… Putting a tiny number of posts in the feed (sometimes just one). These types then usually publish 10 articles in the space of two minutes, and wonder why you’re missing 9 of them. …

Now ponder scaling these issues up to (tens?, hundreds?) millions of feeds and users and you can see the challenge facing anyone providing a Google scale feed synch ecosystem.

Also, don’t miss Brent’s favorite way to screw up a feed. Really, hotel connectivity needs to die a horrible death at the hands of 4G and LTE. Conference provisioning is “only” way overpriced. Lodger oriented is overpriced and busted.

Ahhh the wonders of the Web. Thankfully, it’s the least worst, global, open, networked, hypermedia system we have.

With the GReader Apocalypse upon us, I was waiting to hear what the vendors of my two key feedreading tools were going to do. Got an answer now:

First, Black Pixel and The Return of NetNewsWire:

First, we intend to bring sync to future versions of NetNewsWire. It’s too soon to go into details about this, but you should know that we recognize how extremely important it is and that it is a top priority for us.

Second, even though we’ve been quiet about it, we have been working on new versions of NetNewsWire for Mac, iPhone, and iPad. We have some great new features and a modern design that we can’t wait to show you.

Next up, Oliver Fürniß, publisher of Mr. Reader, and his thoughts:

In my opinion all of those Google Reader alternatives should support this API, instead of going the easy way and creating their own. This would be a big win for millions of Google Reader users. Those services will have native clients within a short time-frame across many different platforms and devices. It’s not a big deal for the client developers to adjust the API endpoint. I’ve already changed my code (not uploaded yet) so that the API endpoint can be edited like suggested by Marco (Instapaper developer). Yes, I already hear your feedback that some alternatives provide many additional and awesome features. But do we really need features like ‘folders in folders in folders’ like those implemented in, for example, Tiny Tiny RSS?

Basically, uncertainty. I want to believe Black Pixel, but it’s been so long since a NetNewsWire update, I can’t have complete confidence. But I’ll keep on using it until it’s completely busted.

I really do think it would be great if Yahoo! jumped in and provided GReader compatible infrastructure. Microsoft could probably pull it off as well and maybe such an ecosystem could help out Bing.

A little over three months until D-Day. I’ll be looking to see how it all shakes out.

Man, if Garret can’t hack it in the slowly diminishing, open news ecosystem, it’s a really bad sign:

So, to sum up: I soon may not be able to get the information I want in a form optimized for linkblogging. I can’t offer ‘added value’ because the most-worthy ‘valuables’ are hidden behind paywalls. Trying to overcome these two issues to maintain quality is already taking more time than I can spare. The few items I can succeed with to my satisfaction, are not enough to attract and keep an audience. Quality and volume were a hallmark of my blogging style. The internet is now actively working against me on those two points. I can’t go back to having long lists of links on my site — I just can’t. That was so time-consuming as to be idiotic. It was only the excitement of the early metacosm that kept me going in that fashion.

I’ll also toss in the pernicious creep of embed culture, a kissing cousin of animated GIFs, as another factor, even though I’m a judicious offender myself. Cedes a lot of control back to the content provider and breaks “fixity”.

Link parkin’: vbox, “Yet another Python library of Python bindings for Virtual Box CLI (Command Line Interface).” [embed]https://twitter.com/pypi/status/313271372153491456[/embed]

If I ever get super dissatisfied with vagrant, and it does have a wart here or there, I may have to take another look at vbox.

Currently kicking the tires on Anaconda, Continuum.io’s Python distribution focused on high performance compute processing:

Completely free enterprise-ready Python distribution for large-scale data processing, predictive analytics, and scientific computing

Looks really nice and well put together although it doesn’t seem to play well with virtualenv and has its own environment model and tool, conda:

… Our users need to work with different versions of Python, NumPy, SciPy, and a variety of other packages. Moreover, they must be able to easily share live, runnable versions of their work, including all supporting packages, to their colleagues or the general public.

We created the conda package and environment management system to solve these problems. It allows users to install multiple versions of binary packages (and any required libraries) appropriate for their platform and easily switch between them, as well as easily download updates from an upstream repository. …

Bit of shame as virtualenv is really a core part of the Python ecosystem and personally makes my life a lot easier. But I can understand the tradeoff.

Anyway, Continuum provides the baseline Anaconda, which feels pretty competitive with the Enthought Python Distribution, for free and with a reasonable (YMMV), proprietary friendly license.

Via Catherine Devlin, and her PyCon lightning talk, I just learned of HTSQL

HTSQL is a comprehensive navigational query language for relational databases.

HTSQL is designed for data analysts and other accidental programmers who have complex business inquiries to solve and need a productive tool to write and share database queries.

Maybe Yahoo! could pick up the ball that Google Reader will be dropping:

But Yahoo now has a huge opportunity to make itself relevant again to the types of consumers who wrote the company off long ago. Now that Yahoo is referring to itself as a tech company rather than a media firm, it can bolster its techie cred with a product that provides utility that Google won’t match.

I say it’s not a bad thought because Yahoo! probably still has enough engineering talent and infrastructure to execute on this at scale. Mashable (really Todd Wasserman) misses an even bigger opportunity in that a small team could clone the unofficial GReader API and provide an alternative feed synching ecosystem. That would really be hot for mobile.

So at this point in my multi-vm explorations, it’s pretty clear I’ll need some configuration automation beyond provisioning. Sometimes you just need to shut down and restart services across a bunch of machines in a certain order. SSHing into each one and doing it by hand is now sufficiently painful to consider how the pros do it.

Capistrano has the advantage of getting me deeper into Ruby. Fabric, based on Python, means an easier learning curve. Choices, choices.

Today I Learned about dnsmasq

Dnsmasq is a lightweight, easy to configure DNS forwarder and DHCP server. It is designed to provide DNS and, optionally, DHCP, to a small network. It can serve the names of local machines which are not in the global DNS. The DHCP server integrates with the DNS server and allows machines with DHCP-allocated addresses to appear in the DNS with names configured either in each host or in a central configuration file. Dnsmasq supports static and dynamic DHCP leases and BOOTP/TFTP/PXE for network booting of diskless machines.

Dnsmasq is targeted at home networks using NAT and connected to the internet via a modem, cable-modem or ADSL connection but would be a good choice for any smallish network (up to 1000 clients is known to work) where low resource use and ease of configuration are important.

So far dnsmasq has come in real handy for multivm, host-only, networks of virtual machines. It seems to cleanly handle reverse DNS lookups and is nicely packaged for most distros.

Chuck Shotton gets to the heart of why Google Reader shutting down will be painful:

Here is the incredibly powerful thing that Google Reader provides that will leave a huge, gaping hole in my daily RSS reading:

Synchronization.

Google Reader was at best an average RSS reader. But it excelled at keeping all of my other 3rd party RSS reader apps in sync. By providing a set of APIs that allowed remote readers to mark/unmark individual articles as read, it let me start reading news on my phone with Feeddler, continue on my desktop with Google Reader, and switch to Flipboard on my tablet later in the day without having to wade through the same news articles twice. What was marked as read on my phone never showed up as unread on my tablet. It also gave me centralized management of all my RSS feeds. When I nuked an entire feed on my desktop computer, it disappeared from my mobile devices.

I wouldn’t hold my breath Chuck on someone filling the void. Matching the scale, speed, availability, and cross-client API support of GReader will be tough. Then again scalable data processing techniques and infrastructure have advanced far faster than the RSS reading population has. So maybe a small team can take this on as a side project, or even lifestyle business hustle.

[embed]https://twitter.com/hmason/status/311983572993060864[/embed]

Google Reader, dead July 1, 2013. Victim of another Google Spring Cleaning.

We launched Google Reader in 2005 in an effort to make it easy for people to discover and keep tabs on their favorite websites. While the product has a loyal following, over the years usage has declined. So, on July 1, 2013, we will retire Google Reader. Users and developers interested in RSS alternatives can export their data, including their subscriptions, with Google Takeout over the course of the next four months.

As a feedaholic from the earliest days, this’ll be the end of an era. But maybe it’s the extinction event that will spur a new round of innovation with RSS. I’d do a Munch like Scream, but everyone knew this was coming.

I would like to heartily thank the GReader team for a boatload of utility over the years.

[embed]https://twitter.com/jrauser/status/306636696362037249[/embed]

What he said. ISO 8601 everywhere please.

As a tabaholic, maybe the OneTab plug-in can help with my addiction. It might also get me back on Chrome more regularly although really my main issue is the inconsistent behavior of the 1Password extension.

Whenever you find yourself with too many tabs, click the OneTab icon to convert all of your tabs into a list. When you need to access the tabs again, you can either restore them individually or all at once.

When your tabs are in the OneTab list, you will save up to 95% of memory because you will have reduced the number of tabs open in Google Chrome.

Looks like Emacs 24.3 is an official release now. Mickey at Mastering Emacs has an overview of some of the changes:

A new version of python.el, which provides several new features, including: per-buffer shells, better indentation, Python 3 support, and improved shell-interaction compatible with iPython (and virtually any other text based shell).

I blogged about a new python mode a long time ago and it seems it’s made it into trunk. That’s probably good news for most Python users, but as I haven’t yet explored the new python.el mode yet, I will cover this in much greater detail in another post.

Better iPython integration would be a boon although I’ll admit the Emacs IPython Notebook is pretty saucy. Definitely, a must test drive.