A long time ago, I was an amateur comic book collector. In a tale soon to be told, I recently purchased an 11” iPad Pro. I don’t really want to get back into buying physical comic books, but I’m interested in revisiting some of the classics and reading some recent critically acclaimed works.

Enter Comixology Unlimited. Comixology itself is a venerable digital comics service that got hoovered up by Amazon a few years back. The basic idea is that there’s a Comixology reader available across multiple devices (Web, iOS, Kindles, etc.). You buy digital versions, to own, of graphic novels and single issues through the Comixology platform. The Unlimited part is straight from the Amazon playbook, a monthly subscription service for a lending library that provides access to a large(?) catalog of titles. The library is a subset of titles available on the entire platform and there’s a relatively generous limit on the number of works you can have “borrowed” at any given time.

read more ...

I managed to spark the reading passion once again. First, in January I just picked up the copy of William Gibson’s Agency that had been sitting unread on my bookshelf for months, and started reading. Second, I gave in and began reading books in the Amazon Kindle app on my iPhone. Don’t know what it is, but for some reason that used to feel tawdry. But the best book to read now is the one immediately at hand.

In any event, for recording purposes, here’s the year’s tally so far, in no particular order

• Agency, William Gibson
• All Tommorow’s Parties, William Gibson
• Soccernomics, Simon Kuper and Stefan Szymanski
• The Collapsing Empire, John Scalzi
• Essentialism, Greg McKeown
• Working Backwards, Colin Bryar and Bill Carr
• Stories of Your Life and Others, Ted Chiang

I don’t have a particular urge to start doing reviews, but all were enjoyable. Essentialism was a little fluffy, but I align with its sentiments.

In the meantime, there’s a nice stack of physical books in the backlog, not to mention a healthy lineup of ebooks ready for the Kindle app. We’ll see how long this holds and how many I knock out in 2021.

Last Wednesday marked the completion for me of a 54th circumnavigation of the sun aboard this big blue marble. Have to say, as an eve, May 25th 2021 was a heck of a lot better than May 25th 2020.

While I’ve been writing Python code for a while, and even some for my current job, I feel like I’ve sort of lost the pure coder mentality. Work projects don’t involve building fully maintained apps. There’s just lots of little piecemeal components with a lot of quick and dirty hackery. Plus, I’m not up to date on a lot of the newer Python 3 changes and best practices. Don’t get me wrong, I’m totally proficient n Python, but not really a modern coder.

So I’m link parkin’ a few resources to start working with, as a marker for future motivation:

• PyBites and it’s companion coding challenge platform
• Similarly LeetCode but with more focus on coding interview questions via Steven Heidel with guidance on application
• Reuven Lerner’s Python Workout book and Weekly Python Exercise course
• Ken Youens-Clark’s Tiny Python Projects book
• David Beazely online courses or his open source version of Practical Python

Quoting Reuven Lerner:

So what are these resources missing? What is it that’s going to take you from a “good enough” Python programmer to a fluent one that has full command of the language?

Practice. The very sort of practice I offer in Weekly Python Exercise.

I think I’m really out of practice.

I’ve been subscribing to RSS feeds, text and podcasts, forever. As expected various of them go dark on occasion. I still stay subscribed though, just in case.

A few years ago, a favorite podcast of mine was The Architecht Show, by Derrick Harris. At the time he was covering the emerging “cloud native” space and had a lot of great interviews. I suspect he could never find sustainable sponsorship and so had to go on hiatus to make ends meet with a serious day job.

But now he’s back!. Just popping up in my podcatcher (Overcast) as if he’d never been away.

The Architecht Show returns (!) with Replicated co-founder and CEO Grant Miller talking about how his company is enabling KOTS, or Kubernetes-Off-the-Shelf software. We dive into a number of topics around this, including the pitfalls of relying too heavily on SaaS, the popularity of Kubernetes itself, and how laws like Europe’s GDPR are forcing companies to rethink how and where they manage user data.

Welcome back Derrick!

Link parkin’: Mastering JQ: Part1

This is the first part of an ongoing series on mastering jq. This series does not assume prior experience with jq, but it does assume basic fluency in shell programming.

jq is a valuable tool that every fast coder has in their tool chest. It contains depths of immense power. In part 1, we’ll start off with the basics.

For each application of jq, we’ll lead off with an example that you can copy and paste into your shell to see how it works. The rest of the section discusses the application in more detail.

jq is an amazingly useful command line utility for processing JSON content. I’ve probably mentioned it on this blog, but don’t have search wired up to find the previous posts. In any event, it’s definitely a tool that’s worth getting better at using.

Via Code Faster 😄

For a long time, I didn’t really understand Linux’s Logical Volume Manager (LVM), same as Shawn Powers. Luckily he sat down, plowed through, and wrote up a nice tutorial on LVM.

LVM is an incredibly flexible, ridiculously useful and not terribly complicated to use system. It makes life easier. It makes future storage upgrades and migrations simple. Quite simply, I love it. So in this article, I cover the concepts and usage of LVM. By the time I’m done, hopefully you’ll love it as much as I do!

Now I get it! And really appreciate it.

Listened to this week (LTTW) had one outstanding entrant. Gene Farris, a Chicago House legend, put in an entry for 5Magazine’s #StayHomeDisco series.

Gene Farris needs no intro here - from the loft scene of the ‘90s Chicago House scene to today he’s remained one of the most enduring and in-demand Chicago DJs and producers of his generation. His latest release “So Dope” - the rare track whose title is also a critique - appears alongside tracks from Alan Nieves, TAYA, Serge Santiago & Nolan on Selector Vol 8 from Yousef’s Circus Recordings. Go check that out right now

Just a banging stone cold open that holds momentum while journeying into numerous House classics. Definitely R-rated material. Soundcloud embed below…

read more ...

Link parkin’ The Art of Command Line

Fluency on the command line is a skill often neglected or considered arcane, but it improves your flexibility and productivity as an engineer in both obvious and subtle ways. This is a selection of notes and tips on using the command-line that we’ve found useful when working on Linux. Some tips are elementary, and some are fairly specific, sophisticated, or obscure. This page is not long, but if you can use and recall all the items here, you know a lot.

Simon Willison noted that GitHub sneakily released a new feature that seems to have some interesting potential:

GitHub quietly released a new feature at some point in the past few days: profile READMEs. Create a repository with the same name as your GitHub account (in my case that’s github.com/simonw/simonw), add a README.md to it and GitHub will render the contents at the top of your personal profile page—for me that’s github.com/simonw

Don’t know that I’ll push the self-updating limits that Simon is, but I might use it to add some additonal static text to my profile page.

Additional insights via Hacker News discussion

Sometimes if you just wait long enough, the universe will provide what you’ve been looking for.

I’m a heavy user of Click, the Python library for creating command line interfaces (CLIs). There are a lot of idiomatic ways that I build my CLIs. For example, I’m a bit persnickety about logging configuration. Timestamps should either be UTC ISO 8601 or UNIX epoch if needed. And every CLI should have options for setting the logging level. But I’ve never sat down and put all of my preferences together in one easy to reuse place.

Simon Willison released click-app, a “Cookiecutter template for creating new Click command-line tools.” Using Cookiecutter to roll up all of my Click customizations has been on my to do list. I’ll probably fork Simon’s repo but it’s already got 80% of the soluiton I would have implemented myself.

Slowly but surely making progress on getting the blog authoring home lab set up. Lots of under the hood fiddling with personal machinery across a bunch of machines. Learned a lot of Ansible, NFS, and Samba. The goal was to get an environment where one machine can be mostly headless and build the pages automatically. At the same time, the base blog content could be remotely accessible via a home wifi LAN. A LAN that conveniently reaches my back porch so I can author while sitting outside, a glass of bourbon nearby.

The old scheme with WordPress and a desktop tool definitely had a lighter path to posting. With some work, though, I’ll be able to smooth out the process and get a bunch of upsides from the Pelican ecosystem.

Listened to this week (LTTW) includes Cozmik Beatz, a Drum ’N Bass DJ out of England. Very similar style to Calibre.

Started off with Cozmik Beatz v17 and enjoyed a bunch more from the playlist

read more ...

Let’s see if the summary and oEmbed plugins actually work …

read more ...

Okay, I’m making the move to static site generation. With any luck, this will be my last post using WordPress. There will be some domain name shifting so probably some upset to feed readers. Such is life.

Herewith, some enjoyable email newsletters that I’m subscribed to.

After leaving O’Reilly Media, Ben Lorica decided to keep up the podcasting bug with The Data Exchange. Gradient Flow is his parallel site and newsletter on the topics of data, business, machine learning, and AI.

PyCoder’s Weekly provides a nice assortment of Python links. The overall volume and topic distribution works for me.

I’ve been subscribed to Python Weekly for ages. The number of links is a little higher than is useful to me and the number of unsummarized links has increased recently. That said, there’s usually one or two worth following in every edition.

Link parkin’: Materialize

The simplicity of SQL queries, but with millisecond-level latency for real-time data. That is Materialize, the only true SQL streaming database for building internal tools, interactive dashboards, and customer-facing experiences.

Wire format compatible with PostgreSQL, so you can use the psql command line tool, even though there’s not a Postgres database underneath. Possibly a worthy challenger to ksqlDB

Principally brought to you by that guy who wrote “Scalability! But at what COST?”

Also in the bin of interesting potential side projects, would be deploying an Algo server for a personal VPN.

From the original Algo announcement post:

Today we’re introducing Algo, a self-hosted personal VPN server designed for ease of deployment and security. Algo automatically deploys an on-demand VPN service in the cloud that is not shared with other users, relies on only modern protocols and ciphers, and includes only the minimal software you need.

There’s richer and more recent detail in The Changelog podcast Episode 377.

Note to self, future blog post(s) on recommended podcasts.

The 20.04 Ubuntu Long Term Support release (Focal Fossa) officially supports the Raspberry Pi!

Running Ubuntu Server on your Raspberry Pi is easy. Just pick the OS image you want, flash it onto a microSD card, load it onto your Pi and away you go.

With handy installation instructions to boot.

Long term readers may remember that I put together a stack of Raspberry Pis just for fun one Christmas season. They haven’t been doing much since, but because I’m a heavy user of Ubuntu at work, maybe I can get more mileage out of them.

Inching my way towards using the Pelican static site generator for this blog, I realized that search is actually a must have. The path of least resistance seems to be to use the Tipue Search plugin. Tipue Search basically takes a statically generated search index as a big wad of JavaScript and does search client side. Not sure about the overall merits of this approach but it looks like a lot less hassle than deploying a parallel self-hosted search engine, especially after going to the trouble of static site generation to begin with. Then again, figuring the search part could be a fun side project.

Alternatively, looks like incorporating Algolia wouldn’t be too challenging.

Hat tip to this tutorial by Maxime Laboissonniere.

Personally, 2019 was a real struggle of a year for me. But 2020 is flat godawful from a global societal perspective. And we haven’t even quite hit mf’in June yet!

The fact that the President of the United States was impeached for only the third time in the country’s history will probably be a footnote in this year’s historical record. Back to Normal (TM) is oxymoronic. This is your normal. Best be about making it better, and quick.

More technical and entertainment shenanigans are forthcoming. But a facade of “This is Fine” would be irresponsible. Had to uncork a little and at least acknowledge the turmoil.

Forced to work from home more, I’m at least enjoying the opportunity to play music out load over speakers. Look Ma! No office headphones. Maybe in another post, I’ll document my foray into the world of Sonos, but suffice it to say it’s fun to listen to music out loud for a change.

I’ve also been committed to finding some new artists to get in my rotation. Don’t know how I discovered her, but Ms. Mada has been a revelation, especially her “Late Night With Ms. Mada” playlist on Soundcloud. Broadly speaking, her sets are House Music but not categorizable, at least by me, in any one of its many splintered genres. Noticeably different from the traditional stylings out of Chicago, New York, and San Francisco, but I’m really enjoying the stripped down, excellently blended rhythms.

BTW, Sonos gear may be a bit of a pain in the ass and often counterintuitive, but it integrates well with Internet streaming audio services like Soundcloud, Spotify, and TuneIn. And apparently there’s a Sonos device API, scriptable with Python.

At the day job, we rely heavily on GitLab which is a beast of a platform, even in if you only consider all the features in the FOSS version. Looks like I’m going to have to automate some processes on top of GitLab and thankfully, there’s a rich RESTful API.

Even better for me there’s a nice looking Python client package for the api that also provides a command line interface. Looks like the client library is pretty well maintained although the CLI needs some love.

One new piece of kit in my tech toolbox is the Caddy web server. Small, self-contained, and full featured, it definitely comes in handy for personal tech projects and professional prototypes. Biggest win is that it automatically handles HTTPS certificates through LetsEncrypt. As in, you just put the domain name in the config file and HTTPS cert management is solved.

Brilliant!

Well quite a bit has happened in the world since last year eh?

Still surviving.

This blog will make a comeback. Really. The work on static site generation made a big leap this past weekend. Not too far from going live with new thoughts, new interests, and a new look.

Stay tuned.

Link parkin’: homely

For lo these many years, I’ve never really consolidated and managed my various configuration dotfiles. Finally been regularly working on enough different machines that now’s the time. Being a Pythonista, homely feels like the right tool.

So I’ve said in the past that I wanted to investigate the Discogs.com data dumps as a side project. I’ve basicly failed, other than to keep collecting the data dumps continuously. To date, I believe there are 459 data files, totaling about 380+ Gb of compressed data. My finger in the wind estimate is that the uncompressed total is near about 4 Tb.

Just going to keep trying to take baby steps to build momentum on this. The available dataset now spans over a decade, which makes it interesting in its own right almost independent of what’s in the dumps. But it’s challenging because of data dirt from the early years and data scale recently. Just getting what I think is an accurate listing and count of the file dumps was surprisingly difficult to generate.

Aspiring to be to Discogs data what Mark Litwintschik is to the 1.1 Billion Taxi Rides data.

So I made my own fork of pelican to just extract the raw content from a WordPress XML export. Initial indicators are pretty good. Using pelican’s built-in web serving capability, a small sampling of the derived posts seem to have come through completely intact and appear to do well with pelican’s transformation to HTML.

Next up, exploration of pelican theming to eventually give this blog a fresh face.

Longtime NetNewsWire fan here. Glad to see Brent Simmons has reached a significant milestone in bringing NetNewsWire to full functionality. With the feedbin.me integration NetNewsWire is now really useful on a personal basis. I can once again have a great MacOs desktop RSS experience. Looking forward to what’s coming down the pike.

After five years of work — including getting the name NetNewsWire back, and a beautiful new app icon by Brad Ellis — NetNewsWire 5 has finally hit the alpha stage.

Simon Willison’s datasette project is just totally darn cool. If you have some data in an sqlite database (or in a csv file which is easily turned into an sqlite db) then datasette makes it easy to publish that data on the Web. With data oriented retrieval APIs for free!

The project mostly emerged while I was out in the blogging wilderness. I’ve had opportunity to use it in anger and can highly recommend it. There have been more sophisticated and complex means to achieve the same goals, and maybe even more elegant toolkits that never quite got adoption, but datasette strikes a beautiful balance between pragmatism, utility, and beauty.

Go figure. Based upon these instructions from Codeholics, I started to convert MPR using the Python powered Pelican. Pelican did a good job of cleanly processing my WordPress export, and extracting my posts and other content. It looked like I was on my way.

Except my posts are written in Markdown and stored that way in the WordPress database. Which Pelican’s import function conveniently escapes when it generates Markdown. Sigh.

It may not be the most elegant way, but I think I can work around this with a custom fork of the Pelican source code. At worst, I’ll be flexing some atrophying Git muscles.

I’ve managed to make it fifty two years on this here planet. Yeah me!

Seems like as good a time as any to begin resuscitating this venerable blog. It’s been over 10 years since I first posted on this blog and January 2003 was when I first started at all.

First thing to do is to convert over to serving static HTML.

More to come …

I’ve been tracking the progress of TimescaleDB for a while now. One thing that really stands out is the company’s pragmatic nature. Sure they came up with an innovative way to scale time series data storage, management, and querying. But it seems like they’ve really caught traction by meeting many customers where they’re at: relational DB knowledgeable and okay with using PostgreSQL. In a number of recent podcasts, I haven’t really heard the founders geek out about the underlying techniques but instead focus on how the product, not the technology, addresses customer pain points.

To wit, a recent company blog post on marrying TimescaleDB with the popular Prometheus monitoring and metrics platform:

By using Prometheus and TimescaleDB together, you can combine the simplicity of Prometheus with the reliability, power, flexibility, and scalability of TimescaleDB, and pick the approach that makes most sense for the task at hand. In particular, it is because Prometheus and TimescaleDB are so different that they become the perfect match, with each complementing the other. For example, as mentioned earlier, you can use either PromQL or full SQL for your queries, or both.

In particular, TimescaleDB engineers have done some of heavy lifting in creating a PostgreSQL connector for the Grafana metrics visualization framework. That’s putting skin in the game that customers can see.

Also, “It’s just Postgres,” is a great talking point.

I like where these guys are going.

Well, that was fun.

Started back to full employment this past Monday. The onboarding has been painless so far, even to the point of my personally designed Uplift standing desk being already assembled when I walked in the door.

Obviously a long way to go, but the commute reduction is feeling like a ridiculous win. Plus the team I joined is really even keel, low drama, and generally quiet in the open space lab we occupy, when there are actually people there. About half the company is remote. Sure, I’d love an office with a door, but sometimes you gotta live with the tradeoffs.

More to come …

The great thing about this post from Cloudflare, “How to drop 10 million packets per second”, is all the fun little low level networking tools, (ss, ethwatch, conntrack), I learned about.

Dropping packets hitting our servers, as simple as it sounds, can be done on multiple layers. Each technique has its advantages and limitations. In this blog post we’ll review all the techniques we tried thus far.

One of the things about having an ARM-based RPi cluster is a need to serve custom images. Even though there are a number of well run, cloud stored image registries, including Docker Hub and Google Container Registry, it feels like this is a homebrew style service that one should be able to host on their own. Straight Docker Distribution is surprisingly barebones.

Meanwhile, VMWare has open-sourced Harbor, an image registry which seems much more full featured:

Project Harbor is an an open source trusted cloud native registry project that stores, signs, and scans content. Harbor extends the open source Docker Distribution by adding the functionalities usually required by users such as security, identity and management. Harbor supports advanced features such as user management, access control, activity monitoring, and replication between instances. Having a registry closer to the build and run environment can also improve image transfer efficiency.

While there are quite a bunch of them, the fundamental conceptual elements of Kubernetes are fairly accessible. Nodes? Check. Containers? Check. Pods? Check. Services? Pretty straightforward, although there is some not oft mentioned complexity in the underlying network routing across pods.

A Kubernetes Service is an abstraction which defines a logical set of Pods and a policy by which to access them - sometimes called a micro-service. The set of Pods targeted by a Service is (usually) determined by a Label Selector (see below for why you might want a Service without a selector).

Check out the “Virtual IPs and service proxies,” subhead of the Services docs to see what I mean about networking.

Exposing services to the outside world? Not so much. Alternatively, if you can make sense of this gobbledygook, you’re a better person than I. Service Types are the singular concept where I have yet to see a good, comprehensible tutorial, either written, audio, or video. Something I’ll be on the lookout for.

Link parkin’

Since I’m a Digital Ocean customer, this article was quite handy. Getting Started with Software-Defined Networking and Creating a VPN with ZeroTier One by Sam Cater:

ZeroTier One is an open-source application which uses some of the latest developments in SDN to allow users to create secure, manageable networks and treat connected devices as though they’re in the same physical location. ZeroTier provides a web console for network management and endpoint software for the clients. It’s an encrypted Peer-to-Peer technology, meaning that unlike traditional VPN solutions, communications don’t need to pass through a central server or router — messages are sent directly from host to host. As a result it is very efficient and ensures minimal latency. Other benefits include ZeroTier’s simple deployment and configuration process, straightforward maintenance, and that it allows for centralized registration and management of authorized nodes via the Web Console.

By following this tutorial, you will connect a client and server together in a simple point-to-point network. Since Software-Defined Networking doesn’t utilize the traditional client/server design, there is no central VPN server to install and configure; this streamlines deployment of the tool and the addition of any supplementary nodes. Once connectivity is established, you’ll have the opportunity to utilize ZeroTier’s VPN capability by using some clever Linux functionalities to allow traffic to leave your ZeroTier network from your server and instruct a client to send it’s traffic in that direction.

The following was helpful in getting ZeroTier up and running on my home k8s cluster. Accessing your Raspberry Pi securely from the Internet using ZeroTier by Kelvin Zhang:

When you need to access your Raspberry Pi from home, exposing your public IP/using dynamic DNS and opening ports can expose your Pi to potential security threats, especially if you’re using password-based authentication or running services behind these ports.

The well-known method of doing it is to use a VPN. Whereas OpenVPN is a common solution, ZeroTier heavily outshines it. OpenVPN can be cumbersome to set up and maintain (especially if things go wrong), and provisioning new devices can be a pain with having to generate certificates. In comparison, ZeroTier can be installed with a single bash script, and your virtual network can be managed with their web panel which enables you to provision devices, assign static IPs and more.

Give ’em a read if this stuff interests you.

Last November, I threatened to build a Kubernetes cluster out of Raspberry Pi 3s. Well I actually did it starting during the December holidays and finishing up in January. Here’s a picture of it:

The one warning, that’s not obvious from the construction guide, is that the Raspberry Pi ARM processor architecture typically doesn’t have popular Docker images publicly available. This makes it somewhat challenging to do anything further usefully non-trivial. All-in-all, while not cheap, it was still a fun project and handy to have a k8s lab at home to play with.

I swear I’ve written about ZeroTier somewhere else before, but apparently not on this blog. The company and technology first came across my radar in a PacketPushers podcast episode that was a really deep technical dive. From the current front page of the website:

ZeroTier delivers the capabilities of VPNs, SDN, and SD-WAN with a single system. Manage all your connected resources across both local and wide area networks as if the whole world is a single data center.

Behind the scenes, ZeroTier uses software defined networking and cryptographic techniques to build secure, planetary-scale, virtual Ethernet networks. For the administrator, installation and setup is a relatively painless experience as these things go. Meanwhile, devices in a ZeroTier network can interconnect as if they were on the same local-area network (LAN) wherever they are. ZeroTier endpoints conveniently figure out ways to punch through firewalls and other network obstructions. Sort of like VPNs with 90% less hassle and a 90% more fun from a networking perspective.

Recently I setup ZeroTier on my personal laptop and a home Raspberry Pi 3 cluster. The cluster is behind the firewall of a wireless router and my service provider, but it’s been pretty seamless to remotely SSH into the cluster from just about anywhere.

The only potential downer, if you’re really into this stuff, is that the free service relies on a kernel of centralized infrastructure maintained by the ZeroTier company. Using the service thus places trust in ZeroTier’s security, infrastructure capabilities, technical competence, etc. etc. A not negligible concern to an entity’s business processes. This is counterbalanced by an open source codebase and a commercial option for on-prem deployment if full accountability is needed.

For me though, ZeroTier has worked better than expected and there’s some interesting underlying tech below the surface.