home ¦ Archives ¦ Atom ¦ RSS

Discogs Data Total Size

The next question I have about the Discogs Data is what’s the total amount to download? An initial step is updating my URL gathering script to grab the Content-Length header from an http probe of each url and start generating csv compatible output.


Below is the tweakage. The script has moved from straight Python to a xonsh script. The most interesting piece is line 6 where I shell out to use httpie to make a HEAD request against the url. The @ splices the value of the furl variable into the command line from the outer Python context, while ! captures the output of the inner subcommand for processing in the outer Python context.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
urls.sort()
print(f"{len(urls)} in total", file=sys.stderr) 
for i, url in enumerate(urls):
    clength = -1
    furl = "http:" + url[1]
    for line in !(http --headers HEAD @(furl)):
        if line.strip().startswith("Content-Length"):
            h, v = line.split()
            clength = int(v)
    print(f"{url[0]}, http:{url[1]}, {clength}")

Still need to pull in the csv module for proper output generation. I also want to add the http headers as a column of json data, which will definitely need correct csv escaping. Once that’s all in place, it should be possible to pipe the output into Simon Willison’s sqlite-utils to create an SQLite DB, then run a query to compute the total storage.

This is why I love xonsh. I had to do a little documentation reading to get the right sigils (!, @) in the right places, but once done I can still actually comprehend what the script is doing. If I keep working at it, this should become mental muscle memory and make writing such scripts completely natural. Might be better off learning bash in depth, but this fits my brain way cleaner.

Using questions to be asked of data seems to be a good driver for me to actually write code. Boy do I have plenty of questions for this data.

© 2008-2024 C. Ross Jam. Built using Pelican. Theme based upon Giulio Fidente’s original svbhack, and slightly modified by crossjam.