PostgreSQL is totally awesome. But sometimes it’s more useful to have pure file(s) storage and query for your data. Herewith a collection of data storage engines that somewhat cover the space of more well-known engines:
- sqlite, PostgreSQL, “SQLite is a C-language library that implements a small, fast, self-contained, high-reliability, full-featured, SQL database engine. SQLite is the most used database engine in the world.”
- vedis, Redis, “Vedis is an embeddable datastore C library built with over 70 commands similar in concept to Redis but without the networking layer since Vedis run in the same process of the host application.”
- xapian, Lucene/Solr/ElasticSearch, “Xapian is an Open Source Search Engine Library, released under the GPL v2+. It’s written in C++, with bindings to allow use from Perl, Python 2, Python 3, PHP 5, PHP 7, Java, Tcl, C#, Ruby, Lua, Erlang, Node.js and R (so far!)” Full text search in SQLite is also quite powerful.
- whoosh, Lucene/Solr/ElasticSearch, “Whoosh is a fast, pure Python search engine library.” Fudging a little here but definitely embeddable in Python applications 🤣.
- UnQLite, MongoDB and others, “UnQLite is a in-process software library which implements a self-contained, serverless, zero-configuration, transactional NoSQL database engine. UnQLite is a document store database similar to MongoDB, Redis, CouchDB etc. as well a standard Key/Value store similar to BerkeleyDB, LevelDB, etc.”
- Mongita, MongoDB, “Mongita is a lightweight embedded document database that implements a commonly-used subset of the MongoDB/PyMongo interface.”
- DuckDB, ClickHouse, “DuckDB is an in-process SQL OLAP database management system”
Of course these libraries come with limitations relative to their analogs, SQLite is not a like-for-like Postgres replacement, but they each have their place and time. Most of all, you can go a long way with many of these before having to go full client/server model.
Big shout out to Charles Leifer who did a wonderful job blogging about Python and many of these stores. Along with Simon Willison, his writings taught me a ton about the capabilities of lesser known datastores, especially SQLite. Too bad his blogging seems to be on hiatus.