It’s ornery and has sharp pointy teeth, but I’m coming to appreciate the Sphinx full text indexing and search engine. Might not have the greatest documentation or APIs but damn does it index like a bat out of hell.
I’ve personally seen it rip through approximately 4 Gb of data on a 5 year old server with only 8 Gb of RAM, said data on a suboptimal Linux ext3 filesystem, on top of an untuned kernel, and with no thought given to the IO and HD subsystems. Grand total of 21 minutes.
That is a nice capability to have.
I know Lucene with Solr on top is sort of the default open source choice for full text indexing, but if you’re in the market Sphinx is worth a tire kick.