Check out our new piece: “Fast Data in the Era of Big Data: Twitter's Real-Time Related Query Suggestion Architecture” http://t.co/KBzzvM2q
— Jimmy Lin (@lintool) October 31, 2012
Preprint paper from the guys at Twitter: “Fast Data in the Era of Big Data: Twitter’s Real-Time Related Query Suggestion Architecture”
We present the architecture behind Twitter’s real-time related query suggestion and spelling correction service. Although these tasks have received much attention in the web search literature, the Twitter context introduces a real-time “twist”: after significant breaking news events, we aim to provide relevant results within minutes. This paper provides a case study illustrating the challenges of real-time data processing in the era of “big data”.