home ¦ Archives ¦ Atom ¦ RSS

Yet Another MapReduce DSL

Apache Crunch has been around for a while, but the recent addition for support of Apache Spark and a Scala REPL caught my eye:

Running on top of Hadoop MapReduce and Apache Spark, the Apache Crunchâ„¢ library is a simple Java API for tasks like joining and data aggregation that are tedious to implement on plain MapReduce. The APIs are especially useful when processing data that does not fit naturally into relational model, such as time series, serialized object formats like protocol buffers or Avro records, and HBase rows and columns. For Scala users, there is the Scrunch API, which is built on top of the Java APIs and includes a REPL (read-eval-print loop) for creating MapReduce pipelines.

© C. Ross Jam. Built using Pelican. Theme by Giulio Fidente on github.