#Spark url extractor series
#Spark url extractor download
#Spark url extractor windows

Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0) Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties Log4j:WARN Please initialize the log4j system properly. Log4j:WARN No appenders could be found for logger (2.lib.MutableMetricsFactory). Next I started up the Spark interactive shell. This was for a sandbox environment – I am sure deploying to a real cluster would require more configuration.

#Spark url extractor download

It just worked! This may be why there are not lots of virtual images around – the download was self-sufficient (I already had Java installed).

#Spark url extractor windows

So I just tried to run it on my Windows laptop. In it I found a bin directory with a spark-shell command. (The Hortonworks demo image for example is 4GB!)Įventually, in frustration, I just grabbed and extracted the archive. I found the Spark download pretty easily ( spark-1.3.0-bin-hadoop2.4.tgz, 246MB), but was expecting some installation pain so wanted someone to do that work for me.

I was a bit surprised that it was harder to find than I expected, and similarly for Vagrant boxes. I have written a bit on Docker recently, so I went off looking for a Docker image. So the task is to parse the CSV file, extract the desired columns, then count the number of rows per combination of image size and format. The log file I am using has 16 columns, but only some are relevant. The goal is to work out the distribution of different file size and file format requests, to help work out the likely cache hit rates for a CDN. The project I am going to use in this blog is a CSV log file of image URL requests. Each intermediate step can be reused multiple times if desired. You can also ask for results to be cached on disk for faster processing.

#Spark url extractor series

My example below uses this approach of defining a series of new RDDs from a previous RDD, cleaning up the data step by step. Each view can enrich or restructure the data a bit more, ready for the next level of processing. If I had to pick an analogy, designing a Spark application feels a lot like defining a series of SQL views, one built on top of another. There is even an SQL parser that converts SQL queries into Spark jobs, allowing you to mix SQL queries and arbitrary processing logic in the one processing chain. The map() function is like the SELECT line in SQL, filter() is like WHERE, there a group by functions, join operations, and more. For example, a row could be an array, a Scala tuple (like a relational database row), a JSON object (like MongoDB), or any other serializable class. An RDD consists of a number of rows where each row can have arbitrary structure. A key concept is a RDD (Resilient Distributed Dataset), which you can think of like a database table. This gives you more flexibility in designing your applications. Spark supports map-reduce, but has other processing models as well. Spark is a newer parallel processing platform, but only really replaces the map-reduce engine of Hadoop. Just read and process all the data (like grep) rather than using indexes to speed up searches (like a database). I often think of it as a distributed grep command – it is a great way to bulk process all the input data without trying to be clever about what to read. Hadoop has very much a batch processing feel.

It is really a collection of technologies such as a distributed resilient file system (HDFS), job tracking, and a parallelized map-reduce engine. Hadoop is an older, more mature technology. So what is Spark and how is it different from Hadoop?

I was recently watching someone analyze log files of image URL requests using shell scripts to create a MySQL database and thought it might be an interesting exercise to try it in Spark as well. Apache Spark is a new wave in Big Data computing, an alternative to technologies such as Hadoop.