Generating Spark docs on Windows

The process of Spark docs generation for Linux is described in https://github.com/apache/spark/blob/master/docs/README.md. It might be a little tricky on Windows. You need to install Ruby 2.0, two jems then Python 2.7 and two jems. Nice docs for the first step is at http://jekyll-windows.juthilo.com/.

• Get Ruby 2.0.0 for your architecture http://rubyinstaller.org/downloads/. Install it and check "Add Ruby executables to your PATH"
• Get RubyDev from the same site. Unzip it to C:\RubyDevKit. Install it as follows. Before running the "install" command (last one), check that "config.yml" contains the path to your Ruby installation. Add it, if missing.

cd C:\RubyDevKit
ruby dk.rb init
ruby dk.rb install

• Install "jekyll" as follows specifying your proxy if needed:

gem install --http-proxy http://proxy:port jekyll
gem install --http-proxy http://proxy:port jekyll-redirects-from
groupadd hadoop
usermod -a -G hadoop hduser

• Get Python 2.7 from https://www.python.org/ and install it. Make sure that "Python" and "Python\Scripts" folders were added to your PATH.
• Install "pygments" and "sphinx". To use proxy, you need to have environment variable "http_proxy" with your proxy:port.

pip install pygments
pip install sphinx 

If everything was OK, you will be able to generate docs from docs folder. Lets' skip API docs:

cd %SPARK_HOME%\docs
set SKIP_API=1
jekyll build 

It will create a folder "_site" with all docs generated in HTML

In place update of Spark RDD

RDD in Spark are immutable. When you need to change the RDD, you produce a new RDD. When you need to change all the data in RDD frequently, e.g. when running some iterative algorithm, you might not want to spend time and memory on creation of the new structure. However, when you cache the RDD, you can get the reference to the data inside and change it. You need to make sure that the RDD stays in memory all the time. The following is a hack around RDD immutability and is not recommended to do. Also, you fault tolerance is lost, though you can force check-pointing.

// class that allows modification of its variable with inc function

class Counter extends Serializable { var i: Int = 0; def inc: Unit = i += 1 }

// Approach 1:
create an RDD with 5 instances of this class

val rdd = sc.parallelize(1 to 5, 5).map(x => new Counter())

// trying to apply modification

rdd.foreach(x => x.inc)

// modification did not work: all values are still zeroes

rdd.collect.foreach(x => println(x.i))



// Approach 2:
create a cached RDD with 5 instances of the Counter class

val cachedRdd = sc.parallelize(1 to 5, 5).map(x => new Counter()).cache

// trying to apply modification

cachedRdd.foreach(x => x.inc)

// modification worked: all values are ones

cachedRdd.collect.foreach(x => println(x.i))

Git https ssh errors

I get the error in Linux when trying to git push:
error: The requested URL returned error: 403 Forbidden while accessing https://github.com/avulanov/ann-benchmark.git/info/refs
When I set:
git remote set-url origin https://username@github.com/username/reponame.git
I get:
Gtk-WARNING **: cannot open display:
The following command fixes it:
unset SSH_ASKPASS
http://stackoverflow.com/questions/16077971/git-push-produces-gtk-warning

Installation of the new NVIDIA driver on Red Hat 6.3

In my case, when I download a new driver from http://www.nvidia.com/download/find.aspx and try to run it, I get "ERROR: You appear to be running an X server; please exit X before installing.". I comment x startup in /etc/rc.local and /etc/rc.d/rc.local (in my case there are two lines). After the reboot, there is no X server running, but running the driver installation tells: "unload nvidia module". I can't do this with "rmmod -f" because it is in use. GPU monitoring utility might use it. You might want to kill gmond. In my case the following worked:

sudo yum remove nvidia-kmod 

It uninstalls the kernel module with the driver. So now the new driver can be installed.

IntelliJ IDEA Scala: bad compiler option error message

It happens in 14.1.2. Remove it in Options->Scala Compiler->Additional options
http://stackoverflow.com/questions/26995023/errorscalac-bad-option-p-intellij-idea

GNU screen cheat sheet

Screen allows disconnect and connect to a running shell from multiple locations. If the process is already running you can steal it to screen using reptyr: reptyr PID. Screen cheat sheet from here: http://www.rackaid.com/blog/linux-screen-tutorial-and-how-to/

yum install screen
screen
screen -r #reattach
“Ctrl-a” then “?”. You should now have the screen help page.
“Ctrl-a” “c” new window
“Ctrl-a” “n” for the next window or “Ctrl-a” “p” for the previous
detach from the window using “Ctrl-a” “d”.
“Ctrl-a” “H”, creates a running log of the session.
“Ctrl-a” “M” to look for activity. 
“Ctrl-a” “x”.  This will require a password to access the session again.
“Ctrl-a” “k”.  You should get a message if you want to kill the screen

Cluster configuration and Apache Spark installation, configuration and start

Stand-alone cluster configuration notes

Skip this if you have already configured Hadoop cluster.
Create users on all nodes:

useradd hduser
passwd hduser 
groupadd hadoop
usermod -a -G hadoop hduser

Login as the new user:

sudo su - hduser 

Spark nodes interact with ssh. Password-less ssh should be enabled on all nodes:

ssh-keygen -t rsa -P ''
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

Check that ssh works locally without password:

ssh localhost 

Copy public key from the master node to worker node:

ssh-copy-id -i ~/.ssh/id_rsa.pub user@worker_node 

Spark compilation notes

Spark needs Java (at least 1.7), Scala 2.10 (does not support 2.11) and Maven. Hadoop is optional.
Install Java and Scala using package manager yum or apt-get. Or download rpm files from corresponding sites and run install:

sudo rpm -i java.rpm
sudo rpm -i scala.rpm

Download and unzip maven to /usr/local/maven. Configure proxy for Maven if needed in maven/conf/settings.xml. Another config file might be in ~/.m2/settings.xml.
Add to your ~/.bashrc

export M2_HOME=/usr/local/maven
export M2=$M2_HOME/bin
export PATH=$M2:$PATH

If Java is < 1.8:

export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"

If you need proxy:

export http_proxy=http://my-web-proxy.net:8088
export https_proxy=http://my-web-proxy.net:8088

Sometimes Maven collects these options, so be careful if the following does not compile because of maven trying do download something.
Clone from git, compile and change owner to hduser (the user who has password-less ssh between nodes enabled):

sudo git clone https://github.com/apache/spark.git /usr/local/spark
cd /usr/local/spark
mvn -Dhadoop.version=1.2.1 -Pnetlib-lgpl -DskipTests clean package
sudo chown -R hduser:hadoop /usr/local/spark

Spark installation notes

Assume that Spark compilation was done on the master node. One need to copy Spark to all other nodes in the cluster to /usr/local/spark and change its owner to hduser (as above).
Also, add to hduser ~/.bashrc on all nodes:

export SPARK_HOME=/usr/local/spark
export _JAVA_OPTIONS=-Djava.io.tmpdir=[Folder with a lot of space]

The latter option is needed for Java temporary folder when Spark writes data on shuffle. By default it is /tmp and it is usually small.
Also if there is Hadoop installation it is useful to force Spark read its configuration instead of using the default one (e.g. for replication factor etc.):
export PATH=$PATH:$HADOOP_HOME/conf

Spark configuration notes

Some theory:

• Spark runs one Master and several Workers. It is not recommended to have both Master and Worker on the same node. It worth having only one Worker on one node that owns all RAM and CPU cores unless it has many CPUs or the task is better solved ON many Workers. 
• When one submit a task, Spark creates Driver on Master node and Executors on Worker nodes. 

It would be nice that one has to configure only Master node and all options will be transferred to Workers. However, it is not the case. Though, in there is some minimal configuration when you don't need to touch each Workers's config. It is one Worker per node.
spark/conf/spark-defaults:

spark.master    spark://mymaster.com:7077
spark.driver.memory     16g
spark.driver.cores      4
spark.executor.memory   16g #no more than available, otherwise will fail
spark.local.dir /home/hduser/tmp #Shuffle directory, should be fast and big disk
spark.sql.shuffle.partitions    2000 #number of reducers for SQL, default is 200

List all the Worker nodes in spark/conf/slaves:

my-node2.net
my-node3.net 

Spark start

Start all nodes:

$SPARK_HOME/sbin/start-all.sh

You should be able to see the Web-interface on my-node1.net:8088

Start Spark shell:

$SPARK_HOME/bin/spark-shell.sh --master spark://my-node1.net:7077

Stop all nodes:

$SPARK_HOME/sbin/stop-all.sh