Funny thing about Apache Spark

To build it you need Java and Scala. To build its docs you need Ruby and Python.

Difference between multinomial logistic regression and artificial neural network (multilayer perceptron) without hidden layer

Initially, they seems to be the same: they both have sigmoids within and NxK parameters where N is number of inputs, K is number of outputs. However, regression uses so-called softmax function, so all outputs sum to 1. The difference comes when one needs to run optimization routine. Logistic regression optimizes cross-entropy and ANN optimizes mean squared error. One takes a partial derivatives of these equations which result in slightly different formulas for regression and back-propagation. See more at http://ufldl.stanford.edu/wiki/index.php/Backpropagation_Algorithm and http://ufldl.stanford.edu/wiki/index.php/Softmax_Regression.

Git rebase

Rebase is needed to put your history on top of history of the given branch, when you don't want to see instances of merge in the given branch history. This is important for keeping the main branch history clear.
Rebase process:

git rebase [upstream/master]

Hopefully, everything is ok. If there are conflicts, resolve them by hand or use theirs/ours:

git checkout --theirs (or --ours) filename

Then you need to add the edited file:

git add filename

And continue rebase if hand edited/used theirs or skip it otherwise:

git rebase --continue (or --skip)

HTTPS certificate for Tomcat

Tomcat has its own documentation how to do it. I've just followed it with Tomcat 7:

Generate a personal private key wiht Java keytool:

keytool -genkey -alias www.mysite.com -dname "cn=www.mysite.com, o=
mysite, o=.com" -keysize 2048 -keyalg RSA

Generate a request for certificate:

keytool -certreq -alias www.mysite.com -file www.mysite.com.csr

Submit the resulting request (as text) to the certificate authority. I did it with https://www.startssl.com/ for free. It produced a certificate (text) that I put into a www.mysite.com.signed.crt file.
Before using it, you need to import Root Certificate and the Class 1 domain validation certificate from the authority into your keystore, otherwise import of reply would not find the right chain. In my case it was:

wget http://www.startssl.com/certs/ca.crt

keytool -import -trustcacerts -alias startcom.ca -file ca.crt

wget https://startssl.com/certs/sca.server1.crt

keytool -import -alias startcom.ca.sub -file sub.class1.server.ca.crt

Finally, import certificate to your keystore:

keytool -import -alias www.mysite.com -file www.mysite.com.signed.c
rt

Now, you need to configure Tomcat via server.xml:

     <Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
               maxThreads="150" scheme="https" secure="true"
               clientAuth="false" sslProtocol="TLS"
                keystoreFile="/home/ubuntu/.keystore"
                keystorePass="PASSWORD"
        compression="on"
        compressionMinSize="2048"
        noCompressionUserAgents="gozilla, traviata"
        compressableMimeType="text/html,text/xml,text/javascript,text/css,image$
    />

You may also want all HTTP requests redirected to HTTPS:

    <Connector port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               URIEncoding="UTF-8"
               redirectPort="8443"
        compression="on"
        compressionMinSize="2048"
        noCompressionUserAgents="gozilla, traviata"
        compressableMimeType="text/html,text/xml,text/javascript,text/css,image$
/>

Not to mention, your iptables has route both HTTP and HTTPS requests to your Tomcat port:

#IPTABLES
# forward 80 to 8080 and save tables
# ports 80 and 443 must be opened in AMAZON AWS UI Securtiy Groups
sudo iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 8080
sudo iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 443 -j REDIRECT --to-port 8443
sudo iptables-save
sudo bash -c 'iptables-save > /etc/iptables/rules.v4'
sudo iptables  -nvL -t nat

Apache Spark workers configuration

How to guess the number of Workers per cluster node that will provide the best performance? I performed few tests on 6-node cluster with 3.3GHz 4 core 16GB machines. I used mnist8m classification task as a test. It terms that there should be Workers < cores. Moreover, 1 Worker per node worked the same time as more than one. This means that one worker takes advantage of multiple cores. Also, data partitions should be >= number of workers, otherwise it will be processed by less Workers.

Rule of thumb for classification

There are quite a few machine learning classifiers. It is usually hard to say which is better until every one is tried on the given data and performance is measured. However, there are few rules of thumb:

• Linear classifier is better used when:
• Sparse data (lot of zeroes in feature vector) 
• Feature engineering performed, or deep feature learning
• Up to large datasets (fits one machine)
• Non-linear or kernel-based classifier is better used when
• There are only few features (up to tens)
• Big data - a lot of training examples

Bonus: how to manage imbalanced training set:

• Evaluation: ROC under PR curve
• Negative subsampling
• Weighs for imbalanced classes (also - regularization parameter)

How to run netlib-java/breeze in native mode with BLAS/Lapack binaries on Windows 7 x64

Netlib-java allows fast linear algebra in native mode with BLAS/LAPACK libraries. Breeze is for Scala and is based on netlib as well. The problem is where to get binaries and how to set up netlib-java. So, which libraries are needed?

• libquadmath-0.dll //MINGW
  libwinpthread-1.dll //MINGW
  libgcc_s_seh-1.dll //MINGW
  libgfortran-3.dll //MINGW
  liblapack3.dll //OpenBLAS copy of libopeblas.dll
  libblas3.dll //OpenBLAS copy of libopenblas.dll
  netlib-native_system-win-x86_64.dll //netlib-java

So, few libs from MINGW, one from OpenBLAS copied two times and one from netlib-java. It is important that all libraries have the same architecture: x64 (or x32). Mixing doesn't work. Here are the links where these libraries exist pre-compiled:

• MINGW64: http://cznic.dl.sourceforge.net/project/mingwbuilds/host-windows/releases/4.8.0/64-bit/threads-posix/seh/x64-4.8.0-release-posix-seh-rev2.7z (or http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Automated%20Builds/ or http://sourceforge.net/projects/mingw-w64/?source=typ_redirect (dlls are in bin)
• OpenBLAS: http://www.openblas.net/
• Netlib-java: http://search.maven.org/#search%7Cga%7C1%7Ca%3A%22netlib-native_system-win-x86_64%22 (dll is inside the jar)

You need to place the needed dlls into the project folder or to some folder that is in PATH. Your maven pom project must have the following entry if you use only netlib-java:

•       <dependency>
            <groupId>com.github.fommil.netlib</groupId>
            <artifactId>all</artifactId>
            <version>1.1.2</version>
        </dependency>

or if you use breeze:

•     <dependency>
        <groupId>org.scalanlp</groupId>
        <artifactId>breeze_${scala.binary.version}</artifactId>
        <version>0.9</version>
      </dependency>
    <dependency>
            <groupId>com.github.fommil.netlib</groupId>
            <artifactId>all</artifactId>
            <version>1.1.2</version>
    </dependency>

Another option would be to compile OpenBLAS. You will need MINGW64, OpenBLAS sources plus MSYS64: http://sourceforge.net/projects/mingw-w64/files/External%20binary%20packages%20%28Win64%20hosted%29/MSYS%20%2832-bit%29/ (more details at https://code.google.com/p/tonatiuh/wiki/InstallingMinGWForWindows64)

• unzip OpenBLAS sources
• extract MINGW and MSYS in the same folder, 
• run msys.bat from MINGW/MSYS folder
• cd to OpenBLAS sources dir and run "make BINARY=64". After a while you'll get libopenblas.dll