From Erlang Community

Current revision

[edit] Author

Michael McDaniel

[edit] Updates

Added link to application edids.org MichaelMcDaniel 17:55, 16 January 2011 (PST)

Added updates about R11B-5 SSL distribution problems and fixes BruceFitzsimons 10:12, 14 September 2007 (BST)

[edit] Title

Distributed Erlang using SSL through firewalls

[edit] License

 Copyright (c)  2006 Michael McDaniel.
 Permission is granted to copy, distribute and/or modify this document
 under the terms of the GNU Free Documentation License, Version 1.2
 or any later version published by the Free Software Foundation;
 with no Invariant Sections, no Front-Cover Texts, and no Back-Cover
 Texts.  A copy of the license can be seen by clicking the link entitled "GNU
 Free Documentation License".

[1]GNU Free Documentation License XML

[2]GNU Free Documentation License HTML

URL links in this paper were valid as of Mon Nov 20 22:03:25 PST 2006.

[edit] References

[3]1) Erlang SSL documentation [4]2) message from mailing list [5]3) message from mailing list

[edit] Overview

This howto describes how I got distributed Erlang nodes (each running on different physical servers) to communicate using the Secure Sockets Layer (SSL) protocol over the Internet through firewalls. I had control of the firewalls, and you will need control of your firewalls, to do what I describe here. Firewall configuration is dependent upon what kinds of firewalls are being used, and the network architecture. I also show a patch to inet_ssl_dist.erl which creates a new kernel environment variable to force SSL communication over a pre-determined port. This (or something similar) is a necessity if you want to limit your open firewall ports.

I do not describe the commands needed to configure firewalls. I do, however, give general guidelines which are required for any firewall with regards to opening and forwarding ports to have the Erlang nodes communicate.

Once you configure a node to use SSL, it can only talk with other nodes using SSL.

[edit] Introduction

[edit] Motivation

There are applications where you may want Erlang nodes to communicate over the Internet. Perhaps some distributed processing or data gathering. Perhaps embedded controllers doing remote control or monitoring, or TiVO (tm) like devices "phoning home" or receiving programming updates. Perhaps a distributed Mnesia database. Also useful would be if they communicate over the Internet securely using the SSL protocol.

[edit] Application

It happens that my application is a security application [6]. It determines if certains hosts (IPs) are attempting to compromise a server (what that consists of is a subject for a different howto). Once the application decides that an IP is an attacker, it takes some action(s). Attacks from that IP generally will be made on any server on that network (that is, any server with Internet visibility on that subnet).

Attacking IPs typically attack other servers on other networks, as well. The three networks I use as an example contain four servers of interest with Internet visibility. I often see the same IP attacking other servers on other networks.

Because the application is run on all the servers, any single server is fully capable of determining if an IP is an attacker and taking appropriate action. However, I decided I would have my application "warn" other servers of attacking IPs and save doing the same analysis work repeatedly on each server.

[edit] Main Text

[edit] Communications Overview

If multiple servers are on a single network, having Erlang talk amongst them is trivial. Simply use

Registered_Name ! {From, Message}. 
OR
global:send(Registered_Name, {From, Message}).

and it works (well, almost that simple).

It becomes more complex if you want to get through firewalls over the Internet (and want to use SSL). The servers I wish to communicate amongst each other are distributed over the Internet, and I want them talking privately using SSL.

You will need control of the firewalls and servers in question so that you can set the proper configurations. The techniques discussed map to any application that has inter-node communication using standard Erlang semantics.

Erlang can already "talk SSL". However, presently, I know of no standard way to choose what port to use. I have tried using the

-ssl proxylsport NNNN

command line flag and was unsuccessful in having it work as I expected (I thought it would limit SSL communications to port NNNN). I describe a patch to Erlang which creates a new kernel variable (inet_ssl_port) enabling selection of a specific port over which to "talk SSL" (similar to inet_dist_listen_{min,max} for normal TCP communications, though this new variable for SSL enables setting only a single port rather than a range of ports). I have submitted the patch to the [7]Ericsson Erlang developers, and I am hopeful that it or something like (perhaps a range of ports) it will get incorporated into the [8]standard distribution (or someone explains to me how to specify a single SSL port without my patch!).

If you do not have control of your firewalls, most of the techniques described here will not be useful to you. You will probably have to use something such as over SSL (though certainly, being able to control your SSL port will be useful!).

[edit] Environment

I have four servers (one Erlang node on each server), all running various release 9.x versions of [9]SuSE [10]GNU [11]Linux distributions.

NOTE: R11B-5 has some bugs in the inet_ssl_dist code that prevent SSL distribution from working. [12] There are patches to fix it. [13] BruceFitzsimons 10:12, 14 September 2007 (BST)

[edit] diagram

Much testing and development was done on a laptop loaded with release 6.10 of the [14]Ubuntu desktop [derived from [15]Debian] distribution. All servers run Erlang [16]R11B-2, and are are visible on three different (public) networks distributed across the Internet.

Each server has it's own hardware firewall (even the two on the same public IP block); two of the servers also run software firewalls.

The Erlang port mapper daemon epmd runs on the IANA ([17]Internet Assigned Number Authority) well-known port [18]4369 and is reserved for both TCP and UDP protocols. It was registered by [19]Ericsson Corporation for the [20]Erlang language. Using my patch, I was able to force the SSL communications to run on the same port for each node. This allowed me to open only two extra ports on the firewalls, 4369 for epmd, and a port for SSL traffic.

[edit] Details

$ this is a user command (note $ prompt)
# this is a root command (note # prompt)

System: Ubuntu rel 6.10 (aka Edgy; Debian derived)

$
$ uname -a
Linux delora 2.6.17-10-386 #2 Fri Oct 13 18:41:40 UTC 2006 i686
GNU/Linux
$
$ erl --version -s init stop      (OR erl -s init stop)
Erlang (BEAM) emulator version 5.5.2 [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.5.2  (abort with ^G)
1> 
$
$ gcc --version
gcc (GCC) 4.1.2 20060928 (prerelease) (Ubuntu 4.1.1-13ubuntu5)
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
$
$ echo version | openssl
OpenSSL> OpenSSL 0.9.8b 04 May 2006
OpenSSL> 
$
$ isql --version
unixODBC 2.2.11
$
$ ./configure  --with-odbc  --x-libraries=/usr/lib/unixODBC/ \
  --x-includes=/usr/include/ --with-ssl                      \
  --x-includes=/usr/local/ssl/include/openssl/               \
  --x-libraries=/usr/local/ssl/lib/

  [output not shown]

$ sudo make install

  [output not shown]

$ cd /usr/local/lib/erlang/lib/ssl-3.0.12/src
$ for arg in ../pkix/*.erl ; do sudo ln -s  $arg ${arg##*/}  ; done
$ cd /to/my/application/directory
$ erl -s systools make_script "start_ssl_clean" -s init stop
Erlang (BEAM) emulator version 5.5.2 [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.5.2  (abort with ^G)
1> 
$

Here is my SSL boot script.

%
% make sure all version numbers are correct
% * some will be in /usr/local/lib/erlang/bin/start.script
% * erts vers shows when starting command line erl OR
% * erlang:system_info(system_version)
% * ssl shows with 'ls /usr/local/lib/erlang/lib/ssl'
%
%
{release, {"OTP  APN 181 01","R11B"}, {erts, "5.5.2"},
 [{kernel,"2.11.2"},
  {stdlib,"1.14.2"},
  {sasl,"2.1.4"},
  {os_mon,"2.1.1"},
  {ssl,"3.0.12"}]}.
% see
% file:///usr/local/lib/erlang/lib/ssl-3.0.12/doc/html/ssl_distribution.html
% for ssl info
% 
% cd /usr/local/lib/erlang/lib/ssl-3.0.12/src
% for arg in ../pkix/*.erl ; do sudo ln -s  $arg ${arg##*/}  ; done
%
% systools:make_script("start_ssl_clean",[]).

I use RCS for local source control.

$ rcsdiff inet_ssl_dist.erl
80c80,86
<     case ssl_prim:listen(0, [{active, false}, {packet,4}] ++ 
---
> 
>     case application:get_env(kernel, inet_ssl_port) of
>          {ok, SSL_port} -> SSL_port ;
>          _undefined     -> SSL_port = 0
>     end ,
> 
>     case ssl_prim:listen(SSL_port, [{active, false}, {packet,4}] ++ 

Communication between nodes is initiated over epmd's well-known port 4369. In default behaviour, a different port is (randomly) chosen for the nodes to exchange messages. Since I do not want to open all ports on my firewalls (how would I know which one might be active?), I need some way to limit what ports are used to communicate between nodes over SSL. That is the purpose of the simple hack described above, so I can choose my port.

I start the application using the SSL boot file (-boot ${HOME}/start_ssl_clean). I limit the ports using the sys.config file (-config ${HOME}/sys.config).

merl is a 'safe' account created only to run certain programs, and with limited authority.

#!/bin/sh

# -setcookie is needed else I get 'No home for cookie file'
# the cookie a123 is a throwaway as the app creates a special
# cookie anyway
#
# copy *.pem to /home/merl
# copy start_ssl_clean.* to /home/merl
# 

export HOME=/home/merl
export HOST=`/bin/hostname`


(
/bin/su merl -c                                       \
"
cd /home/merl ;
/usr/local/bin/erl                                   \
-boot ${HOME}/start_ssl_clean                        \
-proto_dist inet_ssl                                 \
-ssl_dist_opt client_certfile ${HOME}/ssl_client.pem \
-ssl_dist_opt server_certfile ${HOME}/ssl_server.pem \
-ssl_dist_opt verify 1 depth 1 -name ${HOST}_myapp   \
-detached -config ${HOME}/sys.config               \
-s my_app start                                      \
"
)

#% end

$ cat sys.config

%% for example
[ {kernel,  [ {inet_ssl_port, 4370} ] }
].

I hope that is enough to get you started using secure communications between your Erlang nodes.