Technology and Software, Tips

Named pipes, ports and Erlang code in an Elixir project

I needed to read from a named pipe in an Elixir program, I made a mistake and I learned a few thing. Follow me.

First create the named pipe:

$ mkfifo "pipe"

In Erlang you read from it like this

$ erl
1> Fifo = open_port("pipe", [eof]).
2> receive
2>   {Fifo, {data, Data}} ->
2>     io:format("Got some data: ~p~n", [Data])
2>   end.

You can check that it works by executing echo hi > pipe from another shell.

So I expected to able to write this in Elixir

$ iex
 iex(1)> fifo = Port.open("pipe", [:eof])

However this is what I get

** (ArgumentError) argument error
:erlang.open_port({"pipe"}, [:eof])

If you can see my noob mistake in the call to Port.open don’t tell anybody yet, read on.

After trying many permutations of the arguments and read twice both http://erlang.org/doc/man/erlang.html#open_port-2 and http://elixir-lang.org/docs/stable/elixir/Port.html I gave up. I resolved to write it as an Erlang module and call it from my Elixir project. It was pretty easy:

1) Create an erlang directory in the main directory of the project. The name is not magical, you can name it as you wish.

2) Add this line into the project options of mix.exs

erlc_paths: ["erlang"],

That’s the directory you created.

3) Create an erlang/namedpipe.erl file with this code

-module(namedpipe).
-export([read/1]).

read(Pipe) ->
  Fifo = open_port(Pipe, [eof]),
  receive
    {Fifo, {data, Data}} ->
      Data
  end.

See how it can almost map 1 to 1 to Elixir. Variables are capitalized, symbols are lowercased and there are statement terminators (comma and full stop). All functions are private to the module except the explicitly exported ones.

4) Run mix and see that it compiles the Erlang file. Great!

But now

$ iex
iex(1)> :namedpipe.read("pipe")
** (ArgumentError) argument error
:erlang.open_port("pipe", [:eof])
erlang/namedpipe.erl:5: :namedpipe.read/1

Oh oh, what’s going on? I finally realized that it’s because of the wrong quote character! Single quotes are needed when passing strings to Erlang. Single quotes in Elixir are character lists, which is what Erlang needs. Double quotes are UTF-8 encoded binary data, which Erlang doesn’t understand.

So this works:

$ iex
iex(1)> :namedpipe.read('pipe')

Now go to another shell and run echo hi > pipe and confirm that it works for you too.

But wait, I did use double quotes in my first failed Elixir attempt. So did I do all of this for nothing? Embarrassingly, yes. This works:

$ iex
iex(1)> fifo = Port.open('pipe', [:eof])
iex(2)> receive do
...(2)>   {fifo, {:data, data}} ->
...(2)>     IO.puts("Got some data: #{data}")
...(2)> end

At least I learned how to embed Erlang code in an Elixir project and to care about single and double quotes.

Finally, you can embed that code in an Elixir module and call it both with a single quoted char list or a double quoted binary. Write this into lib/namedpipe.ex

defmodule NamedPipe do
  def read(pipe) when is_binary(pipe) do
    read(String.to_char_list(pipe))
  end

  def read(pipe) do
    fifo = Port.open(pipe, [:eof])
      receive do
        {fifo, {:data, data}} ->
          data
      end
  end
end

It uses guards to decide which version of the read function to call. Now

$ iex -S mix
iex(1)> NamedPipe.read("pipe")
'hi\n'
iex(2)> NamedPipe.read('pipe')
'hi\n'

Success!

Standard
Technology and Software, Tips

Upgrade a Rails 4 app to Rspec 3

I have a Rails 4 application with Rspec 2. I’m using a mix of should and expect assertions. I wanted to upgrade to Rspec 3 without changing the specs for now. I updated the Gemfile, run bundle install, rake spec and got many errors. Basically most helpers went missing (undefined methods visit, sign_in, root_path, etc., plus anything defined inside app/helpers). Googling around I found a solution for everything but the keys to restore the old behaviour are two.

1) The new rspec doesn’t include helpers based on the directory the specs are stored into. You either define the spec type with stuff like type: :feature or type: :controller or you add

config.infer_spec_type_from_file_location!

to the Rspec.config block.

2) The should syntax has been deprecated and doesn’t work any more by default. You must enable it with

config.expect_with :rspec do |c|
c.syntax = [:should, :expect]
end

Minor quirks:

  • You must remove require ‘rspec/autorun’
  • example doesn’t exist anymore. It has been replaced by RSpec.current_example
Standard
Technology and Software, Tips

Starting the VirtualBox interface at 192.168.56.1

I like to have some virtual machines arranged as I described in my post VirtualBox 4: NAT + Bridged Networking. To let them access the development servers I run on the host system I bind them to the VirtualBox network adapter vboxnet0 at 192.168.56.1. The problem is, this interface doesn’t exist until a VirtualBox’s VM starts and if I run one of those servers with Apache (as a proxy or a mod_php application) Apache won’t start. There is a little trick to start the interface even without running VirtualBox. This is for Linux:

VBoxManage list vms
sudo ifconfig vboxnet0 192.168.56.1

That VBoxManage command comes from here. You can hide it’s output with > /dev/null 2>&1

What it does is listing the virtual machines configured in VirtualBox but as a side effect it creates the network interface. No more tedious operations of running VirtualBox, starting a VM and closing it just to get the interface up!

Standard
Technology and Software, Tips

VirtualBox 4: NAT + Bridged Networking

VirtualBox networking gives us several options. NAT is interesting because it protects our guest systems from the Internet but it’s annoying we need to setup port forwarding to access the guests from the host (I have servers on some guests). Bridged Networking solves that problem easily, but the guests are exposed (not acceptable) and it seems that the communication between guest and hosts is routed outside the machine: in my case the bytes go to the switch my ISP installed in my home and that’s does only 10 Mb/s. I can probably setup some manual routing to solve that problem but if I have to tinker with routing then I can get something better: the combination of both NAT and Bridged Networking.

My setup is:

  • Host system: Ubuntu 10.10 with VirtualBox 4.0.4 (update: I’m on 11.04 and 4.1.8 now and it still works)
  • Guest systems: two headless Debian 4 and Debian 6 servers, three Windows XP clients to run tests with IE6, IE7 and IE8 (all the other browsers run directly on Ubuntu with the exception of Safari.)

The desired network configuration is:

  • The host eth0 interface must be the only access to the Internet
  • A virtual interface inside the host system should NAT the guest systems
  • The guest systems should be able to access the Internet using the internal NAT
  • The guest systems should be able to communicate with the host system using the virtual interface
  • The guest systems should be able to communicate with each other using the virtual interface
Internet <--- eth0 ---> HOST <--- virt/if (NAT) ---+---> guest 1 (server)
                   (server & client)               |
                                                   +---> guest 2 (server)
                                                   |
                                                   +---> guest 3 (client)
                                                   |
                                                   +---> ...

With this configuration I can attach application servers to the address of the virtual interface and make them available to all the guest systems for testing, as well as to any application I run on the host system. I also want to access application servers on the guests so I’m going to use static addresses because I don’t want to setup a dynamic DNS for the internal network.

VirtualBox gives us a virtual interface called vboxnet0 which starts with the 198.162.56.1 IP address. I take advantage of that.

The problems to solve are

  • Start vboxnet0 at boot time, because that address must be always available even if no guest OS is running.
  • Setup the NAT.
  • Assign addresses to the guest systems.

The procedure is based on https://help.ubuntu.com/community/Internet/ConnectionSharing but there are some differences.
The most important one is that connection sharing must not be enabled with the procedure shown there because it will change the address of your eth0 after a reboot and you won’t be able to access the Internet.

Configure the host

First, we bring up the virtual interface

sudo ifconfig vboxnet0 192.168.56.1

Then we configure NAT.

sudo iptables -A FORWARD -o eth0 -i vboxnet0 -s 192.168.56.0/24 -m conntrack --ctstate NEW -j ACCEPT
sudo iptables -A FORWARD -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
sudo iptables -A POSTROUTING -t nat -j MASQUERADE

Update: You lose the configurations on the vboxnet0 device if you remove VirtualBox to update it to a newer release. You have two options: 1) reboot and get them back from the configuration file (I’ll make you generate in the next steps of the procedure) or 2) just repeat the two steps above. I tested option 2 with the upgrade from 4.0 to 4.1.

We want to make those changes permanent.

sudo iptables-save | sudo tee /etc/iptables.sav
sudo vi /etc/rc.local

We add these lines

iptables-restore < /etc/iptables.sav
ifconfig vboxnet0 192.168.56.1

and we enable IP forwarding

sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
sudo vi /etc/sysctl.conf

with these lines

# Internet connection sharing for VirtualBox VMs
# https://help.ubuntu.com/community/Internet/ConnectionSharing
net.ipv4.conf.default.forwarding=1
net.ipv4.conf.all.forwarding=1

Update: VirtualBox still works with that configuration after updating to Ubuntu 11.04 so you can safely chose to keep that file when the update process asks you whether to keep it or overwrite it with the standard one. By the way, it seems that IP forwarding is done with net.ipv4.ip_forward=1 in 11.04 but I didn’t tried it out.

We can restart the networking or reboot (it’s fast).

netstat -r should give something like this now:

Kernel IP routing table
Destination     Gateway              Genmask         Flags   MSS Window  irtt Iface
your.host.ip.addr *                  255.255.255.192 U         0 0          0 eth0
192.168.56.0      *                  255.255.255.0   U         0 0          0 vboxnet0
link-local        *                  255.255.0.0     U         0 0          0 eth0
default           your.gw.ip.addr    0.0.0.0         UG        0 0          0 eth0

Configure the guests

Start VirtualBox. Change the network of every guest OS to be Host Only Adapter, vboxnet0

Startup every VM in turn and change it’s network configuration.

DEBIAN

The configuration on the two Debian servers is a little complicated because they’re headless and everything must be done by command line.

sudo vi /etc/dhcp/dhclient.conf

Uncomment

prepend domain-name-servers comma_separated_list_of_DNS_ip_addresses;

Restart networking. That will update /etc/resolv.conf

sudo vi /etc/network/interfaces

Add a static address for eth0.

allow-hotplug eth0
iface eth0 inet static
address 192.168.56.x # x is any free address in the 192.161.56 network
netmask 255.255.255.0
gateway 192.168.56.1
up route add -net 192.168.56.0 netmask 255.255.255.0 gw 192.168.56.1
down route del -net 192.168.56.0 netmask 255.255.255.0 gw 192.168.56.1

Restart the networking or reboot.
You should be able to ping 192.168.56.1 and access the Internet. Check it with
telnet http://www.google.com 80
From the host system you should be able to ping and ssh this server.

WINDOWS XP

The configuration on the three Windows clients is easier, about as easy as on Ubuntu

Click Start, Control Panel, Network Connections.
Right click on the connection, Properties.
Select Internet Protocol (TCP/IP), click Properties.
Select Use the following IP address and enter
IP address: 192.168.56.x (x = an address you didn’t use yet)
Subnet mask: 255.255.255.0
Default gateway: 192.168.56.1
Preferred DNS server and Alternate DNS server: the addresses of the DNS of your provider.
Click OK and again OK.

You should be able to ping 192.168.56.1 and access the Internet. Start up a browser and check it.
From the host system you should be able to ping this machine.

Epilog

I can start my application servers on the host system and bind them to 192.168.56.1 even if VirtualBox is not started (remember to update the addresses in their configurations, probably from 127.0.0.1 to 192.168.56.1).
I can access the Internet from my guest systems, which is handy for downloading updates, but the Internet cannot access them thank to the internal NAT. Too bad VirtualBox doesn’t give us this setup out of the box. The closest option is Bridged Network but the guest systems are exposed on the Internet.

Standard
Tips

On holiday with a PC and the Internet

I’ve spent one month on vacation in Australia and I had my netbook with me, a SSD-based eeepc 901 with eeebuntu 3. I planned to use it to check my mail, upload pictures and travel notes for my friends to see and occasionally do some urgent work for my customers (it turned out that I had to). I didn’t think about bringing my main machine with me (a notebook) because it’s more than twice as large, three times as heavy and the battery time can’t even be compared: it’s 7 hours for the eeepc vs 2 hours for the notebook (when its battery was new, it’s much worse now).

I learned some lessons in this month on the move and I want to share them. Most of them are about keeping your data safe. That means you can keep using your pc for all the time you’re away from home, which is important: if you don’t you’ll regret that extra weight in your luggage and the things you won’t be able to do.

  1. Have a backup of all programs and data you need on a server on the Internet. If something goes very wrong you can download them and start over. I lost a program I only had on a USB stick and couldn’t reach my pc and my backup disks at home to get it again ;-)
  2. Keep a local copy your mail because you’ll have to spend some time offline. Furthermore if you have local copies you won’t be forced to go online when it’s inconvenient or expensive to do it. I’m using Thunderbird and downloading every message, even from the gmail accounts I have to use for part of my work.
  3. Backup everything on USB sticks or to the Internet. I was rsyncing my mail to the flash drive and pushing code to git remote repositories. I had too many photos for uploading them over the average Australian Internet connection but I had three copies of them on SD cards, the pc’s SSD and the sticks.
  4. USB keys get corrupted sometimes and can’t be read anymore. You’re in trouble if you have something only there. Bring two sticks, maybe 8 GB or more. I had one and had to reformat it.
  5. Internet connections can be very slow and very expensive so don’t count only on the Internet to sync and backup data. Luckily I didn’t but read more about this below.
  6. Don’t plan to rely on 3G unless you know it’s available everywhere you’re planning to go. If it is, check the local data plans before you leave and buy a local data SIM card. I didn’t but I saw many people using them.
  7. Encrypt your file systems just in case your pc gets stolen or lost.
  8. The firewalls of some hotels let through only very few ports, sometimes only port 80. If the ssh port is open you can setup a ssh tunnel to smtp and pop servers (you need your own server on the Internet). It’s a bit painful but you can keep use your mail client. If it doesn’t, you need to use webmail until you get into a friendlier place or, if you planned in advance, you have a server on the Internet accepting ssh connections on port 80 and you tunnel over that.

Internet connections and phones deserve some more thoughts.

In my experience Internet was slow everywhere in Polinesia (tested in 2006) but they only have satellite links. Internet was slow everywhere in Mongolia but in the capital (2008). Australia fares better than them but worse than anywhere in Mexico (2005). That must be excused because bits have to go along cables under the ocean to reach every destination outside the country. Very remote areas such as Cape Tribulation might have incredibly slow connections reminding me of 56k modems. Those connections tend to be very expensive, maybe even more than 1 AUD per 10 minutes. Large cities have cheap connections which sometimes approached 1 Mb/s of real bandwidth. Not much actually but very fast compared to what you can get used to. Must locations in Australia have Internet access with Global Gossip. If WIFI is slow where you’re staying find out if they have a local shop and go there.

Always buy a local SIM card for your phone if you can. This is not possible in every country of the world but I got mine shipped from Australia to my home after making a VIP Backpackers card. I redirected my Skype account to it using call forwarding (you must pay for that but their rates are cheap) so customers and friends could talk to me without paying anything. I also created a Skype online number with my home town area code so people not using Skype could call me at their usual rate (you pay for that too).

When going to Australia keep in mind that phones often don’t work outside cities, even on the highways. The country is as large as Europe or the USA and there are not many people living there (20 millions) so a complete coverage it’s not economically feasible. Combining my experience with what people told me,Telstra has the best coverage followed by Optus. Vodafone has a small network (apparently merged with the one of 3) and it’s worth using it only if you’re staying in large cities. There are some virtual operators so check which network they use.

Standard