Transfer files from an FTP server to S3

At work, for a very long time, we used our network provider's cache server to serve our static files (mainly pictures). We wanted a real CDN (as in globally distributed) that would be both faster and cheaper.

We chose to use the AWS S3+CloudFront combination. It was a true CDN, it was faster than another provider we trialled and we wouldn't have to worry about the space used.

In order to choose the right tools to transfer the files from the old cache server to S3, we had these requirements:

  • The old cache server could only be accessed via FTP.
  • We needed to be able to mirror the source to the destination. Because our staff and apps would continue to use the old cache server while we would be ramping up on S3, we didn't want to retransfer all the files every day.
  • Some of our directories contained several hundreds of thousands files. We needed a tool that could list these directories without crashing.
  • We had approximately 90 GB in 3,3 million files to transfer. Taking days to transfer everything was not an option.

After a little digging, I couldn't find a single piece of software that could connect to the FTP and copy/transfer only the new/updated files on the fly to S3.

Finally, I decided to use the following method:

  1. mirror the FTP files locally
  2. mirror the local directory to S3

Mirror the FTP locally

I used the versatile and robust lftp:

  • It can transfer files in parallel, making it really fast.
  • It can use several alternative methods to list directories, including one that can handle the very large directories without timing out or crashing.
  • The mirror command works flawlessly, I tested it with several combinations of new/updated/deleted files.
  • It keeps the original file timestamps.

So I booted up an m3.xlarge instance on EC2, added a volume big enough to hold all the files and launched this script:

lftp -e " \
    debug -t 2; \
    set net:max-retries 3; \
    set net:timeout 10m; \
    set ftp:use-mlsd on; \
    set ftp:charset iso-8859-1; \
    open -u USER:PASSWORD ftp.oldcacheserver.com; \
    mirror --parallel=30 --log=/tmp/lftpmirror.log; \
    exit; \
    "

debug -t 2: show only warnings and errors

set net:max-retries 3 and set net: timeout 10m: because the defaults are too long

set ftp:use-mlsd on: this is the setting that made listing very large directories work

set ftp:charset iso-8859-1: we had files with special characters in their name uploaded from a Windows computer to the FTP server. Setting this charset made lftp create files with the right characters on the Linux VM with en_US.UTF-8 locale

open -u USER:PASSWORD ftp.oldcacheserver.com: the connection command

mirror --parallel=30 --log=/tmp/lftpmirror.log: mirror the whole FTP server with 30 threads and a logfile

It took approximately 7 hours to transfer all the FTP files to the EC2 instance.

Mirror a local directory to S3

I used s3-parallel-put for the same reasons I used lftp: reliable, fast thanks to parallelism and able to handle very large directories.

These are the command I used:

export AWS_ACCESS_KEY_ID=THEAWSACCESSKEY
export AWS_SECRET_ACCESS_KEY=THEAWSSECRETACCESSKEY

python s3-parallel-put --bucket=BUCKET_NAME --secure --put=update --processes=30 --content-type=guess --log-filename=/tmp/s3pp.log .

First I export my AWS credentials in environment variables because s3-parallel-put needs the values this way.

Then I launch s3-parallel-put:

  • specifying the bucket name
  • forcing the use of the HTTPS API
  • forcing the update method for each file
  • spawning 30 threads to go faster
  • using the content-type=guess option: it allows s3-parallel-put to specify the file type while uploading (it can only detect pictures, but that's what I was uploading). It's important because otherwise AWS doesn't send the right HTTP header to the browser, and when accessed directly a picture is downloaded instead of being shown in the browser.

The EC2 instance being in the same region of my S3 bucket, the initial upload was blazing fast and took only 1.5 hours.

Rinse, repeat

The initial "snapshot" of the FTP to S3 took long, but then mirroring the FTP locally and mirroring to S3 took only the time to traverse all directories to detect and download the changed files — in my case: 35+38 minutes.

I made a small shell script combining the two commands, and scheduled it twice a day.

Going further

There is an intriguing option to the mirror command of lftp: on-change. It seems that if lftp detects a change, it can launch a shell script.

It would be interesting to know:

  • what does a "change" mean? Updated file? New file also?
  • is the script launched for each changed file or more globally at the end of the mirror?
  • what parameters are passed to the shell script?

By quickly browsing the lftp source code, it seems that the script is launched at the end of the mirror. In my case, it would have been more interesting if it were launched for each modified file: I could have used an S3 file uploader to upload each new/modified file individually.

Maybe in a future version, or — if I dare dive into C++ again — in a pull-request! :smile:

Glass

It all started with the most epic demo ever:

Live video feed through thin glasses! In the air!

It is not easy to get Glass in France because Google sells it sparingly, and in the US only. The equation was set:

device from the future + scarcity = gadget-loving-geek getting really excited

I can haz Glass?

I now own a Google Glass (by the way, please don't say "glasses", it has only one glass). me with glass

Being a so-called "Glass Explorer", I chose to wear it all day, every day. I wear it at work but also outside, in the streets, in the subway, in stores, at home.

Basically, if I'm awake, I wear Glass.

You look silly with it!

I have seen different reactions:

  • Some are curious and ask me to try it
  • Some laugh at me, thinking I'm ridiculous with such a big frame on the side and no glass at all, besides the big block in front of my right eye
  • Some gawk at me, not knowing what it is
  • Some stare angrily at me, mostly being afraid of being filmed

Can you think of another device that can spark such a variety of reactions, from enthusiastic to defiant? How cool is that? And it's really exciting to be part of the future-shaping, society-redefining community of Glass Explorers. Oh, and did I mention that you can take pictures of your cat with it?

What Glass can't do?

Let me debunk some myths about Glass:

  • It can't record everything you do continuously. The battery is just not capable of it.
  • It can't do facial recognition. Google forbids this kind of app. Of course, there will be unofficial apps that will do it anyways.
  • I can't see you naked through it. Just like there is no such thing as a "naked scanner" you can download for 1€ on your phone. Guys, seriously!

So, what can you do with Glass?

Glass is not always on. You have to activate it to see something on the screen. To do this, you just tap on the side, or turn your head up. You then see a screen showing the time. There you can either navigate on your timeline and see what you have done with Glass in the past, or you can speak an order. Yes, Glass is able to do voice recognition (english only for the moment). So you can say "take a picture" and voilà, it just took a picture of what you're looking at. You can do a google search, and the most relevant result will be printed on the screen AND spoken in your ear.

Here is an incomplete list of what you can do:

  • Take a picture, in a fraction of time you would need to pull your smartphone out of your pocket.
  • Record a video. In my personal experience, you can shoot about 20 minutes of video before the battery runs out.
  • Search via Google. Just speak what you are looking for and a nice female voice will read you the answer. It tends to give a wikipedia abstract often.
  • Make a phone call. Glass is connected to your smartphone via Bluetooth, so it becomes a hands-free headset. You can initiate or answer a call from Glass, no need to take your phone out of your pocket. And you can see who is calling, of course.
  • Send a message (hangout, SMS, tweet, Facebook status). Voice recognition again, so english only, but it works well.
  • Receive notifications of a new email, SMS, tweet or Facebook message. You hear a sound, and you have to activate Glass to see the notification on the screen.
  • Get directions via the GPS you have in your phone. Speak a destination and the step-by-step directions are shown on the screen. Battery killer.
  • Listen to music. You have to have a Google Play Music account and you hear it on the right ear only. Not the best experience.
  • Translate text. One of the apps that really impressed me is World Lens. It replaces a text with its translation in place. Really cool wow effect when demoing. (it also exists on Android and iOS)

But, what about your phone?

I have a the best phone on Earth.

Some sceptics ask me what I can do with Glass that I can't with a phone. My answer is this:

Try to shoot a video of a ride on bike with your daughter with a smartphone.

Glass is about not getting in the way, and being able to fully live an experience while recording and sharing it with others.

Installing Gentoo on an HP Folio 9470m laptop

This post is the instructions list I followed to install Gentoo on my new work laptop: an HP Folio 9470m. It's a mix of the official Installing Gentoo handbook, @ultrabug's french install guide and various resources I found while googling for this specific laptop.

This is mostly intended to be a reminder to future-me, but I'd be more than happy if it's useful to someone else. If so, let me know by writing a comment!

Prepare the USB drive

Boot on the LiveCD (well, LiveUSB drive)

  • boot on the USB drive
  • UNetbootin has a loader that allows you to choose what to boot, choose gentoo
  • you briefly have the option to change your keyboard layout, type the number associated with the layout of your laptop before the default one (US) is loaded

Tip: if the default was loaded before you could change it, and want to get (e.g.) the french layout, type loadkeys fr once you are in the shell.

Set the date

Verify that your clock is correct by typing date.

If not, you can change it via:

date MMJJhhmmAAAA

Configure Wifi

To verify that wpa_supplicant is available on your LiveCD, type /etc/init.d/wpa_supplicant start

You should get this kind of error:

 * Starting WPA Supplicant Daemon
Successfully initialized wpa_supplicant
Failed to open config file 'etc/wpa_supplicant/wpa_supplicant.conf', error: No such file or directory
Failed to read or parse configuration '/etc/wpa_supplicant/wpa_supplicant.conf'.
 * start-stop-daemon: failed to start '/usr/sbin/wpa_supplicant'
 * Failed to start wpa_supplicant
 * ERROR: wpa_supplicant failed to start

It indicates that wpa_supplicant is installed and that you need to provide it with a valid config file.

Taking info from the Gentoo handbook, edit the following file /etc/wpa_supplicant/wpa_supplicant.conf:

network={
  ssid="simple"
  psk="very secret passphrase"
}

Now retype /etc/init.d/wpa_supplicant start and you should see:

 * Starting WPA Supplicant Daemon
Successfully initialized wpa_supplicant [ok]

An iwconfig should show that you are connected to your SSID, and an ifconfig should show your IP address.

To verify that you really have Internet access, type ping -c 3 www.google.com to see the pings.

Partition the disk

You will make a GPT disk here, and use the without-GUI-but-still-excellent gdisk utility. It allows for automatic partition alignment. (see more tips for maximizing SSD performance)

To enter the partition utility, type gdisk /dev/sda. Then:

  • Print the existing partition table by typing p
  • Delete all the existing partitions (if any). Type d and the partition number.
  • Create the bios-boot partition by typing n, partition number by default (1), first sector by default (2048), last sector +2M, partition type ef02
  • Create the boot partition by typing n, partition number by default (2), first sector by default (6144), last sector +512M, partition type by default (8300)
  • Create the swap partition by typing n, partition number by default (3), first sector by default (1054720), last sector +8192M, partition type 8200
  • Create the root partition by typing n, partition number by default (4), first sector by default (17831936), last sector by default (500118158 if you have a 256MB SSD), partition type by default (8300)
  • Rename the boot partition by typing c, partition 2, name boot
  • Rename the swap partition by typing c, partition 3, name swap
  • Rename the root partition by typing c, partition 4, name root
  • Print the partition table to verify it by typing p
  • Write the partition table and quit by typing w, answer y at the warning

Init the partitions

  • Format the partitions
mkfs.ext2 /dev/sda2
mkfs.ext4 /dev/sda4
  • Init and activate the swap
mkswap /dev/sda3
swapon /dev/sda3
  • Now mount the partitions
mount /dev/sda4 /mnt/gentoo
mkdir /mnt/gentoo/boot
mount /dev/sda2 /mnt/gentoo/boot

All set, you'll now start to configure your final Gentoo system.

Download stage3

You can choose the current stage3 here: http://distfiles.gentoo.org/releases/amd64/autobuilds/current-stage3-amd64/

Note the URL for the stage3-amd64-YYYMMDD.tar.bz2 file

Tip: You are better off not choosing a nomultilib file if you want maximum compatibility.

Tip: You can choose a closer mirror to download the stage3.

cd /mnt/gentoo
wget URL_of_your_stage3_file
tar xpf stage3-*.tar.bz2

Install a portage snapshot

To download and install the latest portage snapshot, the easiest way is to use emerge-webrsync.

Prepare your Gentoo system

Before you chroot to your system, you need to prepare it first:

  • copy the DNS info
cp -L /etc/resolv.conf /mnt/gentoo/etc/
  • mount the filesystems
mount -o remount,nodev,nosuid -t tmpfs shm /dev/shm # to remount shm so you can use it as a temp dir for portage
mount -t proc proc /mnt/gentoo/proc
mount --rbind /sys /mnt/gentoo/sys
mount --rbind /dev /mnt/gentoo/dev
  • now chroot in your new environment
chroot /mnt/gentoo /bin/bash
  • adjust a few settings
env-update && source /etc/profile
export PS1="(chroot) $PS1"

From now on, everything you do is in your final system, not on the LiveCD.

Portage profile

The portage profile aims to pre-fill USE flags. Since this is a laptop and you want some graphical UI, choose the desktop profile.

First, type eselect profile list to see the list of available profiles.

Then type eselect profile set X where X is the desktop profile.

While you are at it, choose your Python version:

First, type eselect python list to see the list of available python versions.

Then type eselect python set X where X is the your prefered version.

make.conf

This laptop has an Intel Core i7-3687U CPU. It's an IvyBridge and as of gcc-4.7, its march is core-avx-i.

So here is the content of /etc/portage/make.conf:

CHOST="x86_64-pc-linux-gnu"
CFLAGS="-march=core-avx-i -O2 -pipe"
CXXFLAGS="${CFLAGS}"

MAKEOPTS="-j3" # 2 cores

USE="bindist mmx sse sse2 -ipv6 -kde -gnome xinerama v4l -llvm -llvm-shared-libs"

PORTDIR="/usr/portage"
DISTDIR="${PORTDIR}/distfiles"
PKGDIR="${PORTDIR}/packages"

PORTAGE_TMPDIR="/dev/shm"

VIDEO_CARDS="intel i915" # despite being an IvyBridge, the video card is still an i915, not an i965
INPUT_DEVICES="synaptics evdev"
ALSA_CARDS="hda_intel"

ACCEPT_KEYWORDS="~amd64" # yes, we will accept unstable packages! Crazy us!
FEATURES="buildpkg" # keep a compiled version of packages for speedier further uses 

EMERGE_DEFAULT_OPTS="--keep-going=y --quiet"

If you need to know which CFLAGS you should use with a different processor, you can go to the Gentoo wiki.

Locales

You need to specify which locales you want on your system.

Here is the content of /etc/locale.gen enabling the use of US english and french locales:

en_US ISO-8859-1
en_US.UTF-8 UTF-8
fr_FR ISO-8859-1
fr_FR@euro ISO-8859-15
fr_FR.UTF-8 UTF-8

You then have to run locale-gen to generate all the specified locales.

Finaly, edit your /etc/env.d/02locale file to match your preferences. Here is mine, mostly french, apart from messages:

LANG="en_US.UTF-8"
LC_CTYPE="fr_FR.UTF-8"
LC_NUMERIC="fr_FR.UTF-8"
LC_TIME="fr_FR.UTF-8"
LC_COLLATE="fr_FR.UTF-8"
LC_MONETARY="fr_FR.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="fr_FR.UTF-8"
LC_NAME="fr_FR.UTF-8"
LC_ADDRESS="fr_FR.UTF-8"
LC_TELEPHONE="fr_FR.UTF-8"
LC_MEASUREMENT="fr_FR.UTF-8"
LC_IDENTIFICATION="fr_FR.UTF-8"
LC_ALL=""

Don't forget to set your timezone. Type these commands for Paris, France:

echo "Europe/Paris" > /etc/timezone
emerge --config sys-libs/timezone-data

If you live somewhere else, check /usr/share/zoneinfo for a list of all the supported timezones.

You should now reload your environment:

env-update && source /etc/profile
export PS1="(chroot) $PS1"

First global update

Because you changed your portage profile and chose to enable unstable packages, it's good to launch a global update of the system. All "mandatory" packages will thus be installed.

emerge --update --deep --with-bdeps=y --newuse @world

Following this big emerge, you'll probably have to run (check the end of the emerge log to see if other messages exist):

rc-update add kmod-static-nodes sysinit

Now that your system is updated, you can remove obsolete and unused packages

emerge --depclean

Build your kernel

There is a very involving process where you can choose every single piece of code that goes into your kernel, and there is a simple one, where you don't have to decide anything.

This is what you'll choose here, and because the genkernel tool takes its config from what was detected at the LiveCD boot, your kernel will mostly be optimized for your laptop.

  • First, grab the kernel sources and the genkernel util
emerge gentoo-sources genkernel
  • Then copy the config used during the LiveCD boot to the file used by genkernel
zcat /proc/config.gz > /usr/share/genkernel/arch/x86_64/kernel-config
  • Edit the /etc/genkernel.conf file:

    • find MAKEOPTS and set it to -j3
    • set the TMPDIR to /dev/shm/tmp/genkernel
  • You can now launch the kernel building

genkernel all

If all succeeds, you'll see these 2 files in /boot: kernel* and initramfs*

Configure the filesystem

Your partitions are listed in /etc/fstab. Gentoo provides a default file that is not valid and must be modified.

Based on the partitions you made earlier, here is what it should look like:

/dev/sda2   /boot        ext2    defaults,noatime     0 2
/dev/sda3   none         swap    sw                   0 0
/dev/sda4   /            ext4    noatime,discard      0 1

shm         /dev/shm     tmpfs   nodev,nosuid         0 0

Note the discard option for our root partition: since it's on an SSD and the fs is ext4, this will enable the TRIM command.

Also: there is no cdrom drive on this laptop, so no need to write an entry in the fstab.

Configure the network

First, define your hostname in /etc/conf.d/hostname

While you are at it, modify your /etc/hosts to fill your host name:

127.0.0.1    *your_host_name* localhost

Now, you will install and configure wpa_supplicant to enable wireless networking.

  • install wpa_supplicant (with some other wireless utilities) by typing emerge net-wireless/wpa_supplicant net-wireless/wireless-tools net-wireless/iw

  • as you did previously, edit the /etc/wpa_supplicant/wpa_supplicant.conf file:

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=wheel

network={
  ssid="simple"
  psk="very secret passphrase"
}
  • now edit the /etc/conf.d/net file:
config_wlo1="dhcp" # wireless interface for this laptop. Run ifconfig to see yours.
config_enp0s25="dhcp" # lan interface for this laptop. Run ifconfig to see yours.
  • and to automatically start networking at boot, type the following commands:
cd /etc/init.d
ln -s net.lo net.wlo1
ln -s net.lo net.enp0s25
rc-update add net.wlo1 default
rc-update add net.enp0s25 default
  • for linux to be able to start the wireless interface, you need to install a microcode for it. This laptop needs the sys-firmware/iwl6005-ucode ebuild:
emerge sys-firmware/iwl6005-ucode
  • finally, don't forget to install a dhcp client by typing emerge dhcpcd

Configure keymap and hwclock

Some misc config now:

  • edit the /etc/conf.d/keymaps file to reflect your keyboard
  • edit the /etc/conf.d/hwclock file and set it to "UTC" "or "local" depending on your BIOS setting. If you came from or dual-boot with Windows, it will be "local".

Change the root password

Just type passwd. This is the root password. You know what it means.

Install some necessary system tools

  • metalog will be your system logger. To install it, type emerge metalog && rc-update add metalog default.
  • the cron daemon will be cronie. Type emerge cronie && rc-update add cronie default
  • install a NTP daemon: emerge ntp && rc-update add ntpd default
  • install ACPI (power button, fans, battery...): emerge acpid && rc-update add acpid default
  • install some useful utils: emerge gentoolkit portage-utils iproute2
  • want to use your mouse in the console? Install gpm : emerge gpm && rc-update add gpm default
  • to run SSH at boot: rc-update add sshd default

Install a bootloader

  • install GRUB2: emerge sys-boot/grub
  • now install the necessary GRUB2 files in your boot disk: grub2-install /dev/sda
  • generate the GRUB2 configuration: grub2-mkconfig -o /boot/grub/grub.cfg

Reboot

Before you can reboot, you have to do some cleaning:

  • type exit to exit the chroot
  • get back to root by typing cd
  • delete the stage3 and portage archives: rm /mnt/gentoo/stage3-* && rm /mnt/gentoo/portage-latest*
  • unmount all the partitions
umount -l /mnt/gentoo/dev{/shm,/pts,}
umount -l /mnt/gentoo{/boot,/proc,/sys,}
  • and now, unplug your USB drive and type reboot

You now have a working Gentoo distribution installed on your laptop. It's time to configure it further.

Install X

  • Install the X11 server:
emerge xorg-server

You will now configure some settings, like the keyboard layout or the touchpad functions.

First, if it does not exist already, create the /etc/X11/xorg.conf.d directory.

If you don't want to use the default (US) keyboard layout, create the /etc/X11/xorg.conf.d/10-keyboard.conf file, containing the following:

Section "InputClass"
    Identifier      "Internal Keyboard"
    MatchIsKeyboard     "True"
    Option "XkbLayout"  "fr" # or whatever your layout is
EndSection

This laptop has a synaptics touchpad, so create the /etc/X11/xorg.conf.d/50-synaptics.conf file, containing:

Section "InputClass"
    Identifier              "Touchpad"
    Driver                  "synaptics"
    MatchIsTouchpad         "True"
    Option "VertEdgeScroll" "off"
    Option "TapButton1"     "1"
    Option "TapButton2"     "2"
    Option "TapButton3"     "3"
    Option "VertTwoFingerScroll"    "1"
    Option "HorizTwoFingerScroll"   "1"
EndSection

You can find the list of available options in the man synaptics page.

  • Install a display manager: SLiM
emerge x11-misc/slim

Then set SLiM as your default display manager by setting DISPLAYMANAGER="slim" in /etc/conf.d/xdm

You'll need to modify the /etc/slim.conf file to set the login_cmd to the line login_cmd exec /bin/bash -login ~/.xinitrc %session and comment the 2 others.

To start SLiM at boot, add xdm to your default runlevel: rc-update add xdm default

Debug tip for later: if when you login with SLiM, it seems that no Window Manager is loading, type: rc-update add dbus default

  • Install i3
emerge x11-wm/i3 i3status

Then edit your ~/.xinitrc to be just: exec i3

  • Install a terminal
emerge terminator

Verify the webcam

To test the webcam, use mplayer (emerge it before):

mplayer tv:// -tv driver=v4l2:device=/dev/video0

If you are in an X session, your video should show. Otherwise, you should see a frame counter rolling. That's good.

Get sound

Install alsa-utils

emerge alsa-utils && rc-update add alsasound default

If you don't get any sound, launch alsamixer to unmute (type m)

That's it... for now!

Everything should work now - mostly. Of course, there is still some tweaking left!

In an upcoming post, I'll detail the exact config files I used for this hardware, including for the kernel.

PiBeacon - Making an iBeacon from a Raspberry Pi

When I first heard about iBeacons, I thought it would be a great idea to build one. It seemed doable for an ex-developer-would-like-to-code-again-but-doesn-t-have-a-lot-of-spare-time like me.

So I grabbed my Rapsberry Pi, and found a couple of articles describing how to make an iBeacon out of it. Unfortunately, none would tell exactly what I wanted: a simple service launched at boot, easy enough to configure, with no complicated C code.

The first article I read is from Radius Networks. They seem to have developped a comprehensive platform enabling marketers to do a lot of location-based things in their apps. The post is almost exactly what I was looking for, except that for some reason their Bluetooth commands didn't work with my adapter.

In the second article from Carson McDonald, the iBeacon part is a C program. It worked instantly with my adapter, but I didn't want to dive into C code to maintain it, should I need/want to. So I kind of reverse-engineered the C program with a bluetooth debugger, to see exactly what commands were transmitted to the adapter.

Combining thes two articles, I was able to determine the commands compatible with my adapter and put together a fully autonomous PiBeacon.

This is what worked for me. Everything is on my github repo.

Shopping list

  • a Raspberry Pi. I had the B model in hand, but the A model should work just the same.
  • an SD Card for the system. Any will do as long as it's at least 4GB.
  • a Bluetooth 4.0 dongle. I chose the Inatek BTA-CSR4B5 because it's tiny, affordable and its chipset (CSR8510) is known to work well with Linux. You can find a list of tested Bluetooth adapters at eLinux.org.

Installation

Raspbian

Nothing fancy here, download the official Wheezy Raspbian, and write it on the SD Card.

Plug the Bluetooth dongle, boot the raspi and log in.

Because you don't want outdated software on your system, upgrade everything that needs to:

sudo apt-get update && sudo apt-get upgrade -y

(Also want a fresh firmware/kernel ? See rpi-update)

The BlueZ stack

Go to the BlueZ site to know what the latest version is and replace it in the following commands on your raspi:

wget https://www.kernel.org/pub/linux/bluetooth/bluez-5.14.tar.xz
tar xvJf bluez-5.14.tar.xz
cd bluez-5.14
./configure --disable-systemd --enable-library
make
make install

Verify that your Bluetooth dongle is recognized: type hciconfig to see if your device is listed. It will most probably be named hci0. If you don't see "UP RUNNING", type sudo hciconfig hci0 up.

Scripts

Now that the Raspi is fully installed, we can transform it into an iBeacon.

We only use programs that are part of the BlueZ stack: hciconfig and hcitool.

There are three parts:

  • a start script, to start the iBeacon advertising
  • a stop script, to stop advertising
  • a config file, used by both scripts

For convenience, create a directory named iBeacon in /home/pi, and put these three files there.

The config file

Since you could have several iBeacons in the same place, this file contains the values enabling us to identify an iBeacon.

Name it ibeacon.conf

#all values must be in hex form separated by spaces between every two hex digits
export BLUETOOTH_DEVICE=hci0
export UUID="e2 c5 6d b5 df fb 48 d2 b0 60 d0 f5 a7 10 96 e0"
export MAJOR="00 16"
export MINOR="00 08"
export POWER="c5"

You can change major/minor values at your linking.

The start script

Name it ibeacon_start

#!/bin/sh
. ./ibeacon.conf
echo "Launching virtual iBeacon..."
sudo hciconfig $BLUETOOTH_DEVICE up
sudo hciconfig $BLUETOOTH_DEVICE noleadv
sudo hciconfig $BLUETOOTH_DEVICE leadv 0
sudo hcitool -i hci0 cmd 0x08 0x0008 1e 02 01 1a 1a ff 4c 00 02 15 $UUID $MAJOR $MINOR $POWER 00
echo "Complete"

The stop script

Name it ibeacon_stop

#!/bin/sh
. ./ibeacon.conf
echo "Disabling virtual iBeacon..."
sudo hciconfig $BLUETOOTH_DEVICE noleadv
echo "Complete"

Test

Type ./ibeacon_start

The iBeacon should start advertising (broadcasting) its identifiers.

You can verify that the iBeacon is running by using an app on your Android or iOS device:

Of course, your device has to be Bluetooth 4.0 capable.

You can stop your iBeacon with the following command: ./ibeacon_stop

Making it a service

Put this script in a file named ibeacon_install_service.sh:

#! /bin/bash
# check root permissions
if [[ $UID != 0 ]]; then
  echo "Please start the script as root or sudo!"
  exit 1
fi

apt-get install bluez -y

sed "s:/home/pi/iBeacon:$(dirname $(readlink -f $0)):" > /etc/init.d/ibeacon << "EOF"
#!/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:$PATH

DESC="iBeacon Application Software"
PIDFILE=/var/run/ibeacon.pid
SCRIPTNAME=/etc/init.d/ibeacon

case "$1" in
  start)
    printf "%-50s" "Starting ibeacon..."
    cd /home/pi/iBeacon
    ./ibeacon_start
    ;;
  stop)
    printf "%-50s" "Stopping ibeacon..."
    cd /home/pi/iBeacon
    ./ibeacon_stop
    ;;
  restart)
    $0 stop
    $0 start
    ;;
  *)
    echo "Usage: $0 {start|stop|restart}"
    exit 1
esac
EOF

chmod +x /etc/init.d/ibeacon

update-rc.d -f ibeacon start 80 2 3 4 5 . stop 30 0 1 6 .

Now sudo ibeacon_install_service.sh to install the ibeacon service.

The ibeacon service starts automatically when the raspi boots. You can also start/stop it manually by typing sudo /etc/init.d/ibeacon start and sudo /etc/init.d/ibeacon stop.

PiBeacon - done

Congratulations, you now have a fully autonomous PiBeacon!

If you want to know a little more about iBeacons and what you can do with it, you can read this post by Dave Addey.

First blog, first post

This blog and this post are long overdue. I thought about starting blogging several times in the past ten years. And the time is now.

Why now

I've bought a Raspberry Pi when it was announced. Like many of my other geeky stuff, I've turned it on for a day or two, and then it has collected dust in a drawer.

But one day, I wanted to hack on a particular project. Because it's easily portable and doesn't draw too much electricity, my raspi was a great platform to build on. (if you're curious about what this project, it's an iBeacon and will soon be described on this blog)

Since this first project, I've found other stuff to hack about, regarding one of my hobbies: home automation.

Home automation

This domain has changed a lot these past years, largely due to the opening of communication protocols (X10, Z-Wave, 433MHz) and the emergence of small hacking platforms like the Arduino, the Raspberry Pi and the Beagle Board.

Being able to watch and control parts of your house is now at your fingertips -- literally, with a smartphone.

This is mainly what I'll be writing about: my attempts to know better and manage my home from afar.

About that name

Er, "hack and cheese", srsly ?

I could tell you that I love to hack and I love cheese. It's true. But no. I just love cheesy puns.