Failverse

Using Temporary URLs on Rackspace Cloud Files

Ryuujinx — Mon, 16 Apr 2012 19:41:43 +0000

The ability to have temporary URLs is something that has been requested for quite some time. Since Cloud Files runs on openstack swift, we get some of the new features available, such as temporary URLs. This will allow you to create temporary URLs so your end users can consume your product, while having it expire after a few days, or minutes, so it isn’t possible to share the link, limiting unauthorized downloads.

Keep in mind, this isn’t officially supported by Rackspace yet, so you’re on your own for it, however it does work and you can start taking advantage of it now. Read on after the jump for the how-to.

The first thing we need(Aside of an account), is the API key. You can locate this in your control panel under ‘Your Account’, if you’re using Cloud Files, you more then likely already have this. Once you have your API key, we need to create authenticate and create/set our key.

Let’s go ahead and authenticate first:

$ curl -I -X GET -H X-Auth-User:USER -H X-Auth-Key:APIKEY https://auth.api.rackspacecloud.com/v1.0
HTTP/1.1 204 No Content
Server: Apache/2.2.3 (Red Hat)
vary: X-Auth-Token,X-Auth-Key,X-Storage-User,X-Storage-Pass
X-Storage-Url: https://storage101.dfw1.clouddrive.com/v1/MossoCloudFS_d58d266b-3601-4d95-8e96-2063c9caef48
Cache-Control: s-maxage=5028
Content-Type: text/xml
Date: Mon, 16 Apr 2012 18:41:46 GMT
X-Auth-Token: ~
X-Storage-Token: ~
X-Server-Management-Url: https://servers.api.rackspacecloud.com/v1.0/473948
Connection: Keep-Alive
X-CDN-Management-Url: https://cdn.clouddrive.com/v1/MossoCloudFS_d58d266b-3601-4d95-8e96-2063c9caef48
Content-Length: 0

Next, let’s go ahead and make a key. It can be whatever you want (Alphanumeric) within the limitations of the header length, however I recommend just echoing out some text and making an md5 or sha of it.

$ echo -n "Random text so I can make my key with it and such" | md5sum
7acc0fe42358c0232053b3fc7254609c  -

Once we have this, we can go ahead and set it on our account:

$ curl -i -X POST -H X-Auth-Token:~token -H X-Account-Meta-Temp-URL-Key:7acc0fe42358c0232053b3fc7254609c https://storage101.dfw1.clouddrive.com/v1/MossoCloudFS_d58d266b-3601-4d95-8e96-2063c9caef48
HTTP/1.1 204 No Content
Content-Length: 0
Content-Type: text/html; charset=UTF-8
X-Trans-Id: txd69ae1123b4d4f8c82ab1a38bfb3a499
Date: Mon, 16 Apr 2012 19:02:22 GMT

A few notes about your key, if you change it, it will invalidate every URL you have generated after about a minute (Due to caching), additionally, you can check it by issuing a HEAD against your account, so you won’t have to invalidate all your old URLS if you forget it -

$ curl -I -H X-Auth-Token:TOKEN https://storage101.dfw1.clouddrive.com/v1/MossoCloudFS_d58d266b-3601-4d95-8e96-2063c9caef48
HTTP/1.1 204 No Content
X-Account-Object-Count: 52
X-Account-Meta-Temp-Url-Key: 7acc0fe42358c0232053b3fc7254609c
X-Account-Bytes-Used: 61839518344
X-Account-Container-Count: 2
Accept-Ranges: bytes
Content-Length: 0
X-Trans-Id: txa90bbf3e259443d5bd4c312adddc1767
Date: Mon, 16 Apr 2012 19:31:12 GMT

After you’ve set up your key on your account, we can go ahead and generate the URL using a python script:

import hmac
from hashlib import sha1
from time import time
method = 'GET'
expires = int(time() + TIME FROM NOW TO EXPIRE, IN SECONDS)
base = 'BASEURL'
path = 'PATH TO OBJECT'
key = 'THE KEY YOU CREATED'
hmac_body = '%s\n%s\n%s' % (method, expires, path)
sig = hmac.new(key, hmac_body, sha1).hexdigest()
print '%s%s?temp_url_sig=%s&temp_url_expires=%s' % (base, path, sig, expires)

For comparison, here is what mine looked like:

import hmac
from hashlib import sha1
from time import time
method = 'GET'
expires = int(time() + 600)
base = 'https://storage101.dfw1.clouddrive.com'
path = '/v1/MossoCloudFS_d58d266b-3601-4d95-8e96-2063c9caef48/test/test.txt'
key = '7acc0fe42358c0232053b3fc7254609c'
hmac_body = '%s\n%s\n%s' % (method, expires, path)
sig = hmac.new(key, hmac_body, sha1).hexdigest()
print '%s%s?temp_url_sig=%s&temp_url_expires=%s' % (base, path, sig, expires)

Now that we have the script made, we can generate our URL and give it to end users:

$ python script.py

https://storage101.dfw1.clouddrive.com/v1/MossoCloudFS_d58d266b-3601-4d95-8e96-2063c9caef48/test/test.txt?temp_url_sig=d36692ce16ff22c80769dfce0712ff6fc9e4dd6c&temp_url_expires=1334603603

$ curl -I "https://storage101.dfw1.clouddrive.com/v1/MossoCloudFS_d58d266b-3601-4d95-8e96-2063c9caef48/test/test.txt?temp_url_sig=d36692ce16ff22c80769dfce0712ff6fc9e4dd6c&temp_url_expires=1334603603"
HTTP/1.1 200 OK
Last-Modified: Fri, 18 Nov 2011 23:32:41 GMT
Accept-Ranges: bytes
Etag: 8d777f385d3dfec8815d20f7496026dc
Content-Type: text/plain
Content-Length: 4
X-Trans-Id: txf8297b7c83d44a85bcb9d28c7141ca08
Date: Mon, 16 Apr 2012 19:03:41 GMT

Again, this exists in the Swift implementation for Rackspace Cloud, however it is not officially supported yet. If you have issues, you’re pretty much on your own. However it seems to work fine, and it will probably be in the documentation soon and officially supported, so you can start utilizing it now!

Creating the Kittynet

Ryuujinx — Sun, 05 Feb 2012 20:44:21 +0000

Have you ever wanted to just set up a wireless network that replaces pictures with cats?

No? What’s wrong with you. In this article we’re going to talk about making the Kittynet. So you can leave it unsecured and have your neighbors be annoyed by pictures of cats everywhere, like so -

(note: don’t go to that site.)

Setting up an Nginx Reverse Proxy

Ryuujinx — Wed, 18 Jan 2012 01:03:15 +0000

You might have noticed by now that I really like Nginx. I’ve had an article for using it as a load balancer with SSL termination already, and eventually I’ll get around to setting it up as a web server as well. For this article, I’ll set a situation. You have some PHP application on apache, and have your mod_rewrite wizardry the way you want it, and everything is working -ok-. You’ve heard of this new Nginx thing and want to give it a shot, but don’t want to mess with testing everything on Nginx. So what you can do, is have Nginx listen and serve all your static content (Which it’s really good at), and pass your dynamic content(and whatever else) back to Apache to process. This article will go over the configuration of an Nginx reverse proxy, and modifying apache to work with it.

The first thing you need to do is install it. On Debian, this is quite simply -

Add the following to /etc/apt/sources.list

deb http://nginx.org/packages/debian/ squeeze nginx
deb-src http://nginx.org/packages/debian/ squeeze nginx

Now install it:

# aptitude update
# aptitude install nginx

Here’s some copy/pasta if you just want the configs:

/etc/nginx/nginx.conf

user  nginx;
worker_processes  6;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    keepalive_timeout  10;

    #Compression Settings
    gzip on;
    gzip_http_version 1.0;
    gzip_comp_level 2;
    gzip_proxied any;
    gzip_min_length  1100;
    gzip_buffers 16 8k;
    gzip_types text/plain text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript;
    # Some version of IE 6 don't handle compression well on some mime-types,
    # so just disable for them
    gzip_disable "MSIE [1-6].(?!.*SV1)";
    # Set a vary header so downstream proxies don't send cached gzipped
    # content to IE6
    gzip_vary on;

    include /etc/nginx/conf.d/*.conf;

}

The first bit is pretty self explanatory. We set the user, the number of processes nginx will spawn, where it’s going to store the pid file, and the number of connections each worker will get. Moving down, we have an include(mime.types sets a lot of the MIME types), and set the default MIME type. Moving further down, we set the log format and allow sendfile (which bypasses using read() and write(), you can read about it some here and here.) We set a keepalive_timeout fairly low (it could probably stand be shorter, but 10 seconds should be fine).

The compression settings should be fairly obvious, if you want to read on the specifics on a parameter you can read about them here.

Next, we need to set up the actual proxying:

/etc/nginx/conf.d/proxy.conf

server {

	listen 80;

	access_log off;
	error_log off;

	location / {
		proxy_pass	http://127.0.0.1:8080;
		proxy_redirect	off;	

		#Proxy Settings
		proxy_redirect     off;
		proxy_set_header   Host             $host;
		proxy_set_header   X-Real-IP        $remote_addr;
		proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
		proxy_max_temp_file_size 0;
		proxy_connect_timeout      90;
		proxy_send_timeout         90;
		proxy_read_timeout         90;
		proxy_buffer_size          4k;
		proxy_buffers              4 32k;
		proxy_busy_buffers_size    64k;
		proxy_temp_file_write_size 64k;
	}

	location ~* \.(?:ico|css|js|gif|jpe?g|png|bmp|html) {
		root /var/www;
		expires max;
		add_header Pragma public;
		add_header Cache-Control "public, must-revalidate, proxy-revalidate";
	}
}

In this section, we do a few things. First we define a server. A server in the context of nginx can be loosely defined as a virtualhost. For instance, if you wanted to have domain1.com and domain2.com, you would end up with two server blocks. In this case, we’re going to specify a server with no server name. If you are going to host multiple domains, you will need to make a .conf for each and set server_name as well as root for each domain unless you don’t want nginx to serve static content for some reason.

The next thing we do is specify two locations. The first specifies anything that matches /, and anything that matches something ending in .ico/css/etc. Nginx matches the most specific location. In this instance, that means /somedir/image.png will match the second rule, as matching “.png” is more specific then matching / (which will match every request to the server). In the ~* \. match, we serve the content with nginx directly – this is because nginx is very good at serving static content and there is no reason to proxy the requests to apache. We add a couple headers (for instance, telling the browser to cache the content as long as possible) and let it go on its way. The other location proxies the request to apache, setting some headers (such as X-Real-IP and X-Forwarded-For), and set the timeouts to the backend (90 seconds) and the buffer sizes. I will refer you (again) to the excellent documentation if you would like to know what all parameters are available, located here.

Finally, we need to change apache to listen on port 8080 instead of 80. The apache configurations vary significantly between distributions, so I won’t point to the specific files, but a grep -Ri should get you what you need.

The configurations we are concerned about are VirtualHost, NameVirtualHost and Listen.

You will need to modify them so they look like this:

NameVirtualHost *:8080
Listen 8080

Once you have done this, check your configurations:

# service nginx configtest
# service apache configtest

(note that it is httpd on redhat based systems), assuming everything checks out, restart apache, start nginx and set it to start on boot -

# service apache2 restart
# service nginx start
# update-rc.d nginx enable

Assuming everything went well, Nginx should now be solving all static content and relaying everything else back to apache. Enjoy your performance boost!

Net-install a custom linux distro on Cloud Servers

Deuce — Sun, 02 Oct 2011 02:24:22 +0000

This guide will walk you through installing a custom linux distro to Rackspace Cloud Servers without the need of taring up a file system from a donor box. This particular guide is specific to openSuse, but the same method can be used to install other distros that support automated/remote install.

This process is entirely unsupported by Rackspace.

First, some documentation before we get started:
http://en.opensuse.org/SDB:Remote_installation
http://en.opensuse.org/SDB:Linuxrc

Spin up a box on the cloud. I went with Debian for this step, but you can use whichever variant of linux you please. First thing to do is to gather up your networking info: IPs, netmasks, gateway, static routes, and DNS servers (on Debian, just cat /etc/network/interfaces and /etc/resolv.conf for all of the info you need). Make a /boot and /boot/grub, and move to /boot. Now we need to grab the xen-compatible installer kernel and initrd:

/boot# wget http://download.opensuse.org/distribution/11.4/repo/oss/boot/x86_64/vmlinuz-xen
/boot# wget http://download.opensuse.org/distribution/11.4/repo/oss/boot/x86_64/initrd-xen

Now edit /boot/grub/menu.lst and make an entry for openSuse. We’re going to want it to look like this (fill in the network config with what you pulled previously):

timeout 5
default 0
title openSuse Install
      root (hd0)
      kernel /boot/vmlinuz-xen textmode=1 noapic ssh=1 sshpassword="yoursshpassword" addswap=/dev/xvda2 install=http://download.opensuse.org/distribution/11.4/repo/oss/ hostip=serverip netmask=servernetmask gateway=servergateway nameserver=serverdnsserver
      initrd /boot/initrd-xen

Submit a ticket to Rackspace support to enable PV-GRUB for you. Go ahead, I’ll wait.

If all goes well, you should see the box pinging after they switch it over to PV-GRUB (if all did not go well, kick the box into rescue mode and double check your work). SSH into it using user root and the password that you set in the “sshpassword” kernel option. Once you’re at the shell, run “yast”. This will launch the text based installer.

On the Installation Mode screen, un-select “Use Automatic Configuration”. On the Desktop Selection screen, select “Other” -> “Minimal Server Selection (Text Mode)”. On the “Suggested Partitioning” screen, select “Edit Partition Setup…”. On the next screen tab to /dev/xvda1 and press enter. Then tab to “Edit” and hit enter. Match these settings (NOTE: Only use EXT3! PV-GRUB freaked out when I tried EXT4):

  Edit Partition /dev/xvda1
                        ┌Formatting Options───────────┐
                        │ (x) Format partition        │
                        │     File System             │
                        │     Ext3                    │
                        │     [     Options...      ] │
                        │ ( ) Do not format partition │
                        │     File system ID:         │
                        │     0x83 Linux              │
                        │ [ ] Encrypt device          │
                        └─────────────────────────────┘
                        ┌Mounting Options─────────────┐
                        │ (x) Mount partition         │
                        │     Mount Point             │
                        │     /                       │
                        │     [  Fstab Options...   ] │
                        │ ( ) Do not mount partition  │
                        └─────────────────────────────┘

Then hit “Finish”, then “Accept”. Continue on with creating your user, uncheck “Automatic Login” (not sure how that would work for a server….). I also unchecked “Use this password for system administrator” since I wanted to set a password for root. On the Installation Settings screen, be sure to enable SSH and open it in the firewall. Now hit Install. It should chug along just fine until it gets to installing grub. Don’t bother to retry installing the boot loader, just continue along. Stop it from rebooting and open up a second SSH connection to the box and run the following commands:

~# mount /dev/xvda1 /mnt/
~# sed -i 's/hd0,0/hd0/' /mnt/boot/grub/menu.lst
~# umount /mnt/

Now it is safe to reboot. Try to SSH to the box when it starts responding to ping. Congrats, you’ve just installed openSuse on Cloud Servers the less ghetto way =P.

Creating a Non-standard Size Bootable Floppy Image for PXE Boot

Deuce — Sun, 07 Aug 2011 01:32:18 +0000

The majority of motherboard manufactures still only allow you to update your BIOS either from within Windows, from a USB stick within the BIOS itself, or from a floppy with DOS. The first option doesn’t work with a linux box for obvious reasons. While the second option is nice for updating one box, it quickly becomes a hassle when you have an entire rack you need to update. And the third option is antiquated by any meaning of the word… or is it? While the days of floppies are long gone, the reign of the floppy image is still going strong in the world of PXE boot.

The biggest limitation of a floppy image is easily its size. 1.44MB is almost useless in today’s world of terabyte hard drives. Since BIOS images take up about 1MB, that leaves room for not much else. Meaning no scripting, no fancy menus, just the flasher program and your BIOS image. One of the most common methods around this limitation is to offsite your BIOS images to a samba share and instead use the 1.44MBs of space for network utilities. While this does work fine, it brings back bad memories of networking in DOS that I’d rather not experience again. Instead, I’ll walk you through how to expand a floppy image to whatever size is comfy for you, and most importantly, keep it bootable.

Things you will need:

A bootable floppy image (http://www.bootdisk.com/ has a bunch)
dd
mkfs.msdos
hexdump, for verification purposes
A PXELINUX PXE boot environment

1. Procure a bootable floppy image

Download your bootable floppy image and save it to a directory of your choosing. If your downloaded it from http://www.bootdisk.com/, chances are it’s saved as a Windows executable. Have no fear, just run unzip on it and it should extract the tasty image within:

david@bt:~$ wget http://www.dq.com.pl/pliki/boot%20disk/boot98c.exe
 --2011-08-06 16:52:31--  http://www.dq.com.pl/pliki/boot%20disk/boot98c.exe
 Resolving www.dq.com.pl... 85.128.137.63
 Connecting to www.dq.com.pl|85.128.137.63|:80... connected.
 HTTP request sent, awaiting response... 200 OK
 Length: 864362 (844K) [application/x-msdownload]
 Saving to: `boot98c.exe'
100%[======================================>] 864,362      529K/s   in 1.6s
2011-08-06 16:52:33 (529 KB/s) - `boot98c.exe' saved [864362/864362]
david@bt:~$ unzip boot98c.exe
 Archive:  boot98c.exe
 warning [boot98c.exe]:  105508 extra bytes at beginning or within zipfile
 (attempting to process anyway)
 file #1:  bad zipfile offset (local header sig):  211016
 (attempting to re-compensate)
 inflating: boot98c.IMA
 david@bt:~$ ls
 boot98c.exe  boot98c.IMA

2. Create a new floppy image with the size you want

Now that we have our donor bootable floppy image with its delicious boot code, we need to create an empty floppy image with the new size. First, figure out how much space you need. For this tutorial I’m going with 10MB. Convert that value to KB (you can do this from within bash by running “expr $((sizeinMB << 10))”), in my case that’s 10240. This is going to be the last argument we use in mkfs.msdos:

david@bt:~$ expr $((10 << 10))
10240
david@bt:~$ mkfs.msdos -I -v -C test.img 10240
mkfs.msdos 3.0.3 (18 May 2009)
test.img has 64 heads and 32 sectors per track,
logical sector size is 512,
using 0xf8 media descriptor, with 20480 sectors;
file system has 2 16-bit FATs and 4 sectors per cluster.
FAT size is 20 sectors, and provides 5101 clusters.
Root directory contains 512 slots.
Volume ID is df50ef01, no volume label.
david@bt:~$ du -h test.img
52K     test.img

You may notice that the file size is only 52K. This is normal. This command just created the filesystem’s metadata. You can still mount it and the OS will see it as 10MB of space. Take note of the sector size (512 in this example), amount of heads (64), and sectors per track (32). You will need these later to calculate the amount of cylinders.

3. Copy over boot information to new image

Let’s compare the boot sector from the bootable image to the one we just created:

david@bt:~$ hexdump -C -n 512 boot98c.IMA
00000000  eb 3c 90 2a 34 68 55 3c  49 48 43 00 02 01 01 00  |.<.*4hU8-t.`|
000000d0  b1 0b be d8 7d f3 a6 61  74 3d 4e 74 09 83 c7 20  |....}..at=Nt... |
000000e0  3b fb 72 e7 eb dd fe 0e  d8 7d 7b a7 be 7f 7d ac  |;.r......}{...}.|
000000f0  98 03 f0 ac 98 40 74 0c  48 74 13 b4 0e bb 07 00  |.....@t.Ht......|
00000100  cd 10 eb ef be 82 7d eb  e6 be 80 7d eb e1 cd 16  |......}....}....|
00000110  5e 1f 66 8f 04 cd 19 be  81 7d 8b 7d 1a 8d 45 fe  |^.f......}.}..E.|
00000120  8a 4e 0d f7 e1 03 46 fc  13 56 fe b1 04 e8 c2 00  |.N....F..V......|
00000130  72 d7 ea 00 02 70 00 52  50 06 53 6a 01 6a 10 91  |r....p.RP.Sj.j..|
00000140  8b 46 18 a2 26 05 96 92  33 d2 f7 f6 91 f7 f6 42  |.F..&...3......B|
00000150  87 ca f7 76 1a 8a f2 8a  e8 c0 cc 02 0a cc b8 01  |...v............|
00000160  02 80 7e 02 0e 75 04 b4  42 8b f4 8a 56 24 cd 13  |..~..u..B...V$..|
00000170  61 61 72 0a 40 75 01 42  03 5e 0b 49 75 77 c3 03  |aar.@u.B.^.Iuw..|
00000180  18 01 27 0d 0a 49 6e 76  61 6c 69 64 20 73 79 73  |..'..Invalid sys|
00000190  74 65 6d 20 64 69 73 6b  ff 0d 0a 44 69 73 6b 20  |tem disk...Disk |
000001a0  49 2f 4f 20 65 72 72 6f  72 ff 0d 0a 52 65 70 6c  |I/O error...Repl|
000001b0  61 63 65 20 74 68 65 20  64 69 73 6b 2c 20 61 6e  |ace the disk, an|
000001c0  64 20 74 68 65 6e 20 70  72 65 73 73 20 61 6e 79  |d then press any|
000001d0  20 6b 65 79 0d 0a 00 00  49 4f 20 20 20 20 20 20  | key....IO      |
000001e0  53 59 53 4d 53 44 4f 53  20 20 20 53 59 53 7f 01  |SYSMSDOS   SYS..|
000001f0  00 41 bb 00 07 60 66 6a  00 e9 3b ff 00 00 55 aa  |.A...`fj..;...U.|
00000200
david@bt:~$ hexdump -C -n 512 test.img
00000000  eb 3c 90 6d 6b 64 6f 73  66 73 00 00 02 04 01 00  |.<.mkdosfs......|
00000010  02 00 02 00 50 f8 14 00  20 00 40 00 00 00 00 00  |....P... .@.....|
00000020  00 00 00 00 00 00 29 01  ef 50 df 20 20 20 20 20  |......)..P.     |
00000030  20 20 20 20 20 20 46 41  54 31 36 20 20 20 0e 1f  |      FAT16   ..|
00000040  be 5b 7c ac 22 c0 74 0b  56 b4 0e bb 07 00 cd 10  |.[|.".t.V.......|
00000050  5e eb f0 32 e4 cd 16 cd  19 eb fe 54 68 69 73 20  |^..2.......This |
00000060  69 73 20 6e 6f 74 20 61  20 62 6f 6f 74 61 62 6c  |is not a bootabl|
00000070  65 20 64 69 73 6b 2e 20  20 50 6c 65 61 73 65 20  |e disk.  Please |
00000080  69 6e 73 65 72 74 20 61  20 62 6f 6f 74 61 62 6c  |insert a bootabl|
00000090  65 20 66 6c 6f 70 70 79  20 61 6e 64 0d 0a 70 72  |e floppy and..pr|
000000a0  65 73 73 20 61 6e 79 20  6b 65 79 20 74 6f 20 74  |ess any key to t|
000000b0  72 79 20 61 67 61 69 6e  20 2e 2e 2e 20 0d 0a 00  |ry again ... ...|
000000c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|
00000200

For those who are interested, you can see what each offset means at Wikipedia.

As you can see, the disk we made is missing a bunch of information (the asterisk means bunches of zeros). We want to copy over all of the boot instructions, but we don’t want to overwrite the geometry information so we can retain our 10MB image. The jump instructions and the OEM data are located in the first 11 bytes (offset 00 to 0A in hexadecimal). We want to copy that from the bootable image to the new one. From offset 0B to offset 3E we have the geometry information, which we want to retain on the new image. From offset 3E to offset 200, we have the actual OS boot code. We obviously want to copy that over to the new image. To make all of these changes we bust out dd:

david@bt:~$ dd if=boot98c.IMA of=test.img bs=1 count=11 conv=notrunc
11+0 records in
11+0 records out
11 bytes (11 B) copied, 0.000163496 s, 67.3 kB/s
david@bt:~$ dd if=boot98c.IMA of=test.img bs=1 skip=62 seek=62 conv=notrunc count=450
450+0 records in
450+0 records out
450 bytes (450 B) copied, 0.0491807 s, 9.1 kB/s

The reason we only counted to 450 is because 512-62=450. Now let’s compare the two boot sectors:

david@bt:~$ hexdump -C -n 512 boot98c.IMA                                      
00000000  eb 3c 90 2a 34 68 55 3c  49 48 43 00 02 01 01 00  |.<.*4hU8-t.`|
000000d0  b1 0b be d8 7d f3 a6 61  74 3d 4e 74 09 83 c7 20  |....}..at=Nt... |
000000e0  3b fb 72 e7 eb dd fe 0e  d8 7d 7b a7 be 7f 7d ac  |;.r......}{...}.|
000000f0  98 03 f0 ac 98 40 74 0c  48 74 13 b4 0e bb 07 00  |.....@t.Ht......|
00000100  cd 10 eb ef be 82 7d eb  e6 be 80 7d eb e1 cd 16  |......}....}....|
00000110  5e 1f 66 8f 04 cd 19 be  81 7d 8b 7d 1a 8d 45 fe  |^.f......}.}..E.|
00000120  8a 4e 0d f7 e1 03 46 fc  13 56 fe b1 04 e8 c2 00  |.N....F..V......|
00000130  72 d7 ea 00 02 70 00 52  50 06 53 6a 01 6a 10 91  |r....p.RP.Sj.j..|
00000140  8b 46 18 a2 26 05 96 92  33 d2 f7 f6 91 f7 f6 42  |.F..&...3......B|
00000150  87 ca f7 76 1a 8a f2 8a  e8 c0 cc 02 0a cc b8 01  |...v............|
00000160  02 80 7e 02 0e 75 04 b4  42 8b f4 8a 56 24 cd 13  |..~..u..B...V$..|
00000170  61 61 72 0a 40 75 01 42  03 5e 0b 49 75 77 c3 03  |aar.@u.B.^.Iuw..|
00000180  18 01 27 0d 0a 49 6e 76  61 6c 69 64 20 73 79 73  |..'..Invalid sys|
00000190  74 65 6d 20 64 69 73 6b  ff 0d 0a 44 69 73 6b 20  |tem disk...Disk |
000001a0  49 2f 4f 20 65 72 72 6f  72 ff 0d 0a 52 65 70 6c  |I/O error...Repl|
000001b0  61 63 65 20 74 68 65 20  64 69 73 6b 2c 20 61 6e  |ace the disk, an|
000001c0  64 20 74 68 65 6e 20 70  72 65 73 73 20 61 6e 79  |d then press any|
000001d0  20 6b 65 79 0d 0a 00 00  49 4f 20 20 20 20 20 20  | key....IO      |
000001e0  53 59 53 4d 53 44 4f 53  20 20 20 53 59 53 7f 01  |SYSMSDOS   SYS..|
000001f0  00 41 bb 00 07 60 66 6a  00 e9 3b ff 00 00 55 aa  |.A...`fj..;...U.|
00000200
david@bt:~$ hexdump -C -n 512 test.img                                         
00000000  eb 3c 90 2a 34 68 55 3c  49 48 43 00 02 04 01 00  |.<.*4hU8-t.`|
000000d0  b1 0b be d8 7d f3 a6 61  74 3d 4e 74 09 83 c7 20  |....}..at=Nt... |
000000e0  3b fb 72 e7 eb dd fe 0e  d8 7d 7b a7 be 7f 7d ac  |;.r......}{...}.|
000000f0  98 03 f0 ac 98 40 74 0c  48 74 13 b4 0e bb 07 00  |.....@t.Ht......|
00000100  cd 10 eb ef be 82 7d eb  e6 be 80 7d eb e1 cd 16  |......}....}....|
00000110  5e 1f 66 8f 04 cd 19 be  81 7d 8b 7d 1a 8d 45 fe  |^.f......}.}..E.|
00000120  8a 4e 0d f7 e1 03 46 fc  13 56 fe b1 04 e8 c2 00  |.N....F..V......|
00000130  72 d7 ea 00 02 70 00 52  50 06 53 6a 01 6a 10 91  |r....p.RP.Sj.j..|
00000140  8b 46 18 a2 26 05 96 92  33 d2 f7 f6 91 f7 f6 42  |.F..&...3......B|
00000150  87 ca f7 76 1a 8a f2 8a  e8 c0 cc 02 0a cc b8 01  |...v............|
00000160  02 80 7e 02 0e 75 04 b4  42 8b f4 8a 56 24 cd 13  |..~..u..B...V$..|
00000170  61 61 72 0a 40 75 01 42  03 5e 0b 49 75 77 c3 03  |aar.@u.B.^.Iuw..|
00000180  18 01 27 0d 0a 49 6e 76  61 6c 69 64 20 73 79 73  |..'..Invalid sys|
00000190  74 65 6d 20 64 69 73 6b  ff 0d 0a 44 69 73 6b 20  |tem disk...Disk |
000001a0  49 2f 4f 20 65 72 72 6f  72 ff 0d 0a 52 65 70 6c  |I/O error...Repl|
000001b0  61 63 65 20 74 68 65 20  64 69 73 6b 2c 20 61 6e  |ace the disk, an|
000001c0  64 20 74 68 65 6e 20 70  72 65 73 73 20 61 6e 79  |d then press any|
000001d0  20 6b 65 79 0d 0a 00 00  49 4f 20 20 20 20 20 20  | key....IO      |
000001e0  53 59 53 4d 53 44 4f 53  20 20 20 53 59 53 7f 01  |SYSMSDOS   SYS..|
000001f0  00 41 bb 00 07 60 66 6a  00 e9 3b ff 00 00 55 aa  |.A...`fj..;...U.|
00000200

Much better.

4. Copy over OS files to new disk

Now that we have our shiny new bootable floppy, we can copying over the OS from the old floppy:

david@bt:~$ mktemp -d /tmp/old.XXXXXX
/tmp/old.rvviBJ
david@bt:~$ mktemp -d /tmp/new.XXXXXX
/tmp/new.NFM4WP
david@bt:~$ sudo mount -o loop test.img /tmp/new.NFM4WP/
david@bt:~$ sudo mount -o loop boot98c.IMA /tmp/old.rvviBJ/
david@bt:~$ sudo cp -r /tmp/old.rvviBJ/* /tmp/new.NFM4WP/
david@bt:~$ ls /tmp/new.NFM4WP/
attrib.exe    chkdsk.exe    edit.com    label.exe    scandisk.exe  xcopy32.mod
autoexec.bat  command.com   edit.hlp    mem.exe      scandisk.ini  xcopy.exe
cd1.sys       config.sys    fdisk.exe   move.exe     scanreg.exe
cd2.sys       deltree.exe   format.com  mscdex.exe   smartdrv.exe
cd3.sys       diskcopy.com  himem.sys   msdos.sys    sys.com
cd4.sys       drvspace.bin  io.sys      mtmcdai.sys  xcopy32.exe
david@bt:~$ df -h
Filesystem            Size  Used Avail Use% Mounted on
*
/dev/loop0             10M  1.4M  8.7M  14% /tmp/new.NFM4WP
/dev/loop1            1.4M  1.3M   97K  94% /tmp/old.rvviBJ
david@bt:~$ sudo umount /tmp/new.NFM4WP/
david@bt:~$ sudo umount /tmp/old.rvviBJ/

5. Final configurations for PXELINUX

Remember how I told you to jot down the sector size, number of heads, and sectors per track? Good, because we need it now. To ensure that PXELINUX is aware of the geometry, we need to provide it with the number of sectors per track, amount of heads, and the amount of cylinders the image has. To calculate the amount of cylinders we use the following equation:

Cylinders = TotalNumberOfSectors / NumberOfHeads / NumberOfSectorsPerTrack

Since the sector size is 512, we can take the filesystem size in KB from earlier and just double it for the total of 20480 sectors. We then divide that by 64 heads, then divide that by 32 sectors. Our number of cylinders should come out to 10. In your pxelinux config file you’ll want to put this information in like this:

label flash_bios
      kernel memdisk
      append initrd=test.img floppy c=10 s=32 h=64

And there you go, a 10MB bootable floppy image. Hooray!

There’s a neat script that can do all that we did in this article, available here. Have fun and happy hacking!

Load balancing with SSL termination

Ryuujinx — Thu, 07 Jul 2011 03:22:46 +0000

I wrote an article a while back about load balancing with HA Proxy. If you’re wanting to do SSL, it lets you do it, but SSL will terminate on each individual webhead. This works quite well for performance, and it is designed with performance in mind. Unfortunately there are some cases where you want the SSL to terminate on the load balancer (for instance if you’re making use of the X-Forwarded-For header). This article will explain how to setup Nginx as a load balancer with SSL termination. Read on after the jump for the howto.

The first thing you’ll need, is to make sure that your webheads are configured to listen on the internal interface. It can listen on the external as well, but the load balancer is going to communicate to it over the private network.

# netstat -ntlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      2342/apache2
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      2307/sshd
tcp6       0      0 :::22                   :::*                    LISTEN      2307/sshd

0.0.0.0 signifies all available IPs, so we’re fine. If your server is set to explicitly listen on it’s public IP, you’ll need to either change it to listen on all or on the internal.

Once we have done this we need to install nginx. The modules we will be using – upstream and proxy – are both part of the core HTTP set, so we don’t need to build it any specific way. You can either grab the source and build it, or just download it from your repository -

 # aptitude install nginx

Now that we’ve got this installed, we need to make some configuration changes. First off, go ahead and create a directory to save all your SSL keys in, I’m going to use /etc/nginx/ssl, feel free to use whatever makes sense to you. Open up /etc/nginx/nginx.conf with your editor of choice and find the http section. Comment out the include for sites-enabled and then create another include, I’ll be using lb.con – again, feel free to use whatever you want.

user www-data;
worker_processes  1;

error_log  /var/log/nginx/error.log;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
    # multi_accept on;
}

http {
    include       /etc/nginx/mime.types;

    access_log	/var/log/nginx/access.log;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;
    tcp_nodelay        on;

    gzip  on;
    gzip_disable "MSIE [1-6]\.(?!.*SV1)";

    include /etc/nginx/conf.d/*.conf;
#    include /etc/nginx/sites-enabled/*;
    include /etc/nginx/lb.conf;
}

In your new config file, you’ll need to set up the server and upstream sections. Here is my basic configuration, before we get any further:

upstream backend {
	server 10.179.73.92 max_fails=3 fail_timeout=15s;
	server 10.179.73.148 max_fails=3 fail_timeout=15s;
	server 10.179.73.170 max_fails=3 fail_timeout=15s;
	server 10.179.73.197 max_fails=3 fail_timeout=15s;
}

server {
	listen 80;

	location / {
		proxy_pass http://backend;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	}
}

server {
	listen 443;
	ssl on;
	ssl_certificate /etc/nginx/ssl/server.crt;
	ssl_certificate_key /etc/nginx/ssl/server.key;

	location / {
		proxy_pass http://backend;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	}
}

“upstream” defines a pool of servers, and basically manages them. You can specify the max times they can fail before they are pulled out of the rotation (I feel 3 is sufficient, you may choose to increase this. You might also want to increase the timeouts, it should be fairly straight forward.)

The second section is what does most of the work – you need to specify the listen ports, if you have multiple SSL websites that you want to terminate, you will need a server block for each, as well as binding to a specific IP. Additionally you need to make sure the webheads are set up to work with name based virtualhosts, as that header should get based to them. In my case, I only have one SSL website so I simply specify a single listen for 443, and another for 80.

The SSL parts should be fairly straight forward, which leaves the location block – you can do some pretty cool things with this section, but for our purposes we just want to proxy everything so we use “/”. We need to tell it to proxy specifically to port 80, and use the upstream “backend”. The last part tells nginx to add the head X-Forwarded-For, which you probably want enabled.

If you want session persistence you have to enable ip_hash in your backend. Simply put “ip_hash;” in your upstream block above your servers

More then likely, you’ll want to fine tune your configuration. For that you might check out the documentation on proxy as well as upstream as nginx’s website.

As a final note, you’re traffic is sent from the load balancer to the webheads in the CLEAR. While this shouldn’t be a big problem for you in most cases (you can’t sniff the traffic unless you broadcast it for some reason), it’s something to keep in mind. The most notable time this might be a problem is on a shared network where you can’t control it, the biggest problem might be a MiTM attack – if you’re concerned about this, you can set up arptables to prevent ARP poisoning, but the best bet would be to just ask your provider. If they don’t have any kind of ARP poisoning countermeasures in place, you might think about a host with your security as a higher priority.

Two Factor Authentication Made Easy: Google-Authenticator

Ryuujinx — Thu, 30 Jun 2011 00:03:34 +0000

I’m a pretty big fan of two factor authentication, it lets you secure a server significantly without inconveniencing your users too much. I’ve used ppp-pam before, and use RSA SecurID for a few things as well, they’re great implementations. Today it came to my attention that Google had made an authenticator for Google apps account, but also made a PAM module. It works fairly close to RSA SecurID – you put in your password, after that works you give it the code that the app on your phone displays, it changes every 30 seconds or so based on it’s algorithm.

This article is going to cover how to set it up on your own Linux server. I’ll be doing this on a Debian 6 install, you may need to alter commands, so go ahead and read on after the jump for the how to.

First off, we need to install the tools to install everything. On Debian this is easily achieved with

 # aptitude install build-essential

You’ll also need mercurial to check out the code, and some libraries.

 # aptitude install mercurial libpam-dev

Additionally if you want to automate setting up your phone’s app you’ll want to install the qrencode library -

 # aptitude install libqrencode-dev

Now we have all the libraries installed, we need to download the source -

 # hg clone https://google-authenticator.googlecode.com/hg/ google-authenticator

Then after it checks it out, move to the google-authenticator/libpam directory and issue

 # make install

This will setup the module, and an application to generate your code. Let’s start with that:

# google-authenticator

This will first off display a qr code if you have libqrencode installed, alternately you can copy the URL to a browser and display the QR code. Alternately you can manually enter the information, regardless you need to open your phone’s app and get it added – the easiest method would be to use the QR code by selecting the add button, then “scan barcode”. Use your camera to focus the QR code and it will set up the account for you. Otherwise you need to set it manually using the provided information. You will need to do this for each account on the system.

After your phone is set up, look back at your terminal and answer the questions it asks. Most of them are concerning rate limiting and other security features. Once done, you need to edit two files. The first is to tell pam to use the module:

 # vim /etc/pam.d/common-auth

In this file add the following line:

 auth    required                        pam_google_authenticator.so

This tells PAM that it is a required module in order to authenticate. If you plan on using this for SSH (which you probably are), you’ll also need to allow SSH to send challenge requests:

 # vim /etc/ssh/sshd_config

Fine the Challenge Response Authentication line and set it to yes:

 ChallengeResponseAuthentication yes

Once this is done, restart SSH and open a NEW shell to test your change:

# ssh root@10.1.1.20
Password:
Verification code:
Linux Sharaa 2.6.32-5-xen-amd64 #1 SMP Thu May 19 01:16:47 UTC 2011 x86_64

Hopefully it all works and you’ll be able to enjoy your two factor authentication.

Using your own OS on Cloud Servers

Ryuujinx — Sat, 04 Dec 2010 23:38:45 +0000

I see complaints across twitter and the feedback page, as well as various blog posts about how the Rackspace Cloud doesn’t support this OS, or that OS. With the introduction of PV-Grub, you should be able to run nearly any OS you want – with a bit of work.

Keep in mind, this process is entirely unsupported by Rackspace. If your OS breaks on you, and you’re using some OS no one has heard of, they’ll be hard pressed to support it.

For this process you will need the following:

A Tar archive of the filesystem for the OS you want to use, excluding /proc, /sys and /dev
The New Cloud Server
A good understanding of the OS you want to use.

For my example, I will be using Suse 11.3, it should work the same for other OSes. Let’s go ahead and get right into it.

Step1 – making your tar image

The first thing you’ll have to do is boot into the OS you want to use. My method for this, was to use my home computer and use Virtualbox to boot into a 64bit VM, and proceeded to install Suse11.3. Once in the VM, I switched to the root user and ran

tar -cf / suse.tar --exclude "/bin" --exclude "/proc" --exclude "/sys"

While this ran, I watched some shows on my new BoxeeBox (you should get one too, they’re awesome).

A Note about VMs and 64Bit: You need a recent Processor that supports virtualization technology, VT-x , if your personal computer does not support this, see if a friend can help you out.

After it finished, I ran

scp suse.tar root@suse-staging.failverse.com:suse.tar

And let it ran. This took me at least a couple hours, due to only having a couple Mbits up on my home connection.

Step 2: Rescue Mode and Things to do!

Now that your image is finally uploaded, you will boot your server into rescue mode. This brings up a new server along side it, and emails you a temporary root password. As it uses the same IP and has different keys, you might need to delete a line out of known_hosts before you can connect.

First, you will need to mount your server’s drive. I use the /mnt directory, feel free to mount it wherever you feel.

 mount /dev/sda1 /mnt

Inside of the drive you’ll want to make backups of your tar, dev, proc and sys.

 mkdir stuff
mv /root/suse.tar stuff
mv dev stuff
mv proc stuff
mv sys styff
mv stuff /

Now delete everything from the drive.

 rm -rf /mnt/*

Now lets take the tar and extract everything into the drive.

mv /stuff/suse.tar /mnt/
tar -xvf suse.tar

After this, move your old directories back -

mv /stuff/* /mnt/

And finally, we need to edit some things.

In /etc/resolv.conf
For DFW servers:

nameserver 72.3.128.240
nameserver 72.3.128.241

for ORD servers:

nameserver 173.203.4.8
nameserver 173.203.4.9

In /etc/fstab

proc            /proc       proc    defaults    0 0
/dev/sda1       /           ext3    defaults,errors=remount-ro,noatime    0 1
/dev/sda2       none        swap    sw          0 0

You will also need to edit your interface scripts, for Suse these are found in /etc/sysconfig/network

ifcfg-eth0

BOOTPROTO='static'
IPADDR='YOURSERVERPUBLICIP'
NETMASK='255.255.255.0'
STARTMODE='auto'
USERCONTROL='no'

ifcfg-eth1

BOOTPROTO='static'
IPADDR='YOURSERVERINTERNALIP'
NETMASK='255.255.224.0'
STARTMODE='auto'
USERCONTROL='no'

After that, you will need to set up your IP routes,

My routes looked like this:

#Destination    Gateway        Mask           Device

173.203.218.0   0.0.0.0        255.255.255.0  eth0
10.177.160      0.0.0.0        255.255.224.0  eth1
10.191.192.0    10.177.160.1   255.255.192.0  eth1
10.176.0.0      10.177.160.1   255.248.0.0    eth1
default         173.203.218.1  0.0.0.0        eth0

The second two destinations seem to always be the same for eth1, the gateway for them is the first eth1 entry ending in a .1 octet. If your internal network doesn’t work after a reboot, check what the distro set your route too – the internal Ip of this server was 10.177.164.x, but it didn’t work unless I used that route.

Now that you’re route is set, make sure SSH is set to start on startup, and exit rescue mode.

If all went well, you’ll be able to SSH to your new OS and be good to go – though you may need to tweak settings here and there. If your OS needs a custom kernel, make sure your kernel and grub are configured, and go through my PV-Grub article.

Compiling kernel modules for a Rackspace Cloud Server

Coolj — Mon, 15 Nov 2010 05:00:25 +0000

To compile kernel modules on a Cloud Server you need to complete the following steps, making sure to change the kernel version and directories where appropriate (–for example, 2.6.33.5-rscloud extracts to linux-2.6.33.5, whereas 2.6.35.4-rscloud extracts to 2.6.35.4-rscloud).

Grab the patched kernel source for your kernel from the repository (check your kernel version with uname -a).
```
wget http://kernel.slicehost.com/2.6.35.4-rscloud/patched_source/2.6.35.4-rscloud.tar.gz
```

Extract the kernel to /usr/src.

sudo tar xpf 2.6.35.4-rscloud.tar.gz -C /usr/src

Create a link to /usr/src/linux and link to build in /lib/modules

sudo rm -rf /usr/src/linux
sudo ln -s /usr/src/2.6.35.4-rscloud /usr/src/linux
sudo rm -rf /lib/modules/2.6.35.4-rscloud/build
sudo ln -s /usr/src/linux /lib/modules/2.6.35.4-rscloud/build

Create the configuration file and prepare the modules

zcat /proc/config.gz | sudo tee /usr/src/linux/.config > /dev/null
sudo make oldconfig
sudo make modules_prepare
sudo make INSTALL_HDR_PATH=/usr headers_install

If all went well you can now build custom kernel modules.

Creating backups for Cloud Servers larger than 2GB

Judinous — Sat, 06 Nov 2010 00:08:17 +0000

According to the website, backups for Cloud Servers are only available for instances with a memory size of 2GB or less. However, this restriction only applies to (some of) the control panel, and not the API itself. It is still quite possible to take snapshots of your larger servers, though there are a few caveats along the way.

To take snapshots of your >2GB Cloud Servers through the control panel, you first need to go to Hosting -> Cloud Servers and click on the “My Server Images” tab. Note that you do not want to go to an individual server’s overview page to take the image; it will not work for >2GB servers.

Just hit the “New Image” button on this page and you can select any of your servers from the list. In this case, I will take a backup of my 4GB server called “sinatra”.

Pick a name for your image (I called mine “sinatra-backup”), wait for it to complete, and voila: you have a backup of your >2GB server.

You can also create backups for >2GB servers directly through the API. You can use the script in this earlier Failverse blog post to create backups of your servers as well; simply run the script directly instead of putting it in a cron job.

Caveats

Although you can use the two aforementioned methods to create backups of >2GB servers, there are a number of restrictions still in place.

Linux:

The total amount of disk space currently used cannot be higher than 75GB. You can check how much space you are using with the “df -h” command.
The total number of inodes in use cannot exceed 3 million. You can check how many inodes are in use with the “df -i” command. If your inode usage is inordinately high, deleting unnecessary files (generally log files or stale connections are the culprit here) or temporarily zipping up multiple files into tarballs will reduce your usage.
If it takes longer than 2 hours for the file transfer portion of the backup to finish, it will fail. This may occur if you have a large amount of both disk space and inodes in use, though not enough to trigger the hard caps. Reducing your inode usage can help reduce the time it takes to perform the backup.

Windows:

The total size of the sparce disk file that your virtual hard disk resides upon cannot be larger than 160GB. What this means is that the highest amount of disk space in use at one time in your server’s history cannot have exceeded 160GB. If you were using more disk space than that in the past, you are basically SOL; the only way to make a backup image is to copy your data to a new server (assuming you have <160GB in use currently) and then take a snapshot of that new machine.