Planet Kai RSS feed

dabase
Docker container update workflow

I setup Greptweet a few months ago on a DO "Droplet" aka VPS in London running CoreOS stable.

Over that period of time, there was at least one PHP update (as per bloody usual) and a cache bugfix (backstory) I need to rollout.

How do I test the fix?

This fix was to do with the nginx.conf, not the actual code, so it was easy to build the docker image locally, e.g. sudo docker build -t g5 . & sudo docker run -v /srv/www/greptweet.com:/srv/http/u -p 80:80 -ti g5

NEED A SOLUTION: What I found difficult to do however, is test any local code changes, like a Bootstrap update. Since the Dockerfile checks the source code out from git, but I need to test my working changes.

UPDATE: Neerav Kumar from DockerSG suggested I use ADD instead of git clone in my Dockerfile.

How did I deploy the fix

On my CoreOS droplet, I built a new image (from scratch) with docker build --no-cache -t greptweet . from a git checkout. I wasn't too sure what was going to happen, since there was already an image called "greptweet" there and in fact running. The new build seemed to simply replace that currently running build and all I then needed to do was sudo systemctl restart greptweet.service for the systemd service file to serve it.

NEED A SOLUTION: Er, so what happened to the old build of Greptweet? Seems to have disappeared by the build that replaced it. What happens if I want to downgrade, just git checkout oldversion and Docker build from there??

UPDATE: People suggested tags and updating the service file but I think git checking out and older version is a better approach for me.

Gotchas

WTH: Initially I built like so docker build -t greptweet . and noticed no changes on restart and an old version number on the app. It seems that Docker caching can't seem to tell when a step is actually likely to change (new changes in git) and invalidate it.

UPDATE: I'm told Docker can sense changes with ADD but not the RUN. So hopefully the change will make the builds better.

Had some issues with nginx configuration syntax error, independent of Docker.

Posted
dabase
Vhost docker container
23:19 <hendry> biggest feature missing for me is dockers lack of vhosting support. hosting off a random port is a bit silly, no?
23:20 <dstufft> hendry: you can't generically do vhosting
23:24 <hendry> dstufft: not quite sure why vhosting is SUCH a hard feature
23:24 <niloc132> hendry: for a start, its http-specific
23:26 <dstufft> and even for the protocols that do support a vhost feature, there isn't a standard protocol agnostic way of getting that information
23:26 <exarkun> You can set up your own vhosting in a container.  Docker doesn't /have/ to support it.  So it's probably better if Docker doesn't, given the lack of a single, obvious, complete (ie, supporing all protocols) solution.
23:26 <exarkun> And you can find lots of http vhosting images out there now because people do want to do this and have already solved the problem.
23:27 <hendry> i don't want to solve it in the container. I guess I need to study some nginx reverse proxy thing
23:27 <dstufft> nginx can do it
23:27 <dstufft> or haproxy
23:27 <dstufft> or any number of things
23:28 <dstufft> I like haproxy for it, it's a pretty good tool
23:28 <exarkun> Do you have a reason that you don't want to solve it in a container?  The way you state it, it sounds like an arbitrary constraint.
23:28 <hendry> dstufft: why haproxy over nginx?
23:28 <hendry> exarkun: because i would be building more complexity in my container that i want to keep dead simple? or is the functionality running in another seperate container?
23:29 <dstufft> hendry: haproxy isn't HTTP specific, so if you find yourself wanting to do more you don't need to drop in another thing to handle it
23:29 <dstufft> it would be running in another seperate container
23:29 <hendry> dstufft: everything i do is HTTP.... (though does Websockets run over port 80/443?)
23:29 <exarkun> hendry: As dstufft said, *not* in your container.
23:30 <exarkun> That's the point.  Independent, composeable units.  Containers.
23:30 <hendry> exarkun: can you point to such a solution for a container to dispatch vhosts IIUC to another container ?
23:30 <exarkun> You can find one with two minutes of Googleing, I think.
23:31 <hendry> "vhost docker container" not looking good
23:33 <hendry> https://registry.hub.docker.com/search?q=vhost not looking great either
23:33 <hendry> exarkun: i give up
23:35 <exarkun> To your credit, you did spend three minutes.
23:35 <exarkun> I don't know what more effort anyone could be asked to expend than that.
23:36 <exarkun> (I'm certainly not going to!)
23:37  * hendry sighs
Posted
dabase
'invalid value for project' google compute engine

Google wasted my time by having a distinction between PROJECT NAME & PROJECT ID.

The SDK will ask you to set gcloud config set project NAME

NAME is the PROJECT ID

When things go wrong:

ERROR: (gcloud.compute.instances.create) Some requests did not succeed:
 - Invalid value for project: localkuvat

When things go right:

$ gcloud config set project numeric-rig-758
$ gcloud compute instances create outyet \
    --image container-vm-v20140925 \
    --image-project google-containers \
    --metadata-from-file google-container-manifest=containers.yaml \
    --tags http-server \
    --zone us-central1-a \
    --machine-type f1-micro
Created [https://www.googleapis.com/compute/v1/projects/numeric-rig-758/zones/us-central1-a/instances/lk].
NAME ZONE          MACHINE_TYPE INTERNAL_IP   EXTERNAL_IP    STATUS
lk   us-central1-a f1-micro     10.240.89.159 146.148.60.109 RUNNING

Hat tip: https://blog.golang.org/docker

Posted
dabase
Experiencing CoreOS+Docker

Docker Logo

Once upon a time there was chroot (notice, it's just 50LOC?!). chroot was a simple way of sandboxing your application, namely just filesystem isolation. chroot lacked some features people wanted and alongs came Docker, which is a front end to LXC. It works, though it has a LOT of SLOC/complexity/features. Docker is monolithic and depends on Linux.

Today we have general packaged distributions like Debian & Archlinux. Their main fault was probably being too general, poor abilities to upgrade and downgrade. Along comes CoreOS, a lightweight Linux OS with (a modern, but hugely controversial init) systemd, heavyweight btrfs & docker. CoreOS is also monolithic and depends on Linux.

I've attempted to understand CoreOS before, though since I needed to move Greptweet to a VPS with more disk space... quickly... I "deep dived" into CoreOS & Docker and here is my writeup of the experience. Tip #1, the default user for CoreOS is "core", e.g. ssh core@178.62.119.197 once you get for e.g. your CoreOS droplet going.

Dockerfile

The 20LOC Greptweet Dockerfile took me almost all day to create, though this was my favourite accomplishment. I studied other Archlinux and Ubuntu docker files on Github to give me guidance how to achieve this.

So now I have a succinct file that describes Greptweet's environment to function. I found it astonishing the container for running a PHP app on Nginx is almost 1GB!

Yes, I need to re-implement greptweet in Golang to greatly reduce this huge bloat of a dependency!

Read only filesystem on CoreOS means no superuser or what?

I was hoping CoreOS would do away with root altogether. I'm seriously tired of sudo. I noticed read only mounts, whilst trying to disable password ssh logins to avoid loads of:

Failed password for root from $badman_IP port $highport ssh2

In my journalctl. Ok, if they are going to fix the config of sshd_config I thought, maybe they would do away with root?! PLEASE.

Update: If you discover WHERE sshd_config is managed in git by the CoreOS, I will give you a MAN HUG. CoreOS have totally obfuscated how they maintain configurations, compared to Archlinux packages!!

Haunting permissions

I hate UNIX permissions, hate hate hate. So with Docker your data is mounted on the host IIUC and your app stays firmly containerized.

But when your app writes data out on to a mount point, what THE HELL should the permissions be? I ended up just chmod -R 777 on my Volume's mountpoint, though I should probably have used setfacl What a mess!

User/group 33

How am I supposed to log CoreOS/Docker?!

I'm confused about Volume mounts. I run Greptweet like so: /usr/bin/docker run --name greptweet1 -v /srv/www/greptweet.com:/srv/http/u -p 80:80 greptweet, and /srv/http/u/ is where the data lives. But HOW am I supposed to get at my container's logs? Another volume mount?

How does CoreOS envision managing httpd logs? I don't understand. And how am I supposed to run logrorate!? "One step forward, two steps back" is playing in my mind.

Init

A jarring thing is that when you run a docker container, you IIUC are expected to run one process, i.e. the httpd.

Unfortunately with nginx, to get PHP working you need to run a separate (FastCGI) PHP process to nginx httpd, hence the Greptweet Dockerfile uses Python's supervisor daemon to manage both processes. Urgh. I copied this paradigm from another Dockerfile. Tbh I was expecting to manage the process with systemd inside the container. Now I have Python crapware in my container for managing nginx/php processes. Suck.

NO Cron

Greptweet used cron to create backups, relay stats and generate reports. Now AFAICT I don't have the basic utility of cron in my container. Now what?!

WTF IS CRON IN COREOS, I NEED IT BACK!

Update: We are suppose to use systemd timers

Update_engine

As mentioned in my previous blog on CoreOS, I was quite excited about have "free" updates to my core host system. Sadly after looking at the logs, I'm not impressed.

There is little visibility to the actual update. I have recently found https://coreos.com/releases/ but it uses some horrible XML manifest to layer on the updates. Why can't the whole rootfs just be in git ffs?

Furthermore I noticed locksmithd which I think reboots the machine, but I'm not sure.

Oct 18 03:11:11 dc update_engine[458]: <request protocol="3.0" version="CoreOSUpdateEngine-0.1.0.0" updaterversion="CoreOSUpdateEngine-0
Oct 18 03:11:11 dc update_engine[458]: <os version="Chateau" platform="CoreOS" sp="444.5.0_x86_64"></os>
Oct 18 03:11:11 dc update_engine[458]: <app appid="{e96281a6-d1af-4bde-9a0a-97b76e56dc57}" version="444.5.0" track="stable" from_track="
Oct 18 03:11:11 dc update_engine[458]: <ping active="1"></ping>
Oct 18 03:11:11 dc update_engine[458]: <updatecheck targetversionprefix=""></updatecheck>
Oct 18 03:11:11 dc update_engine[458]: <event eventtype="3" eventresult="2" previousversion=""></event>
Oct 18 03:11:11 dc update_engine[458]: </app>
Oct 18 03:11:11 dc update_engine[458]: </request>

I've glanced over https://coreos.com/using-coreos/updates/ several times now and it's still not clear to me. As an operating system maintainer myself for Webconverger updates, our gitfs upgrade system is MUCH CLEARER than how CoreOS updates are handled. I wonder wth Docker 1.3 is going to hit CoreOS stable.

Keeping my Archlinux docker container uptodate is also a bit of a mystery to me...

CoreOS packaging is just WEIRD

It took me way too long to figure out how to enter a Docker 1.2 container and have a look. nsenter will be replaced by Docker 1.3's docker exec, but the way it installed was very intriguing.

In fact package management in CoreOS eyes I think means starting a share/privileged container and mapping it back to the host system. That's a pretty darn wild way of doing things imo.

I've been BATTLING TO GET TMUX running. It was suggested that this screen CoreOS install guide might help me. There is also an odd "toolbox" alias to a Fedora container with tools, but it doesn't map back to the host. All this for a terminal multiplexer. OMG.

Starting your Docker container in CoreOS was non-trivial

Here is Greptweet's service file.

CoreOS's launch guide was a bit strange to me. Am I supposed to publish my Greptweet image, so the docker pull works? It could be a lot simpler I feel. I.e. why doesn't the docker daemon manage/start the containers itself!?

Conclusion

I think the basic idea of lightweight host OS (CoreOS) and containers (Docker) has legs. I just wish it was as simple as chroot. Now I'm left thinking how Ansible/Chef/Puppet/Vagrant did such a bad job compared to the Dockerfile. Or perhaps blaming VPS hosters who never really got a decent API together to move/expand/inspect their VPS volumes.

Gosh, how did we get into this mess?!

So now system administrators now run hypervisors aka CoreOS and spin up VPSes aka Docker containers all by themselves. Seems like another level of abstraction that empowers system administrators but at the same time there is going to a raft of bugs/pain to enjoy with this "movement". It's also slightly concerning that CoreOS/Docker seems to fly in the face of the Unix philosophy.

My rating for CoreOS: 1 out of 5 stars

Posted
tips
Finding the rotation of a iPhone video

Using ffprobe which should be included in a ?ffmpeg(https://twitter.com/FFmpeg) distribution:

for m in *.MOV
do
        r=$(ffprobe $m 2>&1 | grep -i rotate | awk '{print $3}')
        case $r in
                90)
                        echo Needs to be $m 90 degrees
                        ;;
                180)
                        echo Needs to be $m 180 degrees
                        ;;
                270)
                        echo Needs to be $m 270 degrees
                        ;;
                *)
                        echo No rotating required $m
        esac
done
Posted
dabase
How much does it cost to run an Archlinux mirror on EC2

AWS Singapore kindly gifted http://hackerspace.sg/ with 500SGD of AWS credits.

Since the mirrors http://mirror.nus.edu.sg/ and http://download.nus.edu.sg, which are two separate competing groups from the NUS which oddly try to outdo each other in incompetence, have had several issues mirroring Archlinux in my two year experience of using either of them, I thought lets use these credits to host an Archlinux mirror!!

After much head scratching with the AWS jargon of {ebs,s3} and {hvm,paravirtual} EC2 Archlinux images, I launched an "ebs hvm" instance of m3.xlarge.

I got a nice 80GB zpool going for the mirror and everything was looking good. However, now to do the budgeting.

On demand pricing is $0.392 an hour

There is roughly 9000 hours in a year. So that's $3528. Eeeek, over budget by just 3000 dollars!

Ignoring added complexity of Spot and EBS enhancements, a one year resevered instance under "Light Utilization Reserved Instances" (I am not sure what that means) is 497 dollars! Yes!!

I'm told "Light utilization means that you will not turn it on all the time". For 1 year I would need heavy utilization!

So a m3.xlarge would be: 981 (down payment) + 24 * 365 * 0.124 = $2067.24, about 1500 dollars over budget.

Oh and bandwidth?

Well, a mirror is going to be a network whore. AWS charges for bandwidth. I tried their calculator (since I couldn't figure out what they charge per GB) with a lowball 1TB a month in and out and that costs almost 200USD.

Wow that's expensive! AWS EC2 (+ 500SGD credit) isn't suitable for an Archlinux mirror! :(

Digital Ocean quote

For a machine with at least 50GB of disk, you would need Digital Ocean's 60GB offering, with

  • 4GB / 2 CPUS
  • 60GB SSD DISK
  • 4TB TRANSFER

So that is 40USD a MONTH or 480USD a year. A lot cheaper than EC2, and bandwidth clearly priced at 2c per GB, so 1TB = 20USD IIUC.

Lessons learnt

Running a mirror is quite expensive on EC2. It's not really feasible on DO either without some free unmetered traffic.

Posted
tips
Working with a directories of unknown files

Using http://mywiki.wooledge.org/BashFAQ/020 as a starting point, you could:

find /tmp -type f -print0 | while IFS= read -r -d '' file
do
   echo properly escaped "$file" for doing stuff
done

However that's a bit ugly. And note that -d '' only works in bash. So none of this is "POSIX".

Another way of writing this, which works from bash 4 is using dotglob/globstar:

shopt -s dotglob  # find .FILES
shopt -s globstar # make ** recurse
for f in /tmp/**
do
    if <span class="createlink"><a href="/ikiwiki.cgi?page=_-f___36__f___38____38_____33___-L___36__f_&amp;from=e%2F13042&amp;do=create" rel="nofollow">?</a> -f &#36;f &#38;&#38; &#33; -L &#36;f </span>
    then
        echo properly escaped "$f" for doing stuff
    fi
done

Another perhaps more POSIX way is

foo () { for i in "$@"; do echo $i; done };export -f foo;find /tmp -type f -exec bash -c 'foo "$@"' - {} + | wc -l

I.e. export a script function to be executed by the -exec parameter of find, or just use a seperate script file.

Posted
dabase
SIGFOX

I attended an event at Hackerspace.SG Running your IoT devices on a low power, long range network, which showcased Lee Lup's slides on SIGFOX at Singtel.

My thoughts are that the 12 byte payload is not really suitable for monitoring, but more for events.

  • Door closed
  • Bin's full
  • Some significant threshold exceeded

It's not super good for monitoring since:

  • payload is too small
  • there is no accurate time
  • you can't monitor any less than 10 minute intervals

I found the people tracking / GPS use cases to be silly, since GPS needs a lot of power.

SIGFOX lends itself to well known fixed locations.

Fixed locations, such as public property that need maintenance and don't have Internet connectivity for one reason or another.

I'm thinking:

  • alerting to street lamps failing
  • alerting to flooding
  • alerting to when a bin is full & needs emptying
  • alerting when a special door/gate is opened

I like how the SIGFOX station seems to send out Github style Webhook payloads to a payload URL.

Posted
dabase
Back to bash after fish

Fish logo

I tried the http://fishshell.com/ instead of bash, mainly because my bash history is continually truncated despite my efforts to keep every command I've ever successfully (return code 0) typed!

x220:~$ wc -l .bash_history
84 .bash_history
x220:~$ history | wc -l
82

The fish shell's 85k SLOC versus bash's 225k was also re-assuring. autotools and cpp did make me cringe however.

So porting my aliases and such from ~/.config/fish/config.fish from my ~/.bashrc was a bit of a PITA since:

  • fish's conditional statements are not POSIX shell compatible <--- insane

Nonetheless I got fish up and running... and I used fish 2.1 for about a month.

So... does shell history work in fish?

Yes, yes it does. fishd provided me the commands I typed in, HOWEVER:

  • it logged failed commands, I DO NOT WANT FAILED COMMANDS SAVED
  • failed commands could be a password. To remove them I had to 'fish_config' and go to history tab.
  • ctrl+r (bash's reverse search) doesn't work. Instead you need to toggle between ctrl+f (autocomplete) and ctrl+p (search)

Tbh I could live with this fishiness. Simply because I had a working shell history. Seriously.

Why I stopped using fish

Consider this code from http://git.suckless.org/dmenu/tree/dmenu_run

dmenu_path | dmenu "$@" | ${SHELL:-"/bin/sh"} &

So I would choose a bash script like screenshot, to be run by dmenu, which is then piped to $SHELL. $SHELL being /usr/bin/fish when using fish. Screenshot's shebang #!/bin/bash does not apply.

IIUC variable expansion in fish is done different causing 99% of shell scripts to fail when run by fish. Nevermind the crazy stderr caret stuff. fish being a non-compatible shell is just a TOTAL FAIL.

Furthermore setting up PATH on Linux is just generally insane, probably because I've (wrongly) setup the PATH mainly in my ~/.bashrc, instead of ~/.profile or is it somewhere else? Anyway my PATH was consistency broken in fish too and I never figured 100% why. Perhaps because of its variable expansion anti-feature.

Back to GNU bash

Tbh, ctrl+r aka (reverse-i-search) is much better & intuitive than fish's ctrl+f & ctrl+p.

Now if only I could get my bash history working. I miss fishd I guess. Tbh I did like fish's fish_config and the way it attempted but kinda failed to integrate with the browser.

Posted
dabase
Macbook Air Early 2014 Archlinux

I upgraded my Mom's old 11 inch Macbook Air since she said the screen was too small. Enter a 13-inch, Early 2014 model with OS X 10.9.3 and enter an opportunity for me to test how well Archlinux works on this gorgeous Apple hardware in the month of June 2014. Yes, I am looking to upgrade from my Thinkpad X220 and Lenovo sadness.

Firstly I must say, Archlinux wiki's Macbook page is a bit of a mess. It over-complicates partitioning and boot loading. However I don't feel like wading in there to try fix it.

It's surprisingly simple to Archlinux going on the Macbook Air.

  1. Boot into OSX. Upgrade.
  2. Resize the disk using Disk Utility, amazingly you can resize a running MacOSX install
  3. Install rEFInd, with the all drivers switch, ./install.sh --alldrivers
  4. Boot the Archlinux ISO on a USB stick by holding down the Alt key
  5. Fdisk, create a new partition /dev/sda4, mkfs.ext4 format it, mount it and pacstrap the mount
  6. Oops, pacstrap doesn't work without Internet and 03:00.0 Network controller: Broadcom Corporation BCM4360 802.11ac Wireless Network Adapter (rev 03) is not supported by the 3.15 Linux kernel :( Had to purchase a USB 3.0 to 10/100/1000 Gigabit Ethernet LAN Network Adapter, Apple IIUC doesn't offer USB3 hardware!?
  7. Reboot and with some luck rEFInd, will pick it up! Hint: Installing refind-efi and refind-mkrlconf should give you more control, e.g. for adding "rw quiet" or hitting F2 (twice).

Wireless is since working well so far with the broadcom-wl package from AUR.

Issues

Keyboard

The Alt key is way too small and I am not sure what to make of the fn & cmd key.

Fixed with setxkbmap -option altwin:swap_alt_win.

Someone needs to tell me where the HELL is the Insert key on a Macbook Air.

TrackPad

I am familiar with the Thinkpad nipple. I'm struggling with the trackpad. I can't figure out how to paste!!!

Is it a "three finger click" to paste? Anyone know the keyboard shortcut?

Better with synclient PalmDetect=1.

BETTER with xf86-input-mtrack-git from AUR.

caffeinate

Suspending, shutdown etc. works out of the box with Archlinux. I think systemd should get the thanks. Really fast.

When I close the lid, the Macbook Air goes to sleep. But sometimes it's doing a long upload or download and I do not want it to sleep.

Sound / Audio output doesn't work

rain:/etc/modprobe.d$ lspci | grep Audio
00:03.0 Audio device: Intel Corporation Device 0a0c (rev 09)
00:1b.0 Audio device: Intel Corporation Lynx Point-LP HD Audio Controller (rev 04)

I'm not sure what I did, but it's working now with pavucontrol. Might be my general confusion with starting pulseaudio.

Brightness buttons don't work

Pommed dies with E: Unknown Apple machine: MacBookAir6,2. Filed a bug.

Is the Thunderbolt port usable? Yes!

User Experience notes

The Air doesn't have a fan, so when I come back to using the X220, I find the X220 noisy.

The Macbook Air so far seems to hover around 51C (~70C with Youtube playback) and its temperature /sys/class/thermal/thermal_zone0/temp doesn't fluctuate as crazily as the X220 does. Which is good!! Its actually cool enough to be on my lap without feeling like I am killing my sperm off.

The 1440x900 screen is nicer, but I expected it to a LOT better. I guess I was stupidly expecting Retina.

The battery seems to drain as fast as the old 9 cell in the Thinkpad worryingly. I have tuned it with PowerTOP and still I'm not very confident at all. I suspect the screen on full brightness doesn't help.

Webconverger

The distro that I maintain and earn a living from, Webconverger works with the Apple Macbook Air hardware too with the USB network device is attached.

Posted
tips
Ensure www-data is always able to write

Ensure your fs is mounted with acl.

 mount | grep acl
/dev/root on / type ext3 (rw,noatime,errors=remount-ro,acl,barrier=0,data=writeback)

And to ensure www-data always has free reign:

setfacl -R -m default:group:www-data:rwx /srv/www
Posted
tips
Xorgs version
12:04 <hendry> i'm using wheezy Xorg packages  1:7.7+3~deb7u1 and http://ix.io/d2p says X.Org X Server 1.12.4
12:04 <hendry> Release Date: 2012-08-27
12:04 <hendry> Is that right?
12:05 <jcristau> probably
12:11 <hendry> wondering why there is a mis-match with versions
12:11 <hendry> is there a newer Xorg available for wheezy? something to eek out performance with intel cards
12:12 <jcristau> there isn't a mismatch
12:12 <jcristau> and no
12:12 <hendry> 1:7.7+3~deb7u1 & 1.12.4 doesn't make sense to me ... :}
12:13 <jcristau> you can't understand that different things can have different versions?
12:20 <hendry> so what does 7.7+3~deb7u1 refer to ?
12:22 <pochu> 7.7 is the upstream version, +3 is the debian revision, and deb7u1 is the first update to Debian 7 (wheezy)
12:22 <psychon> http://www.x.org/wiki/Releases/7.7/
12:28 <jcristau> 7.7 is the base version of X.Org's X11 distribution
12:28 <jcristau> 1.12.4 is the version of the X server

http://www.x.org/wiki/Releases/7.7/

Posted
dabase
S3 permission policy confusion

I do like AWS S3, but on the topic of policy/security languages, IIUC Amazon have fecked up by having THREE of them.

  1. permissions
  2. policy
  3. CORS configuration

What to add anonymous public listing? You could:

  1. Click "Add more permissions"
  2. On the Grantee drop down box, select Everyone (not sure why I'm called hendry60 and "me")
  3. Ensure listing is selected and Save

or you could set a policy with "s3:ListBucket" IIUC. Futhermore I don't see why CORS couldn't be derived from Amazon's own Access Policy Language.

Permissive listing policy

$ bash allow-list.sh b3-webc
ERROR: S3 error: 400 (MalformedPolicy): Action does not apply to any resource(s) in statement

Not sure what's wrong with my policy:

$ cat allow-list.sh
#!/bin/bash
test "$1" || exit
s3_bucket=$1
tmp=$(mktemp)
s3cmd ls > $tmp
if ! grep -q $s3_bucket $tmp
then
        echo Could not find bucket s3://${s3_bucket}
        cat $tmp
        exit
fi
cat <<END > $tmp
{
  "Id": "Policy1400237335721",
  "Statement": [
    {
      "Sid": "Stmt1400237307049",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::${s3_bucket}/*",
      "Principal": {
        "AWS": [
          "*"
        ]
      }
    }
  ]
}
END
s3cmd setpolicy $tmp s3://${s3_bucket}
Posted
tips
Setting a read S3 policy from the command line

Easier than logging into https://console.aws.amazon.com/s3/ since I need to get out my MFA device out everytime.

x220:/tmp$ bash allow-read.sh b3-webc
s3://b3-webc/: Policy updated

allow-read.sh is just a script to help write the policy:

x220:/tmp$ cat allow-read.sh
#!/bin/bash
test "$1" || exit
s3_bucket=$1
tmp=$(mktemp)
s3cmd ls > $tmp
if ! grep -q $s3_bucket $tmp
then
        echo Could not find bucket s3://${s3_bucket}
        cat $tmp
        exit
fi
cat <<END > $tmp
{
  "Version":"2008-10-17",
  "Statement":[{
    "Sid":"AllowPublicRead",
        "Effect":"Allow",
      "Principal": {
            "AWS": "*"
         },
      "Action":["s3:GetObject"],
      "Resource":["arn:aws:s3:::${s3_bucket}/*"
      ]
    }
  ]
}
END
s3cmd setpolicy $tmp s3://${s3_bucket}
Posted
dabase
Howto join MP4 files

What does not work

ffmpeg has an insane interface for simple use cases like joining two MP4 files.

cat draw.mp4 draw2.mp4 > draw3.mp4

ffmpeg -i "concat:draw.mp4|draw2.mp4" -c copy draw3.mp4

ffmpeg -i draw.mp4 -i draw2.mp4 -vcodec copy -acodec copy draw3.mp4

mkfifo temp1 temp2
ffmpeg -i input1.mp4 -c copy -bsf:v h264_mp4toannexb -f mpegts temp1 2> /dev/null & \
        ffmpeg -i input2.mp4 -c copy -bsf:v h264_mp4toannexb -f mpegts temp2 2> /dev/null & \
        ffmpeg -f mpegts -i "concat:temp1|temp2" -c copy -bsf:a aac_adtstoasc output.mp4

What works

ffmpeg -f concat -i <(for f in *.mp4; do echo "file '$(readlink -f $f)'"; done) -c copy output.mp4

With a different sort:

ffmpeg -f concat -i <(for f in $(ls -1 | sort -r); do echo "file '$(readlink -f $f)'"; done) -c copy output.mp4
Posted
tips
Failed to start Verify integrity of password and group files.

Fix it by running pwck.

x220:~$ sudo pwck
user 'umurmur': directory '/home/umurmur' does not exist
pwck: no changes
x220:~$ pacman -Ss umurmur
community/umurmur 0.2.14-1
    Minimalistic Mumble server
x220:~$ sudo vim /etc/passwd
x220:~$ sudo pwck
no matching password file entry in /etc/passwd
delete line 'umurmur:!:16094::::::'? yes
pwck: the files have been updated
Posted
tips
From Apache to Nginx rewrites

Apache:

RewriteRule ^client/(.*)$ /client.php?id=$1               [L,QSA]

Nginx:

rewrite ^/client/(.*)$ /client.php?id=$1&$args last;
  • Notice uris in nginx must start with /
  • Notice &$args is needed to replicate Apache's QSA feature
Posted
dabase
Good riddance netctl

For the last year (2013) Archlinux has been recommending users to use netctl to configure their network interfaces.

Netctl's ethernet-dhcp worked well, but not the wireless interface management which never made sense to me.

Upstream's Jouke Witteveen is sadly unhelpful despite my cries for help.

So on the back of systemd's (>210) systemd-networkd, which I previously blogged about when configuring my Droplet, I now have:

$ cat /etc/systemd/network/eth0.network
[Match]
Name=eth0
[Network]
DHCP=yes

Via journalctl -u systemd-networkd.service -f this seems to be able to detect the carrier is on or off without ifplugd. Nice!

For your DNS nameservers, you need to symlink /etc/resolv.conf to /run/systemd/network/resolv.conf

etc$ sudo ln -s /run/systemd/network/resolv.conf

Wireless

To do all the WPA authentication stuff:

sudo systemctl enable wpa_supplicant@wlan0.service

You need to make sure /etc/wpa_supplicant/wpa_supplicant-wlan0.conf is in place, as you can see in /usr/lib/systemd/system/wpa_supplicant@.service. I prefer to link in /etc/wpa_supplicant.conf where I've also stored my wireless passwords and things.

/etc/wpa_supplicant$ sudo ln -s /etc/wpa_supplicant.conf wpa_supplicant-wlan0.conf

Now to get the IP address, we use systemd-networkd, with the configuration:

$ cat /etc/systemd/network/wlan0.network
[Match]
Name=wlan0
[Network]
DHCP=yes

Wrap up

systemd-analyze for wlan0 is 3.203s and eth0 is 3.085s. I'm happy !

Currently I manually turn toggle wifi using my Thinkpad's wireless switch, for switching between wired and wireless interfaces.

Acknowledgements: WonderWoofy on the the Archlinux forums

Posted
dabase
Systemd network on a Droplet

From roughly systemd version 210, networking in my Droplet changed.

Out went netctl somehow (good riddance), and my network connectivity (uh oh).

Using Digital Ocean's buggy Console Access, I managed to setup my network access manually.

/etc/network.d/ethernet-static gave me the previous network settings:

[root@sgd ~]# cat /etc/network.d/ethernet-static
CONNECTION='ethernet'
INTERFACE='eth0'
IP='static'
ADDR='128.199.252.174'
NETMASK='255.255.192.0'
GATEWAY='128.199.192.1'
DNS=('8.8.4.4' '8.8.8.8' '209.244.0.3')

How to setup a networking interface with ip

I'm more familiar with ifconfig & route, but with some frantic Googling I did figure out ip's odd unfamilar API:

ip addr add 128.199.252.174/255.255.192.0 dev enp0s3
ip route add default via 128.199.192.1

ip a then looks like:

[root@sgd ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 04:01:11:8d:89:01 brd ff:ff:ff:ff:ff:ff
    inet 128.199.252.174/18 brd 128.199.255.255 scope global enp0s3
    inet6 fe80::601:11ff:fe8d:8901/64 scope link
       valid_lft forever preferred_lft forever

Sidenote: I'm not sure why my Droplet's network interface is called enp0s3 and not something simple like eth0.

Ping to test... YES... I can ssh in.

Now to migrate to systemd-networkd.

I created /etc/systemd/network/enp0s3.network:

[root@sgd ~]# cat /etc/systemd/network/enp0s3.network
# Migrated from /etc/network.d/ethernet-static
# https://coreos.com/blog/intro-to-systemd-networkd/
[Match]
Name=enp0s3
[Network]
Address=128.199.252.174/18
Gateway=128.199.192.1
  1. To test systemctl start systemd-networkd.service.
  2. Check systemctl status systemd-networkd.service looks good

      [root@sgd ~]# systemctl status systemd-networkd.service
      systemd-networkd.service - Network Service
         Loaded: loaded (/usr/lib/systemd/system/systemd-networkd.service; enabled)
         Active: active (running) since Thu 2014-03-13 04:02:24 UTC; 11min ago
           Docs: man:systemd-networkd.service(8)
       Main PID: 158 (systemd-network)
         Status: "Processing requests..."
         CGroup: /system.slice/systemd-networkd.service
                 `-158 /usr/lib/systemd/systemd-networkd
      Mar 13 04:02:24 sgd systemd-networkd[158]: enp0s3: link is up
      Mar 13 04:02:24 sgd systemd-networkd[158]: enp0s3: carrier on
      Mar 13 04:02:24 sgd systemd-networkd[158]: enp0s3: link configured
      Mar 13 04:02:24 sgd systemd[1]: Started Network Service.
    

To watch/tail what the systemd-networkd.service is doing, journalctl -u systemd-networkd.service -f

  • Enable it systemctl enable systemd-networkd.service
  • And reboot!

Loose ends, ifplugd

I noticed in htop ifplugd is running. I can't see from systemctl or pstree what invoked it. Mysterious.

[root@sgd ~]# ps aux | grep ifplugd
root       136  0.0  0.0   2200   284 ?        Ss   04:02   0:00 /usr/bin/ifplugd -i enp0p3 -r /etc/ifplugd/netcfg.action -fIns
root     27388  0.0  0.1   2828   584 pts/1    S+   04:16   0:00 grep ifplugd
Posted
dabase
Iperf SG:DigitalOcean versus GPLhost

Since Digital Ocean has opened a new SG datacenter, it's time to pit my existing SG VPS at GPLhost against it!

From my Singtel connection at home, with iperf -s running on either VPS.

x220:~$ iperf -c sg.hackandtell.org # GPLhost
------------------------------------------------------------
Client connecting to sg.hackandtell.org, TCP port 5001
TCP window size: 22.9 KByte (default)
------------------------------------------------------------
[  3] local 192.168.88.249 port 46675 connected with 117.121.241.187 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.2 sec  18.2 MBytes  15.0 Mbits/sec
x220:~$ iperf -c sgo.webconverger.com # Digital Ocean's new SG VPS
------------------------------------------------------------
Client connecting to sgo.webconverger.com, TCP port 5001
TCP window size: 22.9 KByte (default)
------------------------------------------------------------
[  3] local 192.168.88.249 port 52035 connected with 128.199.252.174 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.3 sec  6.00 MBytes  4.89 Mbits/sec

BAD SINGTEL ROUTING: mtr shows the routing from Singtel (SG's largest ISP IIUC) to Digital Ocean's SGP1 is poor. It seems to go to Hong Kong and back and hence the pings are ~80ms, when they should be ~5ms!

I know GPLhost's network configuration is limited to just 50 Mbit/sec. It's a shame I'm not getting close to it from my OpenNET Singtel line at home. :(

Nonetheless GPLhost beats Digital Ocean on repeated iperf tests on average by at least 2x from my Singapore connection.

From a Digital Ocean droplet in the Netherlands

[hendry@nl ~]$ iperf -c sgo.webconverger.com # Digital Ocean
------------------------------------------------------------
Client connecting to sgo.webconverger.com, TCP port 5001
TCP window size: 22.9 KByte (default)
------------------------------------------------------------
[  3] local 146.185.152.215 port 45486 connected with 128.199.252.174 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.1 sec  6.38 MBytes  5.30 Mbits/sec
[hendry@nl ~]$ iperf -c sg.hackandtell.org # GPLhost
------------------------------------------------------------
Client connecting to sg.hackandtell.org, TCP port 5001
TCP window size: 22.9 KByte (default)
------------------------------------------------------------
[  3] local 146.185.152.215 port 60435 connected with 117.121.241.187 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-11.1 sec  1.50 MBytes  1.13 Mbits/sec

Digital Ocean's international peering seems a lot better. The results above are indicative.

From Linode in the UK

hendry@gb ~$ iperf -c sgo.webconverger.com # To the SG Digital Ocean droplet
------------------------------------------------------------
Client connecting to sgo.webconverger.com, TCP port 5001
TCP window size: 20.0 KByte (default)
------------------------------------------------------------
[  3] local 176.58.122.199 port 36559 connected with 128.199.252.174 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.1 sec  46.2 MBytes  38.4 Mbits/sec
hendry@gb ~$ iperf -c sg.hackandtell.org # To my GPLhost VPS in SG M1 datacentre
------------------------------------------------------------
Client connecting to sg.hackandtell.org, TCP port 5001
TCP window size: 20.0 KByte (default)
------------------------------------------------------------
[  3] local 176.58.122.199 port 33729 connected with 117.121.241.187 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-11.2 sec  2.12 MBytes  1.60 Mbits/sec

Digital Ocean's international bandwidth definitely seems a lot better!

Webconverger's VPS are documented upon http://webconverger.org/servers/.

From a Microsoft Azure VPS to either SG VPS

hendry@hackerspacesg:~$ iperf -d -c sgo.webconverger.com
WARNING: option -d is not valid for server mode
------------------------------------------------------------
Client connecting to sgo.webconverger.com, TCP port 5001
TCP window size: 22.9 KByte (default)
------------------------------------------------------------
[  3] local 10.146.222.93 port 53883 connected with 128.199.252.174 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   572 MBytes   480 Mbits/sec

480Mbit/sec from Microsoft to Digital Ocean... NICE! Still that is IIUC less than half a Gigabit connection.

hendry@hackerspacesg:~$ iperf -d -c sg.hackandtell.org
WARNING: option -d is not valid for server mode
------------------------------------------------------------
Client connecting to sg.hackandtell.org, TCP port 5001
TCP window size: 22.9 KByte (default)
------------------------------------------------------------
[  3] local 10.146.222.93 port 40515 connected with 117.121.241.187 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  55.2 MBytes  46.2 Mbits/sec

Ok, here GPLhost's 50Mbit/s limit is plain to see.

Conclusion

Digital Ocean's SG droplet internal connectivity to Singapore is a little worrying. A traceroute indicates traffic is going upto Hong Kong!

I need someone on a 1Gbps line to do some iperf testing. Any My Republic users reading this?

I'm going to continue with GPLhost till at least to the end of May and I guess I will make a decision whether to migrate or not long before then. GPLhost are a small outfit, though they have provided really good service over the period I've used them. DO on the hand seems a bit unpredictable. It took them two days to sort out a networking glitch in Archlinux images made available in Singapore. DO could easily be a victim of their own success.

Update: A network engineer from DO got in touch to say they working to improve local peering. I guess a follow up test from me will be in order...(!)

Update: Outage today http://www.digitaloceanstatus.com/history?update=206#95 and still the routing from Singtel to SGP1 is poor http://ix.io/aWZ after a month. :/

Update: 4 months later, still bad: http://r2d2.webconverger.org/2014-05-18/mtr-sgo.webconverger.com.html

Posted
dabase
USB3 disk testing

So I exchanged my WD Passport Ultra with a poorly fitting USB cable for a Seagate Backup Plus (Slim) portable drive which was roughly the same price at Sim Lim at Cybermind Computer on the 4th floor.

I wanted the Western Digital after reading http://blog.backblaze.com/2014/01/21/what-hard-drive-should-i-buy/ linked on https://news.ycombinator.com/... oh well. The Seagate's cable fits nicely on my X220 or my wife's MBP. Let's hope the Seagate's 3 year warranty works out too.

On my aging X220 I do have one USB3 port. So I thought it would be prudent to measure USB3 against USB2.

Here is my test script:

# cat testusbseagate.sh
set -e # something goes wrong, then exit!
mp=/mnt/sd
test -e $mp || exit
lsusb -v | grep -A 5 Seagate # show we are using USB 2 or 3
mount /dev/sdc1 $mp
mount | grep $mp
cd $mp
#dd if=/dev/zero of=tempfile bs=1M count=1024 conv=fdatasync,notrunc
dd if=tempfile of=/dev/null bs=1M count=1024
echo 3 > /proc/sys/vm/drop_caches
dd if=tempfile of=/dev/null bs=1M count=1024
#rm tempfile
cd /
umount $mp

I wrote the tempfile beforehand on this disk. Measuring the write of the file seemed to give to many false readings, so I went for measuring the read of the file, after a fresh mount.

USB2 gave:

# bash testusbseagate.sh
Bus 001 Device 011: ID 0bc2:ab20 Seagate RSS LLC
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.10

bcdUSB 2.10 indicates USB2.

  bDeviceClass            0 (Defined at Interface level)
--
  idVendor           0x0bc2 Seagate RSS LLC
  idProduct          0xab20
  bcdDevice            1.00
  iManufacturer           2 Seagate
  iProduct                3 Backup+  BK
  iSerial                 1 NA7573AM
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
FUSE exfat 1.0.1
/dev/sdc1 on /mnt/sd type fuseblk (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,blksize=4096)
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 37.6769 s, 28.5 MB/s
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 37.6217 s, 28.5 MB/s

So 28.5 MB/s read for USB2. Let's test my only USB3 port on my X220:

# bash testusbseagate.sh
Bus 003 Device 007: ID 0bc2:ab20 Seagate RSS LLC
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               3.00

bcdUSB 3.00 indicates USB3.

  bDeviceClass            0 (Defined at Interface level)
--
  idVendor           0x0bc2 Seagate RSS LLC
  idProduct          0xab20
  bcdDevice            1.00
  iManufacturer           2 Seagate
  iProduct                3 Backup+  BK
  iSerial                 1 NA7573AM
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
FUSE exfat 1.0.1
/dev/sdc1 on /mnt/sd type fuseblk (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,blksize=4096)
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 9.38842 s, 114 MB/s
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 9.27814 s, 116 MB/s

So USB3 seems 3x the speed of USB2 on this read test. Nice.

I ran Xbench on my wife's MBP on the Seagate and it produced:

Results 34.89   
    System Info     
        Xbench Version      1.3
        System Version      10.9.1 (13B3116)
        Physical RAM        8192 MB
        Model       MacBookPro11,1
        Drive Type      Seagate Backup+ BK
    Disk Test   34.89   
        Sequential  146.62  
            Uncached Write  95.17   58.44 MB/sec [4K blocks]
            Uncached Write  186.69  105.63 MB/sec [256K blocks]
            Uncached Read   149.12  43.64 MB/sec [4K blocks]
            Uncached Read   212.26  106.68 MB/sec [256K blocks]
        Random  19.80   
            Uncached Write  6.05    0.64 MB/sec [4K blocks]
            Uncached Write  66.35   21.24 MB/sec [256K blocks]
            Uncached Read   70.32   0.50 MB/sec [4K blocks]
            Uncached Read   133.45  24.76 MB/sec [256K blocks]

I don't have a USB2 drive to compare it with.

I wonder if Gigabit ethernet between two Archlinux machines with SSD would beat USB3? USB3's 114 MB/s is about 900Mbps...

Posted
tips
Setting up structs

Examples of how to populate a struct.

package main

import "fmt"

type event struct {
    Name string
    Date string
}

type events []event

func main() {
    e1 := event{}
    e1.Date = "Today"
    e1.Name = "Somethiing"
    e2 := event{Date: "Tomorrow", Name: "Dentist"}

    s := events{e1, e2}

    fmt.Printf("%+v\n", s)

}
Posted
tips
Parsing arbitary JSON

For example.json:

{"menu": {
  "id": "file",
  "value": "File",
  "popup": {
    "menuitem": [
      {"value": "New", "onclick": "CreateNewDoc()"},
      {"value": "Open", "onclick": "OpenDoc()"},
      {"value": "Close", "onclick": "CloseDoc()"}
    ]
  }
}}

Desired output:

this.menu.popup.menuitem[0].value = "New"
this.menu.popup.menuitem[0].onclick = "CreateNewDoc()"
this.menu.popup.menuitem[1].value = "Open"
this.menu.popup.menuitem[1].onclick = "OpenDoc()"
this.menu.popup.menuitem[2].value = "Close"
this.menu.popup.menuitem[2].onclick = "CloseDoc()"
this.menu.value = "File"
this.menu.id = "file"

Python implementation

With thanks from Hyperair on Freenode's #hackerspace.sg

#!/usr/bin/python3

import json
import sys

fmt = '{key} = {value}'

def dump_obj(prefix, obj):
    if isinstance(obj, dict):
        for k, v in obj.items():
            if prefix:
                k = prefix + '.' + k

            dump_obj(k, v)

    elif isinstance(obj, list):
        for i, v in enumerate(obj):
            if prefix:
                k = "{prefix}[{idx}]".format(prefix=prefix, idx=i)
            dump_obj(k, v)

    else:
        print(fmt.format(key=prefix, value=json.dumps(obj)))

obj = json.loads(sys.stdin.read())
dump_obj('this', obj)

Golang implementation

Since Golang is statically typed, the important element in the code is t := x.(type) where they type of x gets asserted, which is a required step, so you can then iterate over the resulting structure.

package main

import (
    "encoding/json"
    "fmt"
    "io/ioutil"
    "os"
    "strconv"
)

func dumpobj(prefix string, x interface{}) {

    switch t := x.(type) {

    case map[string]interface{}:
        for k, v := range t {
            dumpobj(prefix+"."+k, v)
        }
    case []interface{}:
        for i, v := range t {
            dumpobj(prefix+"["+strconv.Itoa(i)+"]", v)
        }
    case string:
        fmt.Printf("%s = %q\n", prefix, t)
    default:
        fmt.Printf("Unhandled: %T\n", t)
    }
}

func main() {
    j, err := ioutil.ReadAll(os.Stdin)
    if err != nil {
        panic(err)
    }
    var pj interface{}
    err = json.Unmarshal(j, &pj)
    if err != nil {
        panic(err)
    }

    dumpobj("this", pj)

}

Alternative example with depth: http://play.golang.org/p/QOwxJQ4z4Z

Source is with thanks again from dsal from #go-nuts http://play.golang.org/p/pt4MLJ6HcV

However ideally you know the JSON structure beforehand, and you create a structure for it to map to, e.g. http://play.golang.org/p/bdjC6DLTqo. Only then you can access values easily like:

fmt.Println(myOb.Menu.Popup.Menuitem[0].Value)
Posted
tips
Dovecot IMAP Read Only Archive

http://wiki2.dovecot.org/HowTo/ReadOnlyArchive is outdated.

/etc/dovecot/dovecot.conf

$ sudo doveconf -n
# 2.2.9: /etc/dovecot/dovecot.conf
# OS: Linux 3.9.4-1-ARCH x86_64
auth_mechanisms = plain anonymous
log_path = /home/example/readonly/dovecot.log
mail_location = Maildir:~/mail/inbox
mail_log_prefix = %Us(%r):
passdb {
  args = /etc/anon.passwd
  driver = passwd-file
}
service auth {
  user = nobody
}
ssl = no
userdb {
  args = /etc/anon.passwd
  driver = passwd-file
}
valid_chroot_dirs = /home/example/readonly

For debugging purposes, tail /home/example/readonly/dovecot.log

/etc/anon.passwd

anonymous:{PLAIN}:501:501::/home/example/readonly/::nopassword=1

/home/example/readonly

Permissions are important. You will be doing sudo chown -R 501:501 .

$ tree -ugp readonly
readonly
|-- [-rw-r--r-- root     root    ]  dovecot.conf
|-- [-rw------- 501      501     ]  dovecot.log
`-- [dr-xr-xr-x 501      501     ]  mail
    `-- [drwxr-xr-x 501      501     ]  inbox
        |-- [dr-xr-xr-x 501      501     ]  cur
        |   |-- [-r--r--r-- 501      501     ]  1386050966.M540929P13587.sg.webconverger.com,S=315,W=328:2,S
        |   |-- [-r--r--r-- 501      501     ]  1386051008.M248603P13703.sg.webconverger.com,S=2081,W=2125:2,RS
        |   |-- [-r--r--r-- 501      501     ]  1386051093.M840486P13614.sg.webconverger.com,S=694,W=711:2,S
        |   |-- [-r--r--r-- 501      501     ]  1386051118.M789320P13725.sg.webconverger.com,S=2084,W=2127:2,S
        |   `-- [-r--r--r-- 501      501     ]  1386051172.M378951P13760.sg.webconverger.com,S=1881,W=1920:2,S
        |-- [-rw-r--r-- 501      501     ]  dovecot-uidlist
        |-- [-rw-r--r-- 501      501     ]  dovecot-uidvalidity
        |-- [-rw-r--r-- 501      501     ]  dovecot-uidvalidity.529d7573
        |-- [-rw-r--r-- 501      501     ]  dovecot.index.cache
        |-- [-rw-r--r-- 501      501     ]  dovecot.index.log
        |-- [-rw-r--r-- 501      501     ]  dovecot.index.thread
        |-- [-r--r--r-- 501      501     ]  inbox
        |-- [dr-xr-xr-x 501      501     ]  new
        `-- [dr-xr-xr-x 501      501     ]  tmp

5 directories, 14 files

.muttrc test

$ cat mutt-econv-test
set spoolfile=imap://anonymous@imap.dabase.com
set folder=imap://anonymous@imap.dabase.com
set sort=threads
set sort_aux=reverse-last-date-received

To test:

$ mutt -F mutt-econv-test

See https://github.com/kaihendry/econversations for why am I doing this.

Posted
tips
Reading Dovecot Archives

http://www.dovecot.org/mailinglists.html

set spoolfile=imap://anonymous@dovecot.org/dovecot
set folder=imap://anonymous@dovecot.org/dovecot
set sort=threads
set sort_aux=reverse-last-date-received
set header_cache=~/.mutt/cache/headers
set message_cachedir=~/.mutt/cache/bodies

Tested in Mutt 1.5.22.

When posting you probably want to set set from & set realname too.

Posted
dabase
Minimal Dovecot

The most minimal Dovecot IMAP configuration I could find is:

log_path = /var/log/dovecot.log
ssl = no

passdb {
    driver = pam
}

userdb {
    driver = passwd
}

The default dovecot.sample sucks (>1000 SLOC!!) and so does http://wiki2.dovecot.org/QuickConfiguration.

You need log_path or you won't be able to understand why dovecot fails to come up with dovecot.service: main process exited, code=exited, status=89/n/a typically seen via journalctl --full -f -u dovecot.

The Dovecot configuration format is a little painful. This won't work:

    passdb { driver = pam }

Nor this:

    passdb {
    driver = pam }

Feel free to experiment yourself between sudo systemctl restart dovecot while tail -f /var/log/dovecot.log on a large screen.

In my Archlinux system, by default email sanely ends up in ~/Maildir via Postfix handing mail over to Dovecot's Local Delivery Agent:

$ grep mailbox /etc/postfix/main.cf
mailbox_command = /usr/lib/dovecot/deliver

I could just not run the IMAP part I guess and use mutt -f ~/Maildir/.

When I use dovecot -n to get a canonical view of the configuration I get:

# 2.2.9: /etc/dovecot/dovecot.conf
# OS: Linux 3.2.0-0.bpo.4-amd64 x86_64
log_path = /var/log/dovecot.log
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave
passdb {
  driver = pam
}
ssl = no
userdb {
  driver = passwd
}

That managesieve... stuff is injected from installing the separate pigeonhole package btw.

My Dovecot IMAP configuration goals are:

  • Have the most minimal configuration possible. Every LOC counts!
  • For sent mail to be in the inbox so I can see my replies inline, in the context of a conversation like Gmail
  • To get full text search search working
  • To get DSPAM integrated, since I don't want SPAM and I don't want Perl on my system ideally
  • To get my mailing list traffic in separate folders (I think this is where pigeonhole/sieve comes in), labelled by the listname
  • To get SSL stuff working
  • To be able to access my mail via Apple IOS client on my Iphone

I want to see Dovecot succeed but there are a couple of obvious problems with it:

Posted
dabase
Understanding CoreOS

CoreOS has come up on my radar a couple of times and I decided to see what all the fuss about especially when they were soliciting companies to pilot their software for 5000USD!

All credit to them for managing that.

What's the attraction of running CoreOS over say Archlinux?

  1. Automatic updates keep the base OS of CoreOS secure http://coreos.com/using-coreos/updates/
  2. It runs Docker and Etcd

Ok, automatic updates are easy to understand. Right now I periodically run pacman -Syu across the servers I manage. I update my laptop before my servers and obviously keep a close eye on https://www.archlinux.org/news/. It's not absolutely bleeding edge since I don't use [testing]. For the last years my Archlinux servers have been wonderfully stable and productive. I'm also enjoying Archlinux's pioneering work with systemd, so all my services are now sanely managed!

I am trying to forget the days of using hopelessly out of date software on Debian. For example you can't do interesting stuff with multimedia on Debian if you running ancient versions of ffmpeg.

CoreOS implements an automated upgrade mechanism along the lines of ChromeOS. Tbh I don't have experience of ChromeOS updates, though they must work. My big criticism of this approach is that they might as well use gitfs like Webconverger uses for upgrades. Automatic upgrades are the future, so big thumbs up for me here.

I'm also having a hard time understanding what CoreOS means by "base". In Webconverger, our base is the entire rootfs. With gitfs you can even roll back, as well as get updates.

Is "base" LAMP? I.e. Apache/MariaDB/PHP?! Or all the databases and other crud people think they need to run Node, Rails, Django or some other bloated Web framework? Personally I prefer just one highly tested canonical system (not just base, everything to run that Web app), otherwise things get complex. Supporting different types of configurations via packages sucks, because there is a permutation explosion when it comes to variations in how to do this. Maybe they'll propose a "base system" for Rails, and another for "NodeJS & MongoDB" etc etc. At least that avoids a "permutation explosion" via packages.

I am not so sure if I trust CoreOS developers to make the "base" choices for me. Ultimately people get conservative and start running older and older software. It will probably get bloated too. Suck.

Update: I think the base OS is just for running several instances of say Ubuntu in docker containers. Tbh I thought the containers would inherit the base system OS entirely, but this doesn't seem to be the case.

On a further IRC discussion, Brandon Philips seemed to envision lightweight containers running almost a static build of a Web application:

10:32 <philips> hendry: the applications would talk to etcd. A container isn't
exactly like a virtual machine guest. For example you could have a node.js
application or go application that runs just fine in a container that is just
libc, resolv.conf and a few other required files

Wtf is Docker?

IIUC Docker is a fancy name for LXC aka "chroots done right" (but with a ton more complexity).

So I think the idea is that you git push into a Docker container with the underlying "standardised" / "stable" CoreOS OS and out pops your hosted vhost Web application. I think this is the model OpenStack pushes hard. It's a good model for the unwashed, but I find it a little claustrophobic.

I really miss doing some iterative development from the remote ssh shell. Anyway, most "developers" push out using FTP, so this is a marked improvement, for sure.

And Wtf is etcd?

Etcd as I understand it, is used for two things:

  1. Key value store for configuration data like the Windows Registry, for services running on CoreOS
  2. Some tools to manage the distribution / control of the services

Key, value store is easy to imagine. But how does that map to /etc/nginx/nginx.conf one might be thinking?

Well ideally I'm told that services talk directly with etcd, but if they can't like most Linux services currently, they need to talk via confd, so you end up with something like:

etcd -> confd -> /etc/nginx/nginx.conf

Tbh I don't like this at all. I think you could simply manage service configurations in git and use some sort of hook to expand macros like HOSTNAME or whatever you need machine by machine.

The abstraction or mapping between etcd and a services config file, e.g. /etc/postfix/main.cf is obviously going to be leaky. The only positive point I can think of, is that it maybe over time... services like postfix, nginx, apache et al to talk directly with etcd and gets rid of their particular crufty configuration languages. I.e. make things much simpler for admins!

What about this "tools to manage the distribution / control of the services" feature of etcd

Ok, you would think they would split this out to another tool name. Anyway, as I understand it, etcd also gives you automation abilities.

Lets pretend you are spinning up 30 machines to serve some Web service. Without getting trapped into the AWS suite of tools, you would face lots of problems like:

  1. monitoring the instances
  2. knowing when the instances are online
  3. adding instances to your load balancer

These are the use cases etcd is said to solve. You can't use Avahi for service discovery when VPSes are spinning up around the world, CoreOS proposes Chubby instead.

I'd be interested to see how or if etcd developers will integrate with systemd.

So with that preamble, a good summary is given at the bottom of the http://coreos.com/ page:

  • minimal - linux + systemd (and their base OS they don't mention)
  • containers - chroots ... yay
  • service discovery ... like AWS's pioneering stuff but open

Good luck to the CoreOS team. I will be keeping a very close eye on their developments!

Posted
tips
How to remove a bucket

Using the AWS cli:

$ aws s3 rm --recursive s3://example
delete: s3://example/foo.txt
delete: s3://example/test.jpg
$ aws s3 rb s3://example
remove_bucket: s3://example/
Posted
dabase
Cloning hosts

So, I have unenviable task of moving a VPS from Bitfolk to Linode.

Both run Debian 7 (wheezy).

So the general practice I find is to rebuild from scratch. Argh, doing it by hand takes a long time.

Doing it with Configuration Management would be quicker, if you bothered to spend even MORE TIME developing the recipes to setup the machine. Nevermind the crap load of code that goes into running Puppet or Ansible and their crazy crazy dependencies.

Configuration management for user accounts sounds quite tricky

Previously I've something like create-account-with-ssh-pub.sh, a short ~20 line shell script.

Followed up for disabling password authentication / only allowing key based authentication with PasswordAuthentication No in /etc/ssh/sshd_config.

I recommend maintaining a list of your users ssh public keys online to make things easier, e.g. http://hendry.iki.fi/hendry.pub

Doing this in any CM tool seems absolutely non-trivial!!

Enter rsync

In an ideal world I could just rsync everything as root. But as this resource points out, there are loads of traps. Many mount points that shouldn't be copied over. Network configurations and perhaps VPS specific settings that cannot be overwritten.

root@li524-199:~# cat exclude.txt
/boot
/proc
/sys
/tmp
/dev
/var/lock
/etc/fstab
/etc/mdadm.conf
/etc/mtab
/etc/resolv.conf
/etc/conf.d/net
/etc/network/interfaces
/etc/networks
/etc/hostname
/etc/HOSTNAME
/etc/hosts
/etc/modprobe*
/etc/modules
/etc/udev
/lib/modules
root@li524-199:~# cat rsync.sh
rsync -aPzx --exclude-from=/root/exclude.txt root@uk.webconverger.com:/ /
Posted
tips
Download entire public S3 bucket without credentials

Say for example you have a S3 bucket named s3://p-dl/

If you have made your bucket public (Hint: s3cmd -P) it should reachable upon http://p-dl.s3.amazonaws.com, which may again redirect you to http://p-dl.s3-ap-southeast-1.amazonaws.com/.

Now you should see an XML listing like:

<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Name>p-dl</Name>
<Prefix/>
<Marker/>
<MaxKeys>1000</MaxKeys>
<IsTruncated>false</IsTruncated>
<Contents>
<Key>foo.txt</Key>
<LastModified>2013-10-09T09:12:53.000Z</LastModified>
<ETag>"59ca0efa9f5633cb0371bbc0355478d8"</ETag>
<Size>13</Size>
<StorageClass>REDUCED_REDUNDANCY</StorageClass>
</Contents>
<Contents>
<Key>test.jpg</Key>
<LastModified>2013-10-09T09:12:53.000Z</LastModified>
<ETag>"c9af9358e016984c1445a6102b4c35fc"</ETag>
<Size>89761</Size>
<StorageClass>REDUCED_REDUNDANCY</StorageClass>
</Contents>
</ListBucketResult>

If not, perhaps you are missing the Everyone List permission.

You can download all the resources with a bit of xmlstarlet magic like so:

s3url=http://p-dl.s3-ap-southeast-1.amazonaws.com
curl -s $s3url |
xml sel -N w="http://s3.amazonaws.com/doc/2006-03-01/" -T -t -m "//w:Key" -v . -n |
while read filename
do
    wget -cN $s3url/$filename
done

Is there an easier way? Please let me know. It would be good if there was a JSON API.

Posted
dabase
Cloning a disk

So I bought a "SanDisk SDSSDHP-128G-G25 128GB Ultra Plus SATA 3.0 6GB/s 7mm Internal SSD" aka "sandisk" to complement my existing "OCZ 120GB Nocti SSD - mSATA SATA-II - Read 280MB/s Write 260MB/s" aka "nocti" disk connected to my Mini-SATA port.

I primarily thought I would experiment with cloning nocti to the sandisk, so that if nocti failed I could use sandisk. Or I could see performance differences running the same system. Or I could do fun things like upgrade one, but not the other and see if I could measure or debug something.

Straight up dd

Note: In hindsight you are almost always better off mkfs.ext4 and rsync -a files over instead off dd!

I first tried, whilst running and mounted on /dev/sdb (nocti):

dd if=/dev/sdb | pv | dd of=/dev/sda

pv supposed to "monitor the progress of data through a pipe", however it didn't give me a proper progress bar. It only told me that I was copying at a rate of about 10MB/sec, which I found a bit disappointing.

# parted --list | grep .
Model: ATA SanDisk SDSSDH21 (scsi)
Disk /dev/sda: 128GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:
Number  Start   End    Size   Type     File system  Flags
 1      32.3kB  120GB  120GB  primary  ext4         boot
Model: ATA OCZ-NOCTI (scsi)
Disk /dev/sdb: 120GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:
Number  Start   End    Size   Type     File system  Flags
 1      32.3kB  120GB  120GB  primary  ext4         boot

First problem is that I'm wasting 8GB. Second problem is that I'm dd-ing from a running system. Lets try boot into sandisk!

Uh oh, lots of errors. dd simply didn't work, even after running a fsck upon it. Also scarily, the "sandisk" was referring to itself as "nocti" when I was using it. That's because when I dd-ed it, the label came across:

x220:~$ sudo e2label /dev/sda1
nocti
x220:~$ sudo e2label /dev/sdb1
nocti

Sadly one can't do a top level label, you have to do it on a partition:

sudo e2label /dev/sda sandisk
e2label: Bad magic number in super-block while trying to open /dev/sdb
Couldn't find valid filesystem superblock.

Lets try a dd from a live USB like Webconverger

In 20 minutes I successfully managed to clone "nocti" to "sandisk" using 4M as the block size. I ran e2label /dev/sda1 sandisk and now we have:

$ lsblk -f
NAME   FSTYPE LABEL   UUID                                 MOUNTPOINT
sda
└─sda1 ext4   sandisk 5d40f591-086f-407a-9b6c-fb3fec1dc046 /
sdb
└─sdb1 ext4   nocti   5d40f591-086f-407a-9b6c-fb3fec1dc046

Notice they have the exact same UUID.

hdparm to compare disk speed

So the sandisk shows:

# hdparm -tT /dev/sda
/dev/sda:
 Timing cached reads:   6672 MB in  2.00 seconds = 3336.70 MB/sec
 Timing buffered disk reads: 1466 MB in  3.00 seconds = 488.37 MB/sec

And nocti is:

[root@x220 ~]# hdparm -tT /dev/sdb
/dev/sdb:
 Timing cached reads:   10298 MB in  2.00 seconds = 5152.28 MB/sec
 Timing buffered disk reads: 338 MB in  3.01 seconds = 112.21 MB/sec

I added "noatime,discard" to my mount options and sda seems faster or hdparm is just variable?

[root@x220 ~]# hdparm -tT /dev/sda
/dev/sda:
 Timing cached reads:   10688 MB in  2.00 seconds = 5347.33 MB/sec
 Timing buffered disk reads: 1368 MB in  3.00 seconds = 455.69 MB/sec
[root@x220 ~]# hdparm -tT /dev/sdb
/dev/sdb:
 Timing cached reads:   10156 MB in  2.00 seconds = 5081.24 MB/sec
 Timing buffered disk reads: 338 MB in  3.01 seconds = 112.13 MB/sec

Next, expanding the sandisk to take all the space by recreating the partition (BAD IDEA)

Parted seems unable to detect the dimensions of the disk for partition creation, hence I used fdisk.

# fdisk /dev/sda
Welcome to fdisk (util-linux 2.23.2).

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1):
First sector (2048-250069679, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-250069679, default 250069679):
Using default value 250069679
Partition 1 of type Linux and of size 119.2 GiB is set

Command (m for help): p

Disk /dev/sda: 128.0 GB, 128035676160 bytes, 250069680 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xba6fb7fd

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1            2048   250069679   125033816   83  Linux

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
# resize2fs /dev/sda1
resize2fs 1.42.8 (20-Jun-2013)
resize2fs: Bad magic number in super-block while trying to open /dev/sda1
Couldn't find valid filesystem superblock.

I think the problem is that fdisk seems to think the start is at default 2048, when looking back to original parted output after the dd, it's in fact 32.3kB. I do not know how to specify 32.3kB as the start for fdisk.

WARNING: http://askubuntu.com/a/115337/676 is bad advice, since fdisk will forcibly start at 2048 (default since fdisk 2.17.2), and will not allow to start at a lower sector!

SATA speeds

On my X220 system, I have a number of SATA ports:

# ls /sys/class/ata_port/
ata1  ata2  ata3  ata4  ata5  ata6

To determine the speed:

# dmesg | grep -i sata | grep 'link up'
ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)

IIUC, sandisk is on ata1:

# journalctl -b | grep -i sandisk | grep ata
Oct 07 15:26:41 x220 kernel: ata1.00: ATA-8: SanDisk SDSSDH2128G, X211200, max UDMA/133

And nocti is on ata3:

# journalctl -b | grep -i nocti | grep at
Oct 07 15:26:41 x220 kernel: ata3.00: ATA-8: OCZ-NOCTI, 2.15, max UDMA/133

Finally, the right way to clone a disk

The best approach I've found is to boot Archlinux install disk. Assuming your destination is sandisk at /dev/sda and source is nocti at /dev/sdb.

lsblk -f
fdisk /dev/sda
n, p, enter, enter # Create partition
mkfs.ext4 /dev/sda1
e2label /dev/sda1 sandisk
mkdir /mnt/{nocti,sandisk}
mount /dev/sdb1 /mnt/nocti
mount /dev/sda1 /mnt/sandisk -o noatime,rw,discard
rsync -a /dev/nocti/ /mnt/sandisk
vim /mnt/sandisk/etc/fstab # update LABEL=sandisk
vim /mnt/sandisk/boot/syslinux/syslinux.cfg # update root=LABEL=sandisk
syslinux-install_update.sh -i -a -m -c /mnt/sandisk
Posted
tips
Steps to make a S3 hosted Git repository

Lets create a bucket from the command line using the AWS CLI S3 commands:

$ aws s3 mb s3://monitortest
~/tmp/monitor/.git$ git update-server-info
~/tmp/monitor/.git$ cd ..
~/tmp/monitor$ aws s3 sync .git/ s3://monitortest
upload: .git/objects/info/packs to s3://monitortest/objects/info/packs
upload: .git/info/refs to s3://monitortest/info/refs

WARNING: aws s3 sync can hose your machine on a big transfer

Now comes the painful bit since there is no way to set permissions this from aws cli, where you need to go to https://console.aws.amazon.com/s3 to:

  1. Add permissions
  2. Add more permissions
  3. Change site to Static Website Hosting

If everything goes well, you will be able to do something like this:

$ git clone http://monitortest.s3-website-ap-southeast-1.amazonaws.com
Cloning into 'monitortest.s3-website-ap-southeast-1.amazonaws.com'...
Checking connectivity... done

If you don't, you will get 403 errors. I'm aiming to use S3 as a backup for my family photographs.

Posted
dabase
Google groups is Kafkaesque

There are plenty of Google Groups rants out there, here is mine:

  • 99% of SPAM is Chinese, Non-English and non-primary language as I've configured the group. Still it comes.
  • The buttons to "Report Spam" and to let the Spam through and subsequent spams are NIGHTMARISHLY close together

There is NO WAY to give Google Groups admins feedback. WTF?!

The seemingly only way to offer feedback is to become a Google Top contributor... no thanks.

And it's so hard to migrate away from Google services. http://www.dataliberation.org/ is a sick joke.

Posted
tips
AWS Route 53 health check hostname

Failing AWS health check

Notice the Host Name with http://../ will fail:

curl -H 'Host: http://browserinabox.com/' http://117.121.241.187:80
<html>
<head><title>400 Bad Request</title></head>
<body bgcolor="white">
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx/1.4.2</center>
</body>
</html>

However:

curl -H 'Host: browserinabox.com' http://117.121.241.187:80
<h1>Hello World</h1>

Will work. Update the Host Name field to remove the http syntax and this is what a good health check looks like:

Health AWS health check

Do be aware that the Amazon Route 53 Health Check Service checks very aggressively compared to a bog standard nagios check!

Amazon Route 53 Health Check Service logs

Posted
tips
Nginx 1.4 Virtual hosting

Aiming here to replicate Apache's VirtualDocumentRoot

This is mandatory otherwise nginx won't start. Pretty bizaare:

events {
    worker_connections  1024;
}

This is the Archlinux default:

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile      on;

server {
    charset utf-8;
    listen 80;
    autoindex on;

Here we map the requested domain to /srv/www/$vhost

    server_name ~^(?<vhost>.*)$;
    root /srv/www/$vhost;
    index  index.html index.php index.cgi index.txt;
    access_log /var/log/nginx/$vhost.access.log;

To get PHP/CGI working, it's you need this with php-fpm.service & fcgiwrap.service going beforehand.

    location ~ \.php$ {
        try_files      $uri = 404;
        fastcgi_pass   unix:/run/php-fpm/php-fpm.sock;
        fastcgi_index  index.php;
        include        fastcgi.conf;
    }

    location ~ \.cgi$ {
        include        fastcgi.conf;
        fastcgi_pass   unix:/run/fcgiwrap.sock;
    }
}
}
Posted
tips
Losing power whilst recording with Audacity

After launching Audacity, selecting the autosave you might be greeted with a "not well formed - invalid token" error. That means the XML autosave is corrupted.

Under ~/.audacity-data/AutoSave you will probably find an invalid file like so:

~/.audacity-data/AutoSave$ xmlstarlet val cryptoparty2.autosave 
cryptoparty2.autosave - invalid

Open the file in vim, go to the bottom with the G shortcut and hopefully you will notice some obvious gibberish like:

^@^@^@^@^@^@^@^@^@</project>

Remove it, validate the XML is good. Now you should be able to open it in Audacity.

Posted
tips
Could not copy message

If you get "Could not copy message" or "Can't open PGP subprocess!: No such file or directory (errno = 2)", it means your mutt and GPG are not configured to work with one another.

The solution is to source from your ~/.muttrc a mutt/gpg configuration usually in your mutt package. On Archlinux add to your ~/.muttrc:

source /etc/Muttrc.gpg.dist
Posted
  • flickr: last checked (1 posts)
  • geekout: Not Found (0 posts)
  • twitter: feed not found (4783 posts)
  • dabase: Not Found (39 posts)
  • tips: Not Found (36 posts)
  • android: Not Found (4 posts)
  • mwts: Not Found (5 posts)
  • flickrcomments: last checked (14 posts)