Apt is surprisingly flexible

After a break for a few months, I just shipped a new version of polysquare-travis-container. The main difference here is that we are now able to create and maintain containers without using proot at all, which is a slight improvement on the last big round of changes made in August.

The initial reason for using proot was to provide a simple way to fool apt and dpkg into thinking that it was running from the root directory when it was actually running from a directory owned by user. The theory goes that if you can fool apt into doing that, then you can install and run packages built for other distributions without the overhead of running virtual machines. As explained about two years ago, the best solutions we have available for doing this are chroot (and wrappers around chroot) and Docker. However both of those require root access to make the chroot system call and/or set up cgroups. proot solves that problem by running your program through ptrace, then intercepting and rewriting system calls such that programs think they are running on the root directory.

However, as time went on, I encountered problems with the proot approach. Mixing redirection with environment variables like PATH tended to not work out so well. On newer ubuntu releases proot ran incredibly slowly. Then, finally, it just stopped working at all on Travis-CI, which kind of defeats the purpose of using it in polysquare-travis-container.

Earlier in 2015 I realised the solution was to take the same approach used by polysquare-travis-container’s support for Windows and macOS – just set the right environment variables and support the packages which do the right thing and don’t hardcode absolute paths. This has worked out surprisingly well. Supporting linux package managers was far trickier. I suppose a part of the problem here was that they have historically always run as the root user and as a result wrote directly to /usr and kept their data in /var. After all – they are there to manage the entire filesystem, so having those assumptions makes sense.

After the changes made in August, I wasn’t too optimistic that it would be possible to run these package managers without using chroot or a chroot-alternative. But after some digging I found that Apt itself has a test suite which has to run under the exact same constraints. As it turns out, pretty much every path in Apt is configurable to some extent, so much so that with an Apt configuration like below, you can run apt as a non-root user and have it keep all its changes in a specified directory.

Apt {
    Architecture "amd64";
    Get {
        Assume-Yes true;
    };
};
debug {
    nolocking true;
};
Acquire::Queue-Mode "host";
Dir "fakeroot";
Dir::Cache "fakeroot/var/cache/apt";
Dir::State "fakeroot/var/lib/apt";
Dir::State::status "fakeroot/var/lib/dpkg/status";
Dir::Bin::Solvers "fakeroot/usr/lib/apt/solvers";
Dir::Bin::Planners "fakeroot/usr/lib/apt/planners";
Dir::Bin::Solvers "fakeroot/usr/lib/apt/solvers";
Dir::Bin::Methods "fakeroot/usr/lib/apt/methods";
Dir::Bin::Dpkg "fakeroot/usr/bin/dpkg.w";
Dir::Etc "fakeroot/etc/apt";
Dir::Log "fakeroot/var/log/apt";

Along with Apt, you’ll also need to tell Dpkg to run in a separate root directory. Thankfully, it has command line options to tell it to do this. The only problem is that Apt invokes the Dpkg binary on occasion and so you’ll need to write a wrapper script to ensure that Dpkg gets called with the right command line arguments.

#!/bin/bash
fakeroot/usr/bin/dpkg --root='fakeroot' \
--admindir=fakeroot/var/lib/dpkg \
--log=fakeroot/var/log/dkpkg.log \
--force-not-root --force-bad-path $@

You’ll notice above that I override the Dpkg binary with dpkg.w which contains the script contents above.

The final piece of the puzzle was to disable postinst, postrm and prerm scripts from running. These aren’t necessary in these containers since they are mostly responsible for updating things like system caches or updating configuration files. The containers are meant to be one-off environments so all we care about are the binaries. Disabling them was as simple as removing them.

With all of that effort out of the way, we can now create and run a container without the use of docker, chroot or proot and install a completely separate toolchain and run binaries from it.

$ psq-travis-container-create --distro Ubuntu --release precise --arch x86_64 container --local --packages PACKAGES --repositories REPOSITORIES

Configured Distribution:
 - Release: precise
 - Package System: DpkgLocal
 - Architecture: x86_64
 - Distribution Name: Ubuntu
✓ Using existing folder for proot distro Ubuntu precise amd64
-> Update repositories [apt-get update -y --force-yes]
   Get:1 http://archive.ubuntu.com precise Release.gpg [198 B]
   Get:2 http://archive.ubuntu.com precise-security Release.gpg [198 B]
   Get:3 http://archive.ubuntu.com precise-updates Release.gpg [198 B]
   Get:4 http://ppa.launchpad.net precise Release.gpg [316 B]
   Get:5 http://ppa.launchpad.net precise Release.gpg [316 B]
   Get:6 http://archive.ubuntu.com precise Release [49.6 kB]
   Get:7 http://ppa.launchpad.net precise Release [13.0 kB]
   Ign http://ppa.launchpad.net precise Release
   Get:8 http://llvm.org llvm-toolchain-precise-3.6 Release.gpg [836 B]
   Get:9 http://ppa.launchpad.net precise Release [12.9 kB]
   Ign http://ppa.launchpad.net precise Release
   Get:10 http://archive.ubuntu.com precise-security Release [55.5 kB]
   Get:11 http://ppa.launchpad.net precise/main amd64 Packages [592 B]
   Get:12 http://ppa.launchpad.net precise/main TranslationIndex [196 B]
   Get:13 http://archive.ubuntu.com precise-updates Release [55.4 kB]
   Get:14 http://ppa.launchpad.net precise/main amd64 Packages [49.1 kB]
   Get:15 http://llvm.org llvm-toolchain-precise-3.6 Release [3,355 B]
   Ign http://llvm.org llvm-toolchain-precise-3.6 Release
   Get:16 http://archive.ubuntu.com precise/main Sources [934 kB]
   Get:17 http://ppa.launchpad.net precise/main TranslationIndex [205 B]
   Get:18 http://ppa.launchpad.net precise/main Translation-en [310 B]
   Get:19 http://ppa.launchpad.net precise/main Translation-en [15.2 kB]
   Get:20 http://archive.ubuntu.com precise/restricted Sources [5,470 B]
   Get:21 http://archive.ubuntu.com precise/main amd64 Packages [1,273 kB]
   Ign http://llvm.org llvm-toolchain-precise-3.6/main TranslationIndex
   Get:22 http://archive.ubuntu.com precise/restricted amd64 Packages [8,452 B]
   Get:23 http://archive.ubuntu.com precise/main TranslationIndex [3,706 B]
   Get:24 http://archive.ubuntu.com precise/restricted TranslationIndex [2,596 B]
   Get:25 http://archive.ubuntu.com precise-security/main Sources [146 kB]
   Get:26 http://archive.ubuntu.com precise-security/restricted Sources [4,623 B]
   Get:27 http://archive.ubuntu.com precise-security/main amd64 Packages [664 kB]
   Get:28 http://archive.ubuntu.com precise-security/restricted amd64 Packages [10.8 kB]
   Get:29 http://archive.ubuntu.com precise-security/main TranslationIndex [208 B]
   Get:30 http://archive.ubuntu.com precise-security/restricted TranslationIndex [202 B]
   Get:31 http://archive.ubuntu.com precise-updates/main Sources [500 kB]
   Get:32 http://archive.ubuntu.com precise-updates/restricted Sources [8,840 B]
   Get:33 http://archive.ubuntu.com precise-updates/main amd64 Packages [1,045 kB]
   Get:34 http://archive.ubuntu.com precise-updates/restricted amd64 Packages [15.4 kB]
   Get:35 http://archive.ubuntu.com precise-updates/main TranslationIndex [208 B]
   Get:36 http://archive.ubuntu.com precise-updates/restricted TranslationIndex [202 B]
   Get:37 http://archive.ubuntu.com precise/main Translation-en_AU [4,434 B]
   Get:38 http://archive.ubuntu.com precise/main Translation-en [726 kB]
   Get:39 http://archive.ubuntu.com precise/restricted Translation-en_AU [2,407 B]
   Get:40 http://archive.ubuntu.com precise/restricted Translation-en [2,395 B]
   Get:41 http://archive.ubuntu.com precise-security/main Translation-en [269 kB]
   Get:42 http://archive.ubuntu.com precise-security/restricted Translation-en [2,793 B]
   Get:43 http://archive.ubuntu.com precise-updates/main Translation-en [431 kB]
   Get:44 http://archive.ubuntu.com precise-updates/restricted Translation-en [3,682 B]
   Get:45 http://llvm.org llvm-toolchain-precise-3.6/main amd64 Packages [6,216 B]
   Ign http://llvm.org llvm-toolchain-precise-3.6/main Translation-en_AU
   Ign http://llvm.org llvm-toolchain-precise-3.6/main Translation-en
   Fetched 6,328 kB in 16s (389 kB/s)
   Reading package lists...
   W: GPG error: http://ppa.launchpad.net precise Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY C1DB487B944B6EA7
   W: GPG error: http://ppa.launchpad.net precise Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 1E9377A2BA9EF27F
   W: GPG error: http://llvm.org llvm-toolchain-precise-3.6 Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 15CF4D18AF4F7421
-> Downloading APT packages and dependencies [apt-get -y --force-yes -d install --reinstall nano cmake clang-3.6]
   Reading package lists...
   Building dependency tree...
   Reading state information...
   The following extra packages will be installed:
     adduser binutils bsdmainutils busybox-initramfs ca-certificates cmake-data
     cpio emacsen-common gcc-4.9-base gcc-6-base ifupdown initramfs-tools
     initramfs-tools-bin initscripts insserv iproute klibc-utils libarchive12
     libasan1 libasn1-8-heimdal libatomic1 libblkid1 libbsd0 libc-bin
     libc-dev-bin libc6 libc6-dev libcilkrts5 libclang-common-3.6-dev
     libclang1-3.6 libcomerr2 libcurl3 libcurl3-gnutls libdbus-1-3 libdrm-intel1
     libdrm-nouveau1a libdrm-radeon1 libdrm2 libedit2 libexpat1 libgcc-4.9-dev
     libgcc1 libgcrypt11 libglib2.0-0 libgnutls26 libgomp1 libgpg-error0
     libgssapi-krb5-2 libgssapi3-heimdal libhcrypto4-heimdal libheimbase1-heimdal
     libheimntlm0-heimdal libhx509-5-heimdal libidn11 libitm1 libk5crypto3
     libkeyutils1 libklibc libkrb5-26-heimdal libkrb5-3 libkrb5support0
     libldap-2.4-2 libllvm3.6 liblsan0 libmount1 libncurses5 libncursesw5
     libnettle4 libnih-dbus1 libnih1 libobjc-4.9-dev libobjc4 libp11-kit0
     libpciaccess0 libpcre3 libplymouth2 libpng12-0 libquadmath0
     libroken18-heimdal librtmp0 libsasl2-2 libslang2 libsqlite3-0 libssl1.0.0
     libstdc++-4.9-dev libstdc++6 libtasn1-3 libtsan0 libubsan0 libudev0 libuuid1
     libwind0-heimdal libxml2 libxmlrpc-core-c3 linux-libc-dev lsb-base
     module-init-tools mount mountall ncurses-bin openssl passwd plymouth procps
     sysv-rc sysvinit-utils udev upstart util-linux
   Suggested packages:
     liblocale-gettext-perl perl-modules binutils-doc cpp wamerican wordlist
     whois vacation gnustep gnustep-devel clang-3.6-doc gcc make libarchive1
     isc-dhcp-client dhcp-client ppp rdnssd net-tools bash-completion bootchart
     iproute-doc glibc-doc locales rng-tools gnutls-bin krb5-doc krb5-user
     pciutils libstdc++-4.9-doc nfs-common spell sysv-rc-conf bum sash watershed
     graphviz util-linux-locales kbd console-tools dosfstools
   Recommended packages:
     ecryptfs-utils llvm-3.6-dev python psmisc e2fsprogs libatm1 manpages-dev
     dbus libglib2.0-data shared-mime-info krb5-locales libgpm2 libsasl2-modules
     uuid-runtime xml-core plymouth-theme-ubuntu-text plymouth-theme
   The following NEW packages will be installed:
     adduser binutils bsdmainutils busybox-initramfs ca-certificates clang-3.6
     cmake cmake-data cpio emacsen-common gcc-4.9-base gcc-6-base ifupdown
     initramfs-tools initramfs-tools-bin initscripts insserv iproute klibc-utils
     libarchive12 libasan1 libasn1-8-heimdal libatomic1 libblkid1 libbsd0
     libc-dev-bin libc6-dev libcilkrts5 libclang-common-3.6-dev libclang1-3.6
     libcomerr2 libcurl3 libcurl3-gnutls libdbus-1-3 libdrm-intel1
     libdrm-nouveau1a libdrm-radeon1 libdrm2 libedit2 libexpat1 libgcc-4.9-dev
     libgcrypt11 libglib2.0-0 libgnutls26 libgomp1 libgpg-error0 libgssapi-krb5-2
     libgssapi3-heimdal libhcrypto4-heimdal libheimbase1-heimdal
     libheimntlm0-heimdal libhx509-5-heimdal libidn11 libitm1 libk5crypto3
     libkeyutils1 libklibc libkrb5-26-heimdal libkrb5-3 libkrb5support0
     libldap-2.4-2 libllvm3.6 liblsan0 libmount1 libncurses5 libncursesw5
     libnettle4 libnih-dbus1 libnih1 libobjc-4.9-dev libobjc4 libp11-kit0
     libpciaccess0 libpcre3 libplymouth2 libpng12-0 libquadmath0
     libroken18-heimdal librtmp0 libsasl2-2 libslang2 libsqlite3-0 libssl1.0.0
     libstdc++-4.9-dev libtasn1-3 libtsan0 libubsan0 libudev0 libuuid1
     libwind0-heimdal libxml2 libxmlrpc-core-c3 linux-libc-dev lsb-base
     module-init-tools mount mountall nano ncurses-bin openssl passwd plymouth
     procps sysv-rc sysvinit-utils udev upstart util-linux
   The following packages will be upgraded:
     libc-bin libc6 libgcc1 libstdc++6
   4 to upgrade, 108 to newly install, 0 to remove and 18 not to upgrade.
   Need to get 96.1 MB of archives.
   After this operation, 260 MB of additional disk space will be used.
   WARNING: The following packages cannot be authenticated!
     gcc-6-base libgcc1 libstdc++6 gcc-4.9-base libasan1 libatomic1 libcilkrts5
     libllvm3.6 libgomp1 libitm1 liblsan0 libtsan0 libubsan0 libquadmath0
     libgcc-4.9-dev libstdc++-4.9-dev libobjc4 libobjc-4.9-dev libclang1-3.6
     libclang-common-3.6-dev clang-3.6
   Get:1 http://archive.ubuntu.com/ubuntu/ precise-security/main libc-bin amd64 2.15-0ubuntu10.15 [1,177 kB]
   Get:2 http://ppa.launchpad.net//ubuntu-toolchain-r/test/ubuntu/ precise/main gcc-6-base amd64 6.2.0-3ubuntu11~12.04 [18.1 kB]
   Get:3 http://ppa.launchpad.net//ubuntu-toolchain-r/test/ubuntu/ precise/main libgcc1 amd64 1:6.2.0-3ubuntu11~12.04 [44.6 kB]
   Get:4 http://ppa.launchpad.net//ubuntu-toolchain-r/test/ubuntu/ precise/main libstdc++6 amd64 6.2.0-3ubuntu11~12.04 [391 kB]
   Get:5 http://llvm.org/apt/precise/ llvm-toolchain-precise-3.6/main libllvm3.6 amd64 1:3.6.2~svn240577-1~exp1 [11.5 MB]
   Get:6 http://archive.ubuntu.com/ubuntu/ precise-security/main libc6 amd64 2.15-0ubuntu10.15 [4,636 kB]
   Get:7 http://ppa.launchpad.net//ubuntu-toolchain-r/test/ubuntu/ precise/main gcc-4.9-base amd64 4.9.4-2ubuntu1~12.04 [16.9 kB]
   Get:8 http://ppa.launchpad.net//ubuntu-toolchain-r/test/ubuntu/ precise/main libasan1 amd64 4.9.4-2ubuntu1~12.04 [240 kB]
   Get:9 http://ppa.launchpad.net//ubuntu-toolchain-r/test/ubuntu/ precise/main libatomic1 amd64 6.2.0-3ubuntu11~12.04 [10.8 kB]
   Get:10 http://ppa.launchpad.net//ubuntu-toolchain-r/test/ubuntu/ precise/main libcilkrts5 amd64 6.2.0-3ubuntu11~12.04 [49.6 kB]
   Get:11 http://ppa.launchpad.net//ubuntu-toolchain-r/test/ubuntu/ precise/main libgomp1 amd64 6.2.0-3ubuntu11~12.04 [85.6 kB]
   Get:12 http://ppa.launchpad.net//ubuntu-toolchain-r/test/ubuntu/ precise/main libitm1 amd64 6.2.0-3ubuntu11~12.04 [34.3 kB]
   Get:13 http://ppa.launchpad.net//ubuntu-toolchain-r/test/ubuntu/ precise/main liblsan0 amd64 6.2.0-3ubuntu11~12.04 [136 kB]
   Get:14 http://ppa.launchpad.net//ubuntu-toolchain-r/test/ubuntu/ precise/main libtsan0 amd64 6.2.0-3ubuntu11~12.04 [324 kB]
   Get:15 http://ppa.launchpad.net//ubuntu-toolchain-r/test/ubuntu/ precise/main libubsan0 amd64 6.2.0-3ubuntu11~12.04 [125 kB]
   Get:16 http://ppa.launchpad.net//ubuntu-toolchain-r/test/ubuntu/ precise/main libquadmath0 amd64 6.2.0-3ubuntu11~12.04 [146 kB]
   Get:17 http://ppa.launchpad.net//ubuntu-toolchain-r/test/ubuntu/ precise/main libgcc-4.9-dev amd64 4.9.4-2ubuntu1~12.04 [3,761 kB]
   Get:18 http://archive.ubuntu.com/ubuntu/ precise-security/main libdbus-1-3 amd64 1.4.18-1ubuntu1.8 [146 kB]
   Get:19 http://archive.ubuntu.com/ubuntu/ precise-updates/main libnih1 amd64 1.0.3-4ubuntu9.1 [54.8 kB]
   Get:20 http://archive.ubuntu.com/ubuntu/ precise-updates/main libnih-dbus1 amd64 1.0.3-4ubuntu9.1 [16.0 kB]
   Get:21 http://llvm.org/apt/precise/ llvm-toolchain-precise-3.6/main libclang1-3.6 amd64 1:3.6.2~svn240577-1~exp1 [5,398 kB]
   Get:22 http://archive.ubuntu.com/ubuntu/ precise-updates/main libudev0 amd64 175-0ubuntu9.10 [27.8 kB]
   Get:23 http://archive.ubuntu.com/ubuntu/ precise-updates/main sysvinit-utils amd64 2.88dsf-13.10ubuntu11.1 [60.2 kB]
   Get:24 http://archive.ubuntu.com/ubuntu/ precise/main insserv amd64 1.14.0-2.1ubuntu2 [50.9 kB]
   Get:25 http://archive.ubuntu.com/ubuntu/ precise-updates/main sysv-rc all 2.88dsf-13.10ubuntu11.1 [44.6 kB]
   Get:26 http://archive.ubuntu.com/ubuntu/ precise/main ncurses-bin amd64 5.9-4 [151 kB]
   Get:27 http://archive.ubuntu.com/ubuntu/ precise-updates/main lsb-base all 4.0-0ubuntu20.3 [10.5 kB]
   Get:28 http://archive.ubuntu.com/ubuntu/ precise-security/main libpcre3 amd64 8.12-4ubuntu0.2 [149 kB]
   Get:29 http://archive.ubuntu.com/ubuntu/ precise-updates/main libglib2.0-0 amd64 2.32.4-0ubuntu1 [1,200 kB]
   Get:30 http://archive.ubuntu.com/ubuntu/ precise/main module-init-tools amd64 3.16-1ubuntu2 [105 kB]
   Get:31 http://archive.ubuntu.com/ubuntu/ precise-security/main initramfs-tools-bin amd64 0.99ubuntu13.5 [9,782 B]
   Get:32 http://archive.ubuntu.com/ubuntu/ precise/main libklibc amd64 1.5.25-1ubuntu2 [45.7 kB]
   Get:33 http://archive.ubuntu.com/ubuntu/ precise/main klibc-utils amd64 1.5.25-1ubuntu2 [181 kB]
   Get:34 http://archive.ubuntu.com/ubuntu/ precise-updates/main busybox-initramfs amd64 1:1.18.5-1ubuntu4.1 [183 kB]
   Get:35 http://archive.ubuntu.com/ubuntu/ precise-security/main cpio amd64 2.11-7ubuntu3.2 [116 kB]
   Get:36 http://archive.ubuntu.com/ubuntu/ precise/main libncurses5 amd64 5.9-4 [114 kB]
   Get:37 http://archive.ubuntu.com/ubuntu/ precise/main libslang2 amd64 2.2.4-3ubuntu1 [503 kB]
   Get:38 http://archive.ubuntu.com/ubuntu/ precise-updates/main libblkid1 amd64 2.20.1-1ubuntu3.1 [73.7 kB]
   Get:39 http://archive.ubuntu.com/ubuntu/ precise-updates/main libmount1 amd64 2.20.1-1ubuntu3.1 [71.5 kB]
   Get:40 http://archive.ubuntu.com/ubuntu/ precise-updates/main mount amd64 2.20.1-1ubuntu3.1 [166 kB]
   Get:41 http://archive.ubuntu.com/ubuntu/ precise-updates/main util-linux amd64 2.20.1-1ubuntu3.1 [596 kB]
   Get:42 http://ppa.launchpad.net//ubuntu-toolchain-r/test/ubuntu/ precise/main libstdc++-4.9-dev amd64 4.9.4-2ubuntu1~12.04 [1,867 kB]
   Get:43 http://archive.ubuntu.com/ubuntu/ precise-security/main initramfs-tools all 0.99ubuntu13.5 [49.0 kB]
   Get:44 http://archive.ubuntu.com/ubuntu/ precise/main libncursesw5 amd64 5.9-4 [137 kB]
   Get:45 http://archive.ubuntu.com/ubuntu/ precise-updates/main procps amd64 1:3.2.8-11ubuntu6.4 [233 kB]
   Get:46 http://archive.ubuntu.com/ubuntu/ precise/main adduser all 3.113ubuntu2 [133 kB]
   Get:47 http://archive.ubuntu.com/ubuntu/ precise-updates/main udev amd64 175-0ubuntu9.10 [324 kB]
   Get:48 http://archive.ubuntu.com/ubuntu/ precise-security/main libdrm2 amd64 2.4.52-1~precise2 [26.1 kB]
   Get:49 http://archive.ubuntu.com/ubuntu/ precise-updates/main libpciaccess0 amd64 0.12.902-1ubuntu0.2 [20.8 kB]
   Get:50 http://archive.ubuntu.com/ubuntu/ precise-security/main libdrm-intel1 amd64 2.4.52-1~precise2 [65.6 kB]
   Get:51 http://archive.ubuntu.com/ubuntu/ precise-security/main libdrm-nouveau1a amd64 2.4.52-1~precise2 [14.0 kB]
   Get:52 http://archive.ubuntu.com/ubuntu/ precise-security/main libdrm-radeon1 amd64 2.4.52-1~precise2 [27.8 kB]
   Get:53 http://archive.ubuntu.com/ubuntu/ precise-security/main libpng12-0 amd64 1.2.46-3ubuntu4.2 [133 kB]
   Get:54 http://archive.ubuntu.com/ubuntu/ precise-updates/main libplymouth2 amd64 0.8.2-2ubuntu31.1 [92.0 kB]
   Get:55 http://archive.ubuntu.com/ubuntu/ precise-updates/main plymouth amd64 0.8.2-2ubuntu31.1 [123 kB]
   Get:56 http://archive.ubuntu.com/ubuntu/ precise-updates/main mountall amd64 2.36.4ubuntu0.1 [67.5 kB]
   Get:57 http://archive.ubuntu.com/ubuntu/ precise-updates/main initscripts amd64 2.88dsf-13.10ubuntu11.1 [28.1 kB]
   Get:58 http://archive.ubuntu.com/ubuntu/ precise-updates/main iproute amd64 20111117-1ubuntu2.3 [444 kB]
   Get:59 http://archive.ubuntu.com/ubuntu/ precise-updates/main ifupdown amd64 0.7~beta2ubuntu11.1 [48.3 kB]
   Get:60 http://archive.ubuntu.com/ubuntu/ precise-updates/main upstart amd64 1.5-0ubuntu7.3 [309 kB]
   Get:61 http://archive.ubuntu.com/ubuntu/ precise-updates/main passwd amd64 1:4.1.4.2+svn3283-3ubuntu5.1 [959 kB]
   Get:62 http://llvm.org/apt/precise/ llvm-toolchain-precise-3.6/main libclang-common-3.6-dev amd64 1:3.6.2~svn240577-1~exp1 [1,756 kB]
   Get:63 http://ppa.launchpad.net//ubuntu-toolchain-r/test/ubuntu/ precise/main libobjc4 amd64 6.2.0-3ubuntu11~12.04 [162 kB]
   Get:64 http://archive.ubuntu.com/ubuntu/ precise-updates/main libuuid1 amd64 2.20.1-1ubuntu3.1 [12.8 kB]
   Get:65 http://archive.ubuntu.com/ubuntu/ precise-security/main libsqlite3-0 amd64 3.7.9-2ubuntu1.2 [349 kB]
   Get:66 http://ppa.launchpad.net//ubuntu-toolchain-r/test/ubuntu/ precise/main libobjc-4.9-dev amd64 4.9.4-2ubuntu1~12.04 [799 kB]
   Get:67 http://archive.ubuntu.com/ubuntu/ precise-updates/main libcomerr2 amd64 1.42-1ubuntu2.3 [57.2 kB]
   Get:68 http://archive.ubuntu.com/ubuntu/ precise-security/main libssl1.0.0 amd64 1.0.1-4ubuntu5.38 [1,055 kB]
   Get:69 http://archive.ubuntu.com/ubuntu/ precise-updates/main libroken18-heimdal amd64 1.6~git20120311.dfsg.1-2ubuntu0.1 [46.0 kB]
   Get:70 http://archive.ubuntu.com/ubuntu/ precise-updates/main libasn1-8-heimdal amd64 1.6~git20120311.dfsg.1-2ubuntu0.1 [220 kB]
   Get:71 http://archive.ubuntu.com/ubuntu/ precise/main libbsd0 amd64 0.3.0-2 [31.6 kB]
   Get:72 http://archive.ubuntu.com/ubuntu/ precise/main libgpg-error0 amd64 1.10-2ubuntu1 [14.5 kB]
   Get:73 http://archive.ubuntu.com/ubuntu/ precise-security/main libgcrypt11 amd64 1.5.0-3ubuntu0.6 [282 kB]
   Get:74 http://archive.ubuntu.com/ubuntu/ precise/main libp11-kit0 amd64 0.12-2ubuntu1 [34.3 kB]
   Get:75 http://archive.ubuntu.com/ubuntu/ precise-security/main libtasn1-3 amd64 2.10-1ubuntu1.5 [43.6 kB]
   Get:76 http://archive.ubuntu.com/ubuntu/ precise-security/main libgnutls26 amd64 2.12.14-5ubuntu3.12 [460 kB]
   Get:77 http://archive.ubuntu.com/ubuntu/ precise-security/main libkrb5support0 amd64 1.10+dfsg~beta1-2ubuntu0.7 [24.9 kB]
   Get:78 http://archive.ubuntu.com/ubuntu/ precise-security/main libk5crypto3 amd64 1.10+dfsg~beta1-2ubuntu0.7 [80.1 kB]
   Get:79 http://archive.ubuntu.com/ubuntu/ precise/main libkeyutils1 amd64 1.5.2-2 [7,862 B]
   Get:80 http://archive.ubuntu.com/ubuntu/ precise-security/main libkrb5-3 amd64 1.10+dfsg~beta1-2ubuntu0.7 [355 kB]
   Get:81 http://archive.ubuntu.com/ubuntu/ precise-security/main libgssapi-krb5-2 amd64 1.10+dfsg~beta1-2ubuntu0.7 [119 kB]
   Get:82 http://archive.ubuntu.com/ubuntu/ precise-security/main libidn11 amd64 1.23-2ubuntu0.1 [112 kB]
   Get:83 http://archive.ubuntu.com/ubuntu/ precise-updates/main libhcrypto4-heimdal amd64 1.6~git20120311.dfsg.1-2ubuntu0.1 [103 kB]
   Get:84 http://llvm.org/apt/precise/ llvm-toolchain-precise-3.6/main clang-3.6 amd64 1:3.6.2~svn240577-1~exp1 [37.1 MB]
   Get:85 http://archive.ubuntu.com/ubuntu/ precise-updates/main libheimbase1-heimdal amd64 1.6~git20120311.dfsg.1-2ubuntu0.1 [33.1 kB]
   Get:86 http://archive.ubuntu.com/ubuntu/ precise-updates/main libwind0-heimdal amd64 1.6~git20120311.dfsg.1-2ubuntu0.1 [77.8 kB]
   Get:87 http://archive.ubuntu.com/ubuntu/ precise-updates/main libhx509-5-heimdal amd64 1.6~git20120311.dfsg.1-2ubuntu0.1 [125 kB]
   Get:88 http://archive.ubuntu.com/ubuntu/ precise-updates/main libkrb5-26-heimdal amd64 1.6~git20120311.dfsg.1-2ubuntu0.1 [234 kB]
   Get:89 http://archive.ubuntu.com/ubuntu/ precise-updates/main libheimntlm0-heimdal amd64 1.6~git20120311.dfsg.1-2ubuntu0.1 [16.0 kB]
   Get:90 http://archive.ubuntu.com/ubuntu/ precise-updates/main libgssapi3-heimdal amd64 1.6~git20120311.dfsg.1-2ubuntu0.1 [108 kB]
   Get:91 http://archive.ubuntu.com/ubuntu/ precise-updates/main libsasl2-2 amd64 2.1.25.dfsg1-3ubuntu0.1 [69.1 kB]
   Get:92 http://archive.ubuntu.com/ubuntu/ precise-security/main libldap-2.4-2 amd64 2.4.28-1.1ubuntu4.6 [185 kB]
   Get:93 http://archive.ubuntu.com/ubuntu/ precise/main librtmp0 amd64 2.4~20110711.gitc28f1bab-1 [57.1 kB]
   Get:94 http://archive.ubuntu.com/ubuntu/ precise-security/main openssl amd64 1.0.1-4ubuntu5.38 [524 kB]
   Get:95 http://archive.ubuntu.com/ubuntu/ precise-security/main ca-certificates all 20160104ubuntu0.12.04.1 [208 kB]
   Get:96 http://archive.ubuntu.com/ubuntu/ precise-security/main libcurl3-gnutls amd64 7.22.0-3ubuntu4.17 [228 kB]
   Get:97 http://archive.ubuntu.com/ubuntu/ precise/main libedit2 amd64 2.11-20080614-3ubuntu2 [70.3 kB]
   Get:98 http://archive.ubuntu.com/ubuntu/ precise-security/main libxml2 amd64 2.7.8.dfsg-5.1ubuntu4.15 [677 kB]
   Get:99 http://archive.ubuntu.com/ubuntu/ precise/main libnettle4 amd64 2.4-1 [95.1 kB]
   Get:100 http://archive.ubuntu.com/ubuntu/ precise-security/main libarchive12 amd64 3.0.3-6ubuntu1.3 [274 kB]
   Get:101 http://archive.ubuntu.com/ubuntu/ precise-security/main libc-dev-bin amd64 2.15-0ubuntu10.15 [84.7 kB]
   Get:102 http://archive.ubuntu.com/ubuntu/ precise-security/main linux-libc-dev amd64 3.2.0-119.162 [850 kB]
   Get:103 http://archive.ubuntu.com/ubuntu/ precise-security/main libc6-dev amd64 2.15-0ubuntu10.15 [2,943 kB]
   Get:104 http://archive.ubuntu.com/ubuntu/ precise-security/main libcurl3 amd64 7.22.0-3ubuntu4.17 [237 kB]
   Get:105 http://archive.ubuntu.com/ubuntu/ precise-security/main libexpat1 amd64 2.0.1-7.2ubuntu1.4 [131 kB]
   Get:106 http://archive.ubuntu.com/ubuntu/ precise/main bsdmainutils amd64 8.2.3ubuntu1 [200 kB]
   Get:107 http://archive.ubuntu.com/ubuntu/ precise/main nano amd64 2.2.6-1 [194 kB]
   Get:108 http://archive.ubuntu.com/ubuntu/ precise-security/main binutils amd64 2.22-6ubuntu1.4 [2,653 kB]
   Get:109 http://archive.ubuntu.com/ubuntu/ precise-security/main libxmlrpc-core-c3 amd64 1.16.33-3.1ubuntu5.2 [180 kB]
   Get:110 http://archive.ubuntu.com/ubuntu/ precise/main emacsen-common all 1.4.22ubuntu1 [16.9 kB]
   Get:111 http://archive.ubuntu.com/ubuntu/ precise-updates/main cmake-data all 2.8.7-0ubuntu5 [754 kB]
   Get:112 http://archive.ubuntu.com/ubuntu/ precise-updates/main cmake amd64 2.8.7-0ubuntu5 [4,353 kB]
   Fetched 96.1 MB in 1min 36s (995 kB/s)
   Download complete and in download only mode
-> Unpacking  libc-dev-bin_2.15-0ubuntu10.15_amd64
-> Unpacking  libcilkrts5_6.2.0-3ubuntu11~12.04_amd64
-> Unpacking  libcomerr2_1.42-1ubuntu2.3_amd64
-> Unpacking  libasan1_4.9.4-2ubuntu1~12.04_amd64
-> Unpacking  libc-bin_2.15-0ubuntu10.15_amd64
-> Unpacking  util-linux_2.20.1-1ubuntu3.1_amd64
-> Unpacking  libpng12-0_1.2.46-3ubuntu4.2_amd64
-> Unpacking  libkrb5-3_1.10+dfsg~beta1-2ubuntu0.7_amd64
-> Unpacking  libasn1-8-heimdal_1.6~git20120311.dfsg.1-2ubuntu0.1_amd64
-> Unpacking  insserv_1.14.0-2.1ubuntu2_amd64
-> Unpacking  libgssapi3-heimdal_1.6~git20120311.dfsg.1-2ubuntu0.1_amd64
-> Unpacking  libdrm-intel1_2.4.52-1~precise2_amd64
-> Unpacking  libuuid1_2.20.1-1ubuntu3.1_amd64
-> Unpacking  ca-certificates_20160104ubuntu0.12.04.1_all
-> Unpacking  libitm1_6.2.0-3ubuntu11~12.04_amd64
-> Unpacking  liblsan0_6.2.0-3ubuntu11~12.04_amd64
-> Unpacking  libllvm3.6_1%3a3.6.2~svn240577-1~exp1_amd64
-> Unpacking  libexpat1_2.0.1-7.2ubuntu1.4_amd64
-> Unpacking  upstart_1.5-0ubuntu7.3_amd64
-> Unpacking  libstdc++6_6.2.0-3ubuntu11~12.04_amd64
-> Unpacking  libkrb5support0_1.10+dfsg~beta1-2ubuntu0.7_amd64
-> Unpacking  emacsen-common_1.4.22ubuntu1_all
-> Unpacking  libgpg-error0_1.10-2ubuntu1_amd64
-> Unpacking  libp11-kit0_0.12-2ubuntu1_amd64
-> Unpacking  libubsan0_6.2.0-3ubuntu11~12.04_amd64
-> Unpacking  libobjc4_6.2.0-3ubuntu11~12.04_amd64
-> Unpacking  libkeyutils1_1.5.2-2_amd64
-> Unpacking  libbsd0_0.3.0-2_amd64
-> Unpacking  libheimntlm0-heimdal_1.6~git20120311.dfsg.1-2ubuntu0.1_amd64
-> Unpacking  gcc-4.9-base_4.9.4-2ubuntu1~12.04_amd64
-> Unpacking  libheimbase1-heimdal_1.6~git20120311.dfsg.1-2ubuntu0.1_amd64
-> Unpacking  passwd_1%3a4.1.4.2+svn3283-3ubuntu5.1_amd64
-> Unpacking  ifupdown_0.7~beta2ubuntu11.1_amd64
-> Unpacking  libk5crypto3_1.10+dfsg~beta1-2ubuntu0.7_amd64
-> Unpacking  libsasl2-2_2.1.25.dfsg1-3ubuntu0.1_amd64
-> Unpacking  libnih1_1.0.3-4ubuntu9.1_amd64
-> Unpacking  binutils_2.22-6ubuntu1.4_amd64
-> Unpacking  libxml2_2.7.8.dfsg-5.1ubuntu4.15_amd64
-> Unpacking  libdrm-nouveau1a_2.4.52-1~precise2_amd64
-> Unpacking  libblkid1_2.20.1-1ubuntu3.1_amd64
-> Unpacking  libmount1_2.20.1-1ubuntu3.1_amd64
-> Unpacking  libldap-2.4-2_2.4.28-1.1ubuntu4.6_amd64
-> Unpacking  libdrm2_2.4.52-1~precise2_amd64
-> Unpacking  initramfs-tools-bin_0.99ubuntu13.5_amd64
-> Unpacking  libquadmath0_6.2.0-3ubuntu11~12.04_amd64
-> Unpacking  libclang-common-3.6-dev_1%3a3.6.2~svn240577-1~exp1_amd64
-> Unpacking  libpcre3_8.12-4ubuntu0.2_amd64
-> Unpacking  libnih-dbus1_1.0.3-4ubuntu9.1_amd64
-> Unpacking  linux-libc-dev_3.2.0-119.162_amd64
-> Unpacking  libudev0_175-0ubuntu9.10_amd64
-> Unpacking  libatomic1_6.2.0-3ubuntu11~12.04_amd64
-> Unpacking  libgcc1_1%3a6.2.0-3ubuntu11~12.04_amd64
-> Unpacking  libgcc-4.9-dev_4.9.4-2ubuntu1~12.04_amd64
-> Unpacking  libgomp1_6.2.0-3ubuntu11~12.04_amd64
-> Unpacking  libwind0-heimdal_1.6~git20120311.dfsg.1-2ubuntu0.1_amd64
-> Unpacking  ncurses-bin_5.9-4_amd64
-> Unpacking  libarchive12_3.0.3-6ubuntu1.3_amd64
-> Unpacking  libncurses5_5.9-4_amd64
-> Unpacking  libtasn1-3_2.10-1ubuntu1.5_amd64
-> Unpacking  cpio_2.11-7ubuntu3.2_amd64
-> Unpacking  gcc-6-base_6.2.0-3ubuntu11~12.04_amd64
-> Unpacking  libslang2_2.2.4-3ubuntu1_amd64
-> Unpacking  libdbus-1-3_1.4.18-1ubuntu1.8_amd64
-> Unpacking  libnettle4_2.4-1_amd64
-> Unpacking  libcurl3_7.22.0-3ubuntu4.17_amd64
-> Unpacking  libc6_2.15-0ubuntu10.15_amd64
-> Unpacking  libgcrypt11_1.5.0-3ubuntu0.6_amd64
-> Unpacking  libdrm-radeon1_2.4.52-1~precise2_amd64
-> Unpacking  plymouth_0.8.2-2ubuntu31.1_amd64
-> Unpacking  sysv-rc_2.88dsf-13.10ubuntu11.1_all
-> Unpacking  librtmp0_2.4~20110711.gitc28f1bab-1_amd64
-> Unpacking  iproute_20111117-1ubuntu2.3_amd64
-> Unpacking  module-init-tools_3.16-1ubuntu2_amd64
-> Unpacking  libpciaccess0_0.12.902-1ubuntu0.2_amd64
-> Unpacking  libxmlrpc-core-c3_1.16.33-3.1ubuntu5.2_amd64
-> Unpacking  nano_2.2.6-1_amd64
-> Unpacking  sysvinit-utils_2.88dsf-13.10ubuntu11.1_amd64
-> Unpacking  initscripts_2.88dsf-13.10ubuntu11.1_amd64
-> Unpacking  libsqlite3-0_3.7.9-2ubuntu1.2_amd64
-> Unpacking  libc6-dev_2.15-0ubuntu10.15_amd64
-> Unpacking  libhx509-5-heimdal_1.6~git20120311.dfsg.1-2ubuntu0.1_amd64
-> Unpacking  libedit2_2.11-20080614-3ubuntu2_amd64
-> Unpacking  libclang1-3.6_1%3a3.6.2~svn240577-1~exp1_amd64
-> Unpacking  libglib2.0-0_2.32.4-0ubuntu1_amd64
-> Unpacking  cmake-data_2.8.7-0ubuntu5_all
-> Unpacking  libklibc_1.5.25-1ubuntu2_amd64
-> Unpacking  initramfs-tools_0.99ubuntu13.5_all
-> Unpacking  lsb-base_4.0-0ubuntu20.3_all
-> Unpacking  libgnutls26_2.12.14-5ubuntu3.12_amd64
-> Unpacking  libroken18-heimdal_1.6~git20120311.dfsg.1-2ubuntu0.1_amd64
-> Unpacking  libidn11_1.23-2ubuntu0.1_amd64
-> Unpacking  openssl_1.0.1-4ubuntu5.38_amd64
-> Unpacking  libcurl3-gnutls_7.22.0-3ubuntu4.17_amd64
-> Unpacking  libobjc-4.9-dev_4.9.4-2ubuntu1~12.04_amd64
-> Unpacking  mount_2.20.1-1ubuntu3.1_amd64
-> Unpacking  cmake_2.8.7-0ubuntu5_amd64
-> Unpacking  clang-3.6_1%3a3.6.2~svn240577-1~exp1_amd64
-> Unpacking  procps_1%3a3.2.8-11ubuntu6.4_amd64
-> Unpacking  libhcrypto4-heimdal_1.6~git20120311.dfsg.1-2ubuntu0.1_amd64
-> Unpacking  libtsan0_6.2.0-3ubuntu11~12.04_amd64
-> Unpacking  adduser_3.113ubuntu2_all
-> Unpacking  busybox-initramfs_1%3a1.18.5-1ubuntu4.1_amd64
-> Unpacking  libkrb5-26-heimdal_1.6~git20120311.dfsg.1-2ubuntu0.1_amd64
-> Unpacking  libssl1.0.0_1.0.1-4ubuntu5.38_amd64
-> Unpacking  udev_175-0ubuntu9.10_amd64
-> Unpacking  libncursesw5_5.9-4_amd64
-> Unpacking  libgssapi-krb5-2_1.10+dfsg~beta1-2ubuntu0.7_amd64
-> Unpacking  libplymouth2_0.8.2-2ubuntu31.1_amd64
-> Unpacking  mountall_2.36.4ubuntu0.1_amd64
-> Unpacking  klibc-utils_1.5.25-1ubuntu2_amd64
-> Unpacking  bsdmainutils_8.2.3ubuntu1_amd64
-> Unpacking  libstdc++-4.9-dev_4.9.4-2ubuntu1~12.04_amd64
Container has been set up in container

 

 

A conversation

Over the next few days, weeks, months and years there will be a conversation covering one very important question. Depending on the answer that people come to accept, the belief formed will have far reaching implications.

That question is: “is liberalism no longer a viable political ideology in a democracy?”

Many people will tell you that Brexit and Trump became a reality because liberals ignored the interests and views of the “white working class”. This explanation is attractive because it is simple. But simple explanations are dangerous, especially when we consider what some of those interests and views are.

The reality is that in some sense, the “silent majority” who voted to put Trump in the White House and the Republicans in the Congress and Senate must have, at some level, believed that you could do anything you wanted to a woman if you had power and money. They must have at some level believed that the country you were born in determined whether you were a rapist, a criminal or ‘perhaps a good person’. They must have at some level believed that your faith and the community that you grew up in is a reason to prevent you from interacting with some subset of 300 million people in the world, despite the fact that such a subset might want to include you.

We can say that Trump voters had these beliefs because it would take an astounding level of self deception to not hold those beliefs and still vote the way they did. Perhaps how they reached those beliefs involves a degree of moral blameworthiness. Perhaps it involves a degree of socialisation.

Regardless of the source, the question for liberals now is whether those beliefs, views and interests should be catered to for the sake of electability. Be very weary of this.

In a representative democracy elected representatives are not just a mouthpiece for the majority states, or the majority of electorates or the majority of people. They are also there to act on moral principle. We elect representatives because we trust them to make the hard decisions for all of us on our behalf.

One only needs to look to Australia to find out what happens when this principled outlook is lost. Our own politicians figured out that they could make themselves more electable by accommodating racist and xenophobic beliefs. But they quickly found out that racism and xenophobia begets more racism and xenophobia. Australia now runs one the most, if not the most, cruel and inhumane offshore detention schemes with a large degree of bipartisan and public support. And it was only last week that we heard it was to become more cruel. Real people are suffering and even killing themselves as a result.

If you find that liberals are unelectable because because of a racist, xenophobic and sexist silent majority, change the silent majority. We could all (myself included) do a much better job to call out intolerance when we see it.

Silently accepting it is so much worse.

#ITSOKAYTOTALK

There’s a meme going around social media at the moment which I think is particularly worthwhile, called #itsokaytotalk. If you haven’t seen it yet, its primary purpose is to send a message that as individuals, we can and should, seek support from our friends, colleagues and family when we are experiencing problems, feelings or negative thought patterns (often collectively referred to as “mental illness”). The message also points out that, statistically speaking, the leading cause of death for people aged 15-44 is suicide. It has gained a substantial amount of traction recently, so you’re likely to see it if you haven’t already.

I’m usually hesitant about adding my voice to the chorus with these things, primarily because I don’t think I have anything important to add and I don’t want to minimise the stories of others whom I think should be listened to. I had some debate with myself about whether to add my thoughts to the mix, but I think I can contribute to the common good by adding those thoughts.

I should add a disclaimer here that the stories I will tell here are in no way intended to minimise the experiences and stories shared by other people. We should all take the time to listen to their stories and, if appropriate and called for, provide our words of support. I’m also not purporting to act in any capacity as a mental health professional. Most of the stories I will share with you in this post are either anecdotal or personal experience. As we all know, the plural of anecdote is not evidence.

Now some unpacking of this message and the answering of questions which may actually provide some support and insight to people.

Why should I burden others with my feelings, thoughts, fears or problems which just seem so irrational, uninteresting or commonplace?

The analogy I like to draw here is that depression and anxiety are not like a car crash. At least in my own experience and in the experience of others I know, they are experienced not as a reaction to a highly traumatic adverse event but usually start as a small reaction to something that might otherwise seem commonplace(1). Depression and anxiety are more like cancer. For some reason that we don’t understand yet, sometimes people get unlucky and that otherwise commonplace disappointment starts to infect their identity and sense of self-worth. The person starts to believe that they are defective, unworthy, useless or unloveable because of what may have happened to them.

Unfortunately, the human brain is wired to be very good at confirmation bias. The small “network” about our identity grows and as it grows, more thoughts become linked to it. The person afflicted starts to explain everything that happened around them by reference to the  belief about themselves. Then they begin to act out the belief, which reinforces it even more. Soon almost everything begins to remind the afflicted person of how terrible they are and it becomes overwhelming.

Common examples of these small seeds that grow are relationship breakdowns, fights with friends, lack of employment or lack of success at work or education.

As an example to dissect, I’m going to pick unamicable breakdowns of a romantic relationship, because they’re common, they hurt like hell and we’re very embarrassed to talk about them. Usually when a romantic relationship breaks down, the person who wanted to stay in the relationship will search for answers and explanations as part of the grief cycle. It is quite likely that one of those explanations will start with “If only I …” followed by some sort of statement about a person’s internal characteristics, appearance, behaviour or the like. That explanation feeds back into the person’s sense of self-worth, perhaps because they feel the particular characteristic which they thought was the cause of the relationship breakdown makes them inherently defective or unloveable. And so the cycle begins and continues. This isn’t the fault of anyone in particular – it is just a thing that happens.

Because such unamicable breakdowns are a fact of life and happen frequently, we’re generally hesitant to talk about them, viewing them as “drama”. There’s a rather unhelpful adage of “block, gym, lawyer, move on” which reinforces this notion. The thing is that while the breakdown of the relationship might, objectively, seem insignificant to talk about, the consequential impact on identity and self-worth is very significant and could well be a life-or-death conversation. A good way to combat confirmation bias is to have an external source provide undeniable evidence that the internal belief is false. And it is far easier to combat a dogma in its early stages of formation than the point where it becomes well-formed and attached to lots of other things. If you had cancer, would you prefer to nip it in the bud or go through years of excruciating treatment later on down the track? I’m sure you’d probably pick the former.

(1) This is not to say that depression and anxiety can be experienced as a response to a highly traumatic or extreme events nor are the needs of a person in that situation any less important than they would be at present.

If I just exercise and take some antidepressants, the problem will just go away and I won’t have to talk to anyone right?

Actually, no, because depression and anxiety which are sourced from beliefs about yourself don’t work like that.

Both natural (in the sense of endorphins released during exercise and dopamine released during pleasurable activities) and artificial antidepressants have the function of lifting your mood and your energy levels. They reduce the effect of the the negative-self-worth network so that you can function without everything slamming that network and making you feel miserable all the time. But they don’t fix the problem, which is that the network exists in the first place.

Medication and exercise absolutely have an important place and if you are struggling with recurring negative thought patterns or generalised malaise, it never hurts to ask a doctor who can refer you on to the right person and to determine if they’d be right for you.

However, you still need to talk to your friends, family and colleagues if something is troubling you and you’re stuck in a thought loop. See above for why.

What if my friends, family and colleagues don’t care?

Put yourself in their shoes – if someone came to you and wanted to talk about something that was making them feel miserable, would you tell them to “get over it” because you didn’t care?

Of course not.

In fact, they’re more than likely to be humbled by the that you opened up to them and will likely encourage you to talk to them more.

I don’t discount the possibility that someone may well tell you to go away or minimise your feelings. To that extent, I would say that if you are the person being asked to give support, try to look past the particular problem to the internalised belief that the person has. To the person seeking the support, it may be easier for people to give you that support if you talk about the belief as opposed to the event. We can all relate to beliefs about ourselves, whereas it might be harder to relate to particular events.

What if I’m so far down the track that it feels impossible to talk about this stuff?

The more depression and anxiety grow, the more complex the emotions and circumstances behind them become. Often times things become cyclic and entangled and you just don’t even know where to start.

Start somewhere. Even if its just the bad day you happened to have at work. Your discussions with others may prove to be enlightening and help you to find the core belief that’s causing the entangled feeling of misery.

Of course, you can also ask  doctor to refer you to a qualified counsellor who specialises in things like clinical psychology and Cognitive Behavioural Therapy. Those professionals are there to help you unpack what’s going on and change the thoughts, beliefs and behaviours you have might may be reinforcing the negative self-worth. In Australia, you can get ten free private consultations and ten free group sessions per year with a doctor’s referral. That almost gives you one private and one group consultation per month.

I’m not a man and I feel like I can’t talk to anyone either

It would be folly for me to end this post without some discussion of how depression and anxiety affect people who aren’t men. I’ve deliberately tried to adopt a non-gendered approach in this post for reasons I’ll discuss below but I should probably note that it seems to be in vogue right now to talk about mental health as it relates to men and we seem to be forgetting women and other gender minorities in the mix too.

First of all, there’s a gendered assumption that women are better at leveraging their networks for emotional support and further that women are better at talking about emotions generally. Anecdotally at least, I’ve found that to be totally untrue. Now, I have no real way of verifying this, because I have no experience of being female. But out of the people who have come to me for support, men and women included, everyone told me that they felt that there was nobody else they could talk to about the challenges they were facing. Didn’t matter how large or small their networks were. So in that sense, one should not abstain from talking because a gendered assumption tells them that they should have networks which will automatically provide support and advice.

Second of all, there’s another (perhaps implicit) assumption that women face less stigma than men for being afflicted by problems in their emotional wellbeing. This is also totally false. The stigma just has different labels and consequences. Many women are just as concerned as men that talking about their own mental wellbeing and related problems for fear that it could have adverse affects on their career, social standing or future relationship prospects. All I can do in this post is acknowledge and validate the fact that, for both genders, the fear is there and it feels very real, and state further that talking to people you trust about what’s bothering you will still be beneficial.

Closing Thoughts

The #itsokaytotalk meme is important and we should all take heed of its message. I only wrote this post because I think my own experiences with having been there and in supporting others who have been there could contribute some insight that others might find useful.

I don’t want this to become a post about me, so I’d challenge you to do this. Don’t click the “like” button on this post. Don’t reshare it verbatim either. If you think I said something that’s worthwhile to spread, then write about it in your own words and add your own thoughts. If I missed something, talk about that. Criticise me if you think I’m wrong. You don’t even have to tell me that you saw this post as inspiration to do so.

Mental wellbeing is a really complex and hard topic. It is complicated by all sorts of personal, cultural and social factors.  The more people understand about it, the more they’ll able to help themselves, so lets make this an exercise in not just awareness, but understanding.

Improvements to polysquare-travis-container

After some time out, I sat down to make some changes to polysquare-travis-container based on some things I had recently learned about package managers.

My job at Endless requires that I occasionally work with Debian packages. One tool I have been working with quite a lot lately has been chdist from the devscripts
package.

Whilst also being a wonderful tool to compare Debian-based distributions, the fact that a tool like chdist exists quite handily proved that it should be possible to use apt and dpkg in a separate subdirectory, without needing root access. This was the initial raison d’être for polysquare-travis-container.

One of my thoughts recently has been that while proot was a great tool to start out with, it is probably too heavyweight for what I am really trying to do with this project. What I really want is a way to quickly install things that only exist in Debian packages, install them without root access and ensure that the installation, compile and test process for a project requiring those packages is reproducible at least on the same platform. The overall goal is to enable users to run their CI setup locally, as painlessly as possible.

With this new knowledge, I took some time this weekend to look into how chdist works.

In essence, chdist is a perl wrapper around apt which specifies configuration options such that apt is able to run locally. It does that by specifying the Dir, Dir::State and Dir::Cache options. That mirrors closely what you might find on a Debian installation in /var/cache and /var/lib/dpkg.

Assuming that the directory structure both dpkg and apt expect exist at those paths, you can run apt-get update without root on a custom lists file and even start downloading packages along with their dependencies.

My goal was to get polysquare-travis-container to a point where it wasn’t necessary to download an Ubuntu filesystem image or use proot. After hacking around for a little while, I’m more or less settled on the idea that doing so isn’t particularly trivial unless you have access to something like debootstrap because of all the initial circular dependencies. Ideally you want to be running apt from within the nested filesystem as well, since the “jailing” of apt by using the aforementioned variables isn’t perfect.

Most software from the new filesystem can be run without root access somewhat comfortably. Hardcoded paths, particular in shebangs, may cause trouble however. You will need to set the following environment variables:

  • PATH: Where the system should look for executables. Obviously you want this so that typing an executable name into a shell will find the executable in your filesystem tree.
  • LD_LIBRARY_PATH: Where ld.so should look for dynamic libraries. It is effectively mandatory to set this to /usr/lib and /usr/lib/${arch-triple}, otherwise running binaries from the filesystem root will attempt to bring in libraries from your system filesystem (which likely won’t work).
  • PKG_CONFIG_PATH: Needed for building software.
  • LIBRARY_PATH: Needed for some static linking cases.
  • INCLUDE_PATH: Needed if you installed a compiler in your filesystem root, since this is the first place gcc and clang will look for headers.
  • CPATH: See above.
  • CPPPATH: See above.

I have now released polysquare-travis-container 0.0.38 which takes into account these considerations. It now ships with a –local option for Ubuntu, which downloads an Ubuntu core image, minifies it to about 20MB using dpkg and proot and from that point onward uses only environment variables and a specially-configured apt to install packages and run software. It is about as lightweight as you can get for this kind of thing. Please feel free to try it and give feedback.

Safety is not our first priority

I was on a flight back to Australia when something in the routine safety
demonstration video struck me with more significance than it had in the past.

“Safety is our first priority”

I began to realise just how often I had seen this value being espoused and how
often I’ve come to take it for granted. The infrastructure and services I rely
on every day put my safety as their top priority. Above making money. Above
being innovative. Above customer service. If that organisation did not keep me
safe whilst my livelihood was in their hands, the organisation would consider
itself to have failed at its primary objective.

If you look around, you’ll find that this is an incredibly common organisational
value. You will find it in any organisation that operates heavy machinery, any
organisation that is responsible for taking care of a dangerous place and any
organisation that is responsible for building infrastructure upon which the
public relies. It is therefore no surprise that this value has been reflected in
the customary law of negligence for centuries. When someone is vulnerable to
your actions making them unsafe, you are under a duty of care to ensure that
above all else, they are safe.

It is customary for all engineers in the United States and Canada to receive an
iron ring upon their qualification and admission into the profession. It is
folklore that the iron is sourced
from the remains of the Quebec Bridge, which collapsed during construction in
1907, killing seventy five construction workers. The reason the bridge
collapsed was that the engineers responsible for its design and construction cut
corners to get the project to completion more quickly. Those workers were
relying on the engineers to keep them safe and the engineers put profit ahead
of the safety of those people. The ring is intended to be a permanent reminder
to all of those in the profession that such errors should never be repeated.

The priority of safety in software

You will very rarely find this value in organisations that develop software.
Most organisations are even allergic to it, including verbose disclaimers in
software licences purporting to waive any implied warranty that the code you run
is fit for purpose and won’t cause you trouble. Most licenses will require users
to agree to numerous waivers and consent to all sorts of things which may be
harmful to them. There is no way other professions would get away with that
without raising an eyebrow. But for software, we don’t seem to care.

THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM “AS IS” WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.

This is puzzling, because at the same time, professional bodies representing the
computer software industry purport to put the public interest as their number
one priority. For instance, the Australian Computer Society’s (Australian Computer Society code of ethics) says that "You
will place the interests of the public above those of personal, business or
sectional interests." This disconnect is disturbing when we think about the
consequences that flow from such ignorance. A failure to have a safety oriented
culture means that as so-called "engineers" our profession becomes complacent
about the impact of our strategic and engineering choices upon users. It results
in culture that puts the need to stay ahead and continue making a profit above
the need to protect the legitimate safety interests of users. It leads to a
discourse which implicitly accepts that failure is inevitable and can only be
mitigated as opposed to prevented by making trade-offs which may reduce
competitiveness but protect the public.

A resort to realism

Many computer software professionals would retort at this point delivering
perfect quality software is just not realistic. The project may be on a tight
budget or operating under a tight timeframe. It would not be possible to remain
competitive and maintain such a high standard. And nevertheless, software
professionals say, users have come to understand this reality as well. The
software industry has remained unregulated since its inception and the benefits
for society have been explosive growth of technology and innovation at a pace
never before seen.

Such explosive growth and innovation has had real winners and losers. Over the
last forty years technology has moved beyond the realm of academia and military
research and become democratised. Computing is readily accessible in the age of
the smartphone. It is also no surprise then that the wealth generated by
computer software has become concentrated in the hands of the few who were best
able to execute and facilitate this explosive growth.

But there are losers. It is becoming increasingly difficult, and now almost
impossible for people to opt out of relying on software in their daily lives,
just like it is now almost impossible to opt out of roads, electricity and
plumbing. Software controls an increasingly higher degree of the infrastructure
that we rely on every day and increasingly controls the way that we interact
with each other, do business and access essential services. In an age where
software grows explosively and operates the world, people have become
increasingly vulnerable to the choices software development organisations make.
Every day we hear of a new data leak. Confidential information is stolen and
sold to the highest bidder. Lives are meddled with and lives are ruined. User
facing software upon which people rely to make a livelihood breaks and those
users shoulder the burden of the lost time or inconvenience. In extreme cases
people die. Virtual bridges are collapsing around us and the best we can say as
a profession is that such collapses are a cost society is willing to accept in
the name of exponential progress.

Safety, is not our first priority.

What if safety were our first priority?

Engineering failures threatening the safety of others can be avoided so long as
we are willing, as a profession, to make certain trade-offs. If you ever wonder
why the pace of change for aviation, rail transportation and civil architecture
seems to be glacial in comparison, it is because safety is the first priority.

If we want to make safety the first priority, there are some uncomfortable
counter-realities that we may need to accept. It is plausible that in such an
alternate timeline, the smartphone may have never been brought to the mass
market, It is possible that social networking and the writeable web would have
been considered and then quietly abandoned, due to the inherent risk associated
with storing confidential information about millions of people in a central
repository. Your operating system and applications may be considerably more
stable, but the redundancy required to avoid takings risks would mean that its
performance and scope of functionality would be heavily reduced.

These costs, are, I suspect, too much for our cutting-edge society and cut-
throat business culture to bear. One thing I do believe though is that we can
still put the safety of users as our number one priority by focusing on the 20%
which will avoid 80% of the risk. We can, as a profession, still ask the hard
questions which might affect the bottom line. We can still ask whether a
particular feature, though innovative, might bring with it risks to the people
who depend on us that those people would find unacceptable, even if it meant
sacrificing the feature. We can still look at the systems we are designing and
ask "do I understand the impact upon a person if this fails?" We can still sit
down and make sure the math checks out before proceeding on a course of conduct
which may have irreversible consequences.

If we just sat down and accepted that certain trade-offs will have to be made,
we can be a profession that can be proud to say that safety is our first
priority.

Thanks to Wayne Spilsbury for providing feedback on and editing this essay

Naming things

Naming things is apparently one of the hardest problems in software engineering.

I have a bunch of side projects with rather generic and uninspiring names. I’d like to see them gain some wider usage and I think the names are putting people off. Unfortunately, I can’t think of any decent names. So I’d like to throw it out to the community to see if we can find a better name for them! Here’s a list of projects and what they do. If you can think of a better name, just post it in the comments.

polysquare-ci-scripts

Elevator Pitch: Getting software to run on CI environments like Travis-CI requires installing a bunch of dependencies, activating environments and doing other setup. This creates a lot of duplicate code in configuration files. These extensible scripts, written in Python, can be directly fetched with CURL and executed. They set up any required language environments, install dependencies and do deployment-specific steps.

polysquare-travis-container

Elevator Pitch: System package managers are great, but they make life painful when trying to reproduce builds between systems. They often require system level access – something you don’t always have or want. Docker and Vagrant partially solve this problem, but one only works for linux guests and the other is quite heavy-weight. This project creates a local version of your operating system’s package manager so you can install just what you need and nothing else. You run binaries through it and it will automatically set up any required PATHs or LD_LIBRARY_PATHS to make it work.

cmake-ast

Elevator Pitch: Parse CMake files and create an abstract syntax tree, usable from Python.

polysquare-cmake-linter

Elevator Pitch: Catches bad practice in CMake files. Like cmake-lint, but it checks for other things, especially variable quoting.

polysquare-generic-file-linter

I can’t think of a worse name!

Elevator Pitch: Ensures that each source code file’s header is consistently styled and checks for spelling mistakes in comments and user facing strings. For instance, it checks to make sure that every file contains a copyright notice, or that if the name of the file appears at the top of its copyright notice, that the name is actually correct. It also makes sure that anything referred to in a code comment can actually be found in the code if it is not an english word.

polysquare-setuptools-lint

Elevator Pitch: Integrates every decent python linting tool into a setuptools command. Collects all the output into a single format and de-duplicates any warnings. Runs prospector, flake8, pyroma and polysquare-generic-file-linter. Caches results and parallelises the linter processes where possible to speed up builds.

travis-bump-version

Elevator Pitch: Bumps your project’s version number, tags a new release and pushes tags to git on request. Uses bumpversion under the good. Designed to be used in conjunction with Travis-CI.

tooling-cmake-util

Elevator Pitch: A library for CMake that makes it easy to integrate new static analysis tools into your build. Just run psq_run_tool_for_each_source on a target with your tool’s binary and arguments and that tool will run every time that target is updated during your build.

common-universal-cmake

Elevator Pitch: Add it to your project, add executables and libraries through it, and you get amazing tooling like CPPCheck, clang-tidy, include-what-you-use, vera++ and others for free. Adds an option to build code with AddressSanitizer, UndefinedBehaviourSanitizer, MemorySanitizer and ThreadSanitizer. Adds an option which turns on pre-compiled headers and unity builds without having to make any underlying changes to the build system.

cmake-header-language

Elevator Pitch: Examine a header file to determine all of its dependencies and whether it is C only or involves C++. Many tools require that the language be specified manually for such headers.

 

If you can think of a better name for any of these, please let me know. I’ll take any suggestion!

Bringing back the old animations

One of the other casualties when we switched to using Modern OpenGL in Compiz was the loss of the older animation plugins, such as animationaddon, simple-animations, animationjc and animationsplus.

I took some time last weekend to make the necessary changes to bring them back to life and get them merged back into mainline.

One of the more interesting parts of all this was the polygon-based animations. You might remember this as the “glass shatter” or “explode” animations. Unlike most of the other code in Compiz plugins that did transformations on windows, the “polygon animation” mode actually completely took over window drawing. This meant that there was a lot more work to do in terms of getting them to work again.

glDrawElements

Compiz has had (for a few years now) a class called GLVertexBuffer which encapsulates the entire process of setting up geometry and drawing it. If you want to draw something, the process is usually one of getting a handle for something called the “streaming buffer”, resetting its state, adding whatever vertices, texture co-ordinates, attribute and uniform values you needed then calling its render method.

Under the hood, that would populate vertex buffer objects with all the data just before rendering and then call glDrawArrays to render it on screen using the defined vertex and pixel processing pipeline.

glDrawArrays can be cumbersome to work with though, especially with primitive types where you might end up having a lot of repeated vertex data. You have to repeat the components of each vertex for every single triangle that you want to specify.

glDrawElements on the other hand allows you to set up an array of vertices once, adding that array to the vertex buffer, then specifying a little bit later the order in which those vertices will be drawn. That means that if you were drawing some object in which triangles always had a point of (0, 0, 0), then you could just refer to that vertex as “1”, so long as it was the second vertex in the vertex buffer. This is very handy when you have complex 3D geometry.

Quite understandably, animationaddon’s polygon animation mode didn’t use glDrawArrays but glDrawElements.

In order to support both OpenGL and GLES it was necessary add some sort of support for this in GLVertexBuffer, since the old code was using client side vertex and attribute arrays. The quickest way to do this was to just add some overloads to GLVertexBuffer’s render method, so now as a user you can specify an array of indices to render. Its a little more OpenGL traffic, but it makes things a lot easier as a user.

Re-tessellation

All the geometry for those 3D animations was rendered using the GL_POLYGON primitive type. Polygons are essentially untesselated concave shapes. GLES only supports triangles, triangle fans and triangle strips which threw a spanner in the words.

The polygon animation mode supported splitting windows into rectangles, hexagons and glass shards.

At first I was wondering how to convert between the two geometries, but it turns out that for concave shapes there’s an easy way to split it up into triangles. Just take a reference point, then make a line from that reference point to each of its neighbours, bar its neighbours.

path4156.png

That can be represented with this simple function:

namespace
{
    enum class Winding : int
    {
    Clockwise = 0,
    Counterclockwise = 1
    };

    /* This function assumes that indices is large enough to
     * hold a polygon of nSides sides */
    unsigned int determineIndicesForPolygon (GLushort *indices,
                         GLushort nSides,
                         Winding direction)
    {
    unsigned int index = 0;
    bool front = direction == Winding::Counterclockwise;

    for (GLushort i = 2; i < nSides; ++i)
    {
        indices[index] = 0;
        indices[index + 1] = (front ? (i - 1) : i);
        indices[index + 2] = (front ? i : (i - 1));

        index += 3;
    }

    return index;
    }
}

Depth Buffer

We never really used the depth (or stencil buffers) particularly extensively in Compiz, even though the depth buffer is a common feature in most OpenGL applications.

The depth buffer is a straightforward solution to a hard problem – given a bunch of geometry, how do you draw it so that geometry which is closer to the camera is drawn on top of geometry that is further away?

For simple geometry, the answer is usually just to sort it by Z order and draw it back to front. For the vast majority of cases, compiz does just that. But this solution tends to break down once you have a lot of intersecting geometry. And those animations have a lot of intersecting geometry.

Incorrect Depth Buffer.png

Note in this image how the white borders around each piece are drawn on top of everything else?

The better alternative is to use the depth buffer. It isn’t perfect and doesn’t allow for transparency as between objects whilst the depth buffer is enabled, but it does handle the intersecting geometry case very well.

The way it works is to create an entirely separate framebuffer where each “pixel” is a single 24 bit floating point number. Compiz uses an implementation where the other 8 bits are masked out and used for the stencil buffer. Every time OpenGL is about to write a pixel to the framebuffer, it keeps track of how far away that pixel is in the scene. It does that during something called the “rasterisation stage”. This is where a determination is made as to where to draw pixels. That’s done by interpolating between each vertex to reach a position and its relatively trivial to keep track of depth too by similar methods. Then, OpenGL compares the depth to the existing value at that position in the depth buffer. The usual depth test is GL_LESS – so the value in the depth buffer is updated and the framebuffer write is allowed.

The result is that parts of geometry which were already occluded are simply not drawn, where as geometry which occludes other previously-drawn geometry overwrites that geometry.

Correct Depth Buffer.png

I this image, you’ll notice that each piece correctly overlaps each other piece, even if they are intersecting.

Trying it out

The newly returned plugins should be back in the next Compiz release to hit Yakkety. They won’t be installed or enabled by default, but you can install the  compiz-plugins package and compizconfig-settings-manager to get access to them.

If you’re ever curious about how some of those effects work, taking the time to re-write them to work with the Modern OpenGL API is a great way to learn. In some cases it can take a lot of head-scratching and debugging, but the end result is always very pleasant and rewarding. There’s still a few more to do, like group, stackswitch and bicubic.