Google Apps, Mail.app and the iPhone

I've started consolidating my domains' mail and calendaring services under Google Apps. Besides being free for the standard edition, Google provides many of the same services I was using at my previous host, LuxSci, plus group calendaring, docs, etc. However, the support of labels in Gmail is what has really intrigued me.

I've been an IMAP user for years. I'm extremely organized with my email, using a large number of filters on MacOS X's Mail.app to filter email into the right folder for organization. Unfortunately, when I got my iPhone last summer, I began to realize this strategy had its problems. Namely, if Mail.app is downloading and filtering email to folders other than the Inbox, I would miss them on my iPhone, due to its failure to synchronize the entire IMAP folder structure. To handle this, I would disable mail checking from my Mac when I was away from it, allowing all mail to flow to my inbox where I could check it on my iPhone. Unfortunately, it means the hundreds of emails I receive daily from mailing lists were causing me to check email incessantly, often to be dissapointed to only find mailing list messages which could wait to be reviewed later. There had to be a better way.

My goal was to have mailing list email automatically filtered to the appropriate folder, skipping the inbox and the related notification(s) while allowing friends and family mail to hit my inbox where I was guaranteed to see it in a timely manner. This would require strong filtering on the server, which GMail provides (sort of... more on this in a moment). I also want to ensure that seen friends and family mail is filtered into the appropriate mailbox as well, requiring a filter that runs occassionally to move read mail to its organized spot in my mail hierarchy. Unfortunately, this is not supported by GMail.

What's a geek to do? While researching the use of labels for GMail I realized that I could have inbound email skip the inbox for mailing lists (yay!) and I could label friends and family mail with the appropriate label, i.e. [Gmail]/Friends, while retaining the default Inbox label as well. Mail.app treats Gmail labels as IMAP folders, therefore, mail labeled with Inbox and [Gmail]/Friends will appear in both my Inbox and my friends folder at the same time! More importantly, if I read the mail in either folder, it is automatically marked as read in both folders with a slight lag-time between IMAP folder synchronizations. But the best part is that if I delete the mail from Inbox, it only removes the Inbox label, effectively "archiving" the mail in my friends folder. Suh-weet! So how does this work in practice?

Let's say an email from a friend arrives and is placed in the inbox and friends folder. If I am using my iPhone, I receive a notification and read the email. I can delete the email from the inbox, removing the label and ensuring that it is archived in the Friends mailbox. If I am using Mail.app, I'll probably first notice the email in my Friends folder and read it there. In either case, it is marked as read and won't show in the unread count in either folder. This is almost ideal, except that when I am using Mail.app the email shows up in both mailboxes, a less than stellar behavior, but one I can deal with. Also, if I read the email in my Friends folder, a copy of the read email still remains in the Inbox where I have to manually delete it. I may handle this with a filter in Mail.app to clean up such occurrences in the future, but for now it doesn't seem to be a big issue since the vast majority of my email is from mailing lists.

The biggest problem I have so far is with Gmail's filtering. It is teh suck. I need a few abilities that it doesn't support:

• Filter ordering and a "stop processing rules" function on filter actions. I need mailing list filters to run first and then friends, family and other filters. Too many people are friends and regular mailing list posters and I don't want to have mailing list mail show up in my inbox, the appropriate mailing list and my friends mailbox. For now, all friends filters must also contain a filter which doesn't match mail sent to mailing lists.
• Filtering based on contact groups. Instead of writing a filter to use many different email addresses, all of which get pushed to the same folder, I should be able to filter based on the group a contact belongs to instead. This will make filter management a LOT easier.

Overall, I can't complain. For a free service this will meet my needs for the near future and save me a few bucks in the process.

Ubuntu Time Capsule Update

I finally did a disaster recovery using my Ubuntu Time Capsule replacement. It took about 5 hours to recover a 160GB drive (95% full) over my home network. Started right up and worked like I had left off just a few hours before!

Here's the set up to use:

Start up from the MacOS X disk and open the Terminal from the Utilities menu. Type the following:

cd /Volumes
mkdir ./YOURDIRNAMEHERE
mount_afp -i afp://ip_address_of_ubuntu_box/VolumeName /Volumes/YOURDIRNAMEHERE
ls -al ./YOURDIRNAMEHERE

If you find the disk image volume in the listing, you're all set. Exit the terminal and using the Utilities menu you can select Restore from Backup Disk.

Sweet.

The Most Difficult Call...

I just made what is possibly the most difficult call of my adult life: the call to schedule an appointment for Java to be euthanized. I'm sitting here with Java at my feet, crying while I type.

Java in the past year has has a number of medical issues: A huge mass in his abdomen, the baby-wipe swallowing incident(s), some type of gastrointestinal issues which caused a large case of the "vapors" emanating from either end and now, diabetes. The diabetes came on suddenly just before the end of the year. One day Java started looking thin, then we noticed he wasn't eating as much, then he wasn't eating at all. He was drinking excessively, and urinating excessively too. He was up at all hours of the night wanting to go outside, probably to drink and pee. We tried feeding him different food, more food, more different food, but he wouldn't eat enough to sustain himself.

One trip to the vet told us he had severe diabetes and was acid ketodotic. His glucose numbers were off the chart. Java has been on insulin since the 2nd, with his doses steadily increasing. We've tried twice now to get a glucose curve to check his response to the increasing insulin doses. These have been to no avail, his numbers are still off the chart. Java is now below 60 pounds, a shell of his former self, all skin and bones. He's now nicknamed "Kate Moss" for his lithe figure. Its easier to laugh about the situation than it is to cry over what we know is coming.

Java can no longer run around, walk up and down more than a handful of stairs and often has trouble getting up off the floor by himself. He spends most of each day in his bed, which has been moved downstairs since he can no longer go up to the bedroom. When he does get up to go outside, its a slow process as he carefully makes his way down the front steps and, sometimes, back up them. The past few nights I have carried him upstairs so he could sleep with us in the bedroom as he has always done. I have also been awaken by him each night to carry him downstairs and outside so he can "do his stuff."

When I picked him up at the vet yesterday we spoke about Java, this treatment and possible outcomes. He's never going to be the same dog again. We can continue to treat him and try to get his diabetes under control, but what else is wrong? What caused this sudden onset of diabetes? At 12 and 1/2 years old, he's a very senior dog. The average lifespan for a lab mix is 10 years.

Last night I knew what the next step should be. Steph and I discussed it ad nauseum, but it wasn't until this evening that we finally were able to make the decision.

The past twelve years have been great. Java has been my companion, my friend, my chick magnet (when he was a puppy!) and a part of my family. He's given me unconditional love, something we all need to learn a little more about. He's been with me as I was a struggling graduate student, trying to find my way in the world. As a grad school dropout, falling into computers and programming. He's been there through many (ex-) girlfriends, a wife, a new dog (Lucy) and the birth of Maya. He was Maya's protector when she was a newborn. I had hoped he'd be around as she grew older, since he is so well behaved with children...

I know I am doing what is best for Java, even though it feels like I am betraying him by failing to take care of him when he is most in need. He was always there for me when I needed him, I feel like I can't do the same right now... This would be so much easier if nature would take its own course, rather than forcing me to make this choice.

We've had a tradition for many years now. On every birthday Java gets an ice cream cone. First it was from McDonald's, later we found that Brewster's Ice Cream does a "dog bowl" which includes a small dog treat. Later this evening I'll be headed to Brewster's to buy Java one last cone. A big vanilla waffle cone, just for him. One last treat for my friend.

May peace be with you Java. You always be with me in my memories.

Ubuntu Geekery Part 2

A few notes since I first got set up:

You can configure printers directly through the CUPS web interface.

Parallel printing was throwing off lots of errors and generally unreliable. I changed the printer to a USB connection with a USB -> Parallel cable and created a new printer through CUPS. Everything works like a charm!

ssh has been locked down to use keys instead of passwords and only allows specific users a remote login. Now this has been opened up to the world at large so I can ssh into the home server.

OpenVPN is set up and running, but I am still having some networking issues and no time to fix them...

Also up and running but not yet working is saned, a scanner daemon. I can't seem to get this working with my OfficeJet 5510, I may try a standalone scanner I have sitting around instead.

I still have yet to to a disaster recovery scenario to validate the backups... perhaps this weekend. The back up server has been rock solid, except for a minor hiccup with Steph's backups...

Geeking Out - Setting Up an Ubuntu Server

I've been working on this project for a few weeks now and its finally coming to completion, so I thought I'd document it here so others can save some of the time and hassles I have had in getting to this point.

When I bought a new MacBook for myself, I convinced Steph to ditch her P.O.S. Windows XP machine and take my old MacBook. We now needed a few thing:

• Home file server for sharing documents, photos, etc
• Automated back up system (TimeMachine would be ideal, but I didn't want to purchase a TimeCapsule for the network based backups. I'm cheap!)
• Shared printing (again, I'm cheap, I don't want to buy a new printer and my existing printers don't support networked printing)
• Home-based VPN to allow me access to the home network when on the road

So the obvious solution was to repurpose Steph's old Dell box into a server to support our needs. Here's what I did:

First, I replaced the hard drve with an unused 160GB SATA drive I had sitting around. After burning a CD from an ISO of Ubuntu 8.10 Server (Intrepid Ibex), I installed the server and got it up and running on my home network with a static IP address. The server is called "papaya". The initial install included an ssh daemon for remote access so I could work away from the console. Eventually the box will be headless, but for now its connected to my home office KVM.

Next, I configured ufw (Uncomplicated Firewall) to start locking down services. First, set the firewall into the "default deny" mode (a.k.a. whitelisting of approved inbound connections):

sudo ufw default deny

And then I added a rule to allow inbound ssh from the local network:

sudo ufw allow proto tcp from 192.168.5.0/24 to 192.168.5.2 port 22

Next I installed netatalk to support AppleTalk, allowing me to connect my MacOS X systems to papaya's shared disks. I started with directions from here and here, but I found they didn't work exactly as presented, and I needed some tuning to make it work. (See the referenced links for more details on all of these steps, I just distilled this down to the most important info.)

First, install netatalk with encryption support (required by MacOS X):

sudo aptitude update
mkdir -p /tmp/src/netatalk
cd /tmp/src/netatalk
sudo aptitude install devscripts cracklib2-dev dpkg-dev libssl-dev
apt-get source netatalk
sudo apt-get build-dep netatalk
cd netatalk-2.0.3
sudo DEB_BUILD_OPTIONS=ssl dpkg-buildpackage -us -uc
sudo debi
echo "netatalk hold" | sudo dpkg --set-selections

Note the last line, it places a hold on any updates to netatalk when running updates on the Ubuntu server, this is necessary to prevent updates from breaking the installed version. Next I configured netatalk by editing a few files. First, /etc/default/netatalk was modified with the following configuration:

ATALKD_RUN=no
PAPD_RUN=no
CNID_METAD_RUN=yes
AFPD_RUN=yes
TIMELORD_RUN=no
A2BOOT_RUN=no

Next, edit /etc/netatalk/afpd.conf by adding the following line to the end of the file:

- -transall -uamlist uams_randnum.so,uams_dhx.so -nosavepassword -advertise_ssh

Now, configure your shared volumes in /etc/netatalk/AppleVolumes.default by adding the following lines to the end of the file (make sure its one line!):

/media/TimeMachine TimeMachine allow:username1,username2 cnidscheme:cdb options:usedots,upriv

This mounts a shared volume named TimeMachine located at /media/TimeMachine (your mount point may vary, this is an external 1TB USB drive at the moment). List the specific users who should be able to login to the share with their credentials. The remaining options have to do with dealing with invisible files/folders which are used by MacOS X. I actually created a handful of shares, each of them with different functions (TimeMachine, generic file share, etc.).

Now we restart netatalk:

sudo /etc/init.d/netatalk restart

Unfortunately, this share is not publicized to the local network, which I wanted for ease of use. So I set up Avahi, a daemon for Apple's Bonjour zero-configuration service:

sudo apt-get install avahi-daemon
sudo apt-get install libnss-mdns

Edit /etc/nsswitch.conf to add the mdns service to the end of the line beginning with hosts. In the end the line should look like:

hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4 mdns

Next, configure Avahi to publish the service. Create a file with the name /etc/avahi/services/afpd.service containing the following XML:

<?xml version="1.0" standalone='no'?><!--*-nxml-*-->
<!DOCTYPE service-group SYSTEM "avahi-service.dtd">
<service-group>
<name replace-wildcards="yes">%h</name>
<service>
<type>_afpovertcp._tcp</type>
<port>548</port>
</service>
<service>
<type>_device-info._tcp</type>
<port>0</port>
<txt-record>model=Xserve</txt-record>
</service>
</service-group>

Note that this sets up Avahi on port 548 and tells Avahi to publish the device as an XServe, which will make papaya appear as an XServe icon on any Macs connected to the network. Open port 548 to listen for inbound connections:

sudo ufw allow proto tcp from 192.168.5.0/24 to 192.168.5.2 port 548

Restart Avahi to publish the servers:

sudo /etc/init.d/avahi-daemon restart

Woohoo, now I have the server publishing its shares to the network which are seen by my Macs, as seen below.



Now we need to configure the Mac to be able to use papaya as a TimeCapsule without paying for Apple's hardware. Drop to a command line and run the following (on your Mac, not Ubuntu!):

defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1

Mount any of the remote volumes from the Mac and use the TimeMachine preferences to select the remote system for backups. TimeMachine will actually create a disk image on the remote system and utilize that image for its backup volume. Or, in my case, maybe not! But we can easily fix that. Use MacOS X Disk Utility to create a new disk image as a "sparsebundle disk image". The naming is very specific and in the format machine_name_MAC_ADDRESS.sparsebundle. For instance, if your machine name is joeyjr and the MAC address of the primary interface (en0) is 00:13:24:1f:12:10 then the name of the disk image should be:

joeyjr_0013241f1210.sparsebundle



Note that the MAC address separators have been removed. How do you find the MAC address of the primary adapter? Drop to a command line and execute the command ifconfig, it should return something like:

lo0: flags=8049 mtu 16384
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
gif0: flags=8010 mtu 1280
stf0: flags=0<> mtu 1280
en0: flags=8863 mtu 1500
inet6 fe80::223:32ff:febf:cd2%en0 prefixlen 64 scopeid 0x4
inet 192.168.5.114 netmask 0xffffff00 broadcast 192.168.5.255
ether 00:23:32:bf:0c:d2
media: autoselect (100baseTX ) status: active
supported media: none autoselect 10baseT/UTP 10baseT/UTP 10baseT/UTP 10baseT/UTP 100baseTX 100baseTX 100baseTX 100baseTX 1000baseT 1000baseT 1000baseT
en1: flags=8863 mtu 1500
inet6 fe80::223:6cff:fe7f:5cd%en1 prefixlen 64 scopeid 0x5
inet 192.168.5.3 netmask 0xffffff00 broadcast 192.168.5.255
ether 00:23:6c:7f:05:cd
media: autoselect status: active
supported media: autoselect
vmnet8: flags=8863 mtu 1500
inet 192.168.94.1 netmask 0xffffff00 broadcast 192.168.94.255
ether 00:50:56:c0:00:08
vmnet1: flags=8863 mtu 1500
inet 192.168.200.1 netmask 0xffffff00 broadcast 192.168.200.255
ether 00:50:56:c0:00:01

The lines after en0 refer to the primary adapter, in this case the ethernet adapter, and the MAC address is found on the line in bold.

Create a sufficiently large image (i.e. 500GB) (note: this image will only take a few MB until it is filled, so this can be done on a pretty full disk) and move it to the root of the share you wish to use for TimeMachine.

Mount the TimeMachine share from above. Open the System Preferences, choose the share as your TimeMachine volume and let it go to work. If you did everything correctly, bacups should begin. My 160GB backup took at least 8 hours, if not longer.

Now backups are great, but they only work for me if you can restore from them in a crisis. How do I define a crisis? Not restoring a file that has been accidentially deleted by using the TimeMachine application, that should work as long as I have the volume mounted. How about Installing MacOS on a blank drive and having the MacOS installer pull all of the data from your networked backup?

In theory, everything should work as follows. I have not done a full disaster recovery yet (though I will before I depend on this as my only backup solution) so take all of this with a large grain of NaCl. Boot from the install CD and start the terminal from the Utilities menu. In the terminal, mount your afp file share:

mkdir /Volumes/timemachine
mount_afp afp://username:password@papaya/TimeMachine /Volumes/timemachine

If the disk is mounted you should be able to find your .sparsebundle in the directory /Volumes/timemachine. Exit the terminal and now open the Restore from Backup utility. The restore utility will find the remotely mounted TimeMachine volume and allow you to restore from that point.

OK, we now know we can back up and restore from the backup. That's all good news. Let's get some printers set up. I have 2 HP printers, a HP LaserJet 1100 which runs on a paralell port, and a HP OfficeJet 5510 All-In-One, running from the USB port. First, ensure hp-lip and cups are installed:

sudo apt-get install hplip
sudo apt-get install cups

Run the hp-setup command to set up your printers (follow the prompts):

sudo hp-setup

Configure cups to listen on port 631 for remote printing and configuration by editing /etc/cups/cupsd.conf

# Allow remote access
Port 631
Listen /var/run/cups/cups.sock

And configure cups to allow remote administration at http://papaya:631/:

<Location />
Allow all
Allow all
# Allow shared printing and remote administration...
Order allow,deny
Allow all
</Location>
<Location /admin>
AuthType Default
Require user @SYSTEM
Allow all
# Allow remote administration...
Order allow,deny
Allow all
</Location>
<Location /admin/conf>
AuthType Default
Require user @SYSTEM
Allow all
# Allow remote access to the configuration files...
Order allow,deny
Allow all
</Location>

See http://ubuntuforums.org/showthread.php?t=736489 for more information on configuring cups. I'm not perfectly happy with this yet, I believe it can be locked down further, but its good for now.

Now, open the firewall to allow access on port 631:

sudo ufw allow proto tcp from 192.168.5.0/24 to 192.168.5.2 port 631

Restart cups:

sudo /etc/init.d/cupsys restart

Now I browse to http://papaya:631/ to manage configured printers. With the printer URLs in hand I can add remote printers to my MacOS X machines through the Printers preference pane. Printing on the LaserJet is a bit wonky at times, but printing to the OfficeJet is perfect. I still need to set up remote scanning, but that's a project for another day.

The VPN is still a work in progress. I'll post details on setting up OpenVPN once I have had chance to configure and test it completely.

(All the info posted herein is for informational purposes only. Use this at your own risk as I will provide no support.)