After the fire I have followed a manual process of backing up our files on an external drive that I store in our fire safe. The problem with this is it requires me to actually do the work, which I often put-off. When Amazon S3 was introduced I immediately saw the potential to use it as an automatic remote backup source. I hadn’t invested much time in it up until now, but I just got a new computer (MacBook Air!!) and while setting it up I thought it would be a good opportunity to get my backup situation in order.

There are some great tools already in existence that can do most of the heavy lifting for you. The primary tool for doing remote directory syncs is called s3sync which is a script written in Ruby. Lucky for us OS X comes with Ruby pre-installed so there isn’t much work to get it working.

Here is my step-by-step guide to getting your machine setup to do automatic daily backups to Amazon. I developed these steps on my MacBook Air running Leopard however they should work for previous versions of OS X as well.

Step 1) First off, your going to need and Amazon Web Services account. Head over to http://aws.amazon.com/ and sign-up for an account to use S3. The prices are very cheap ($0.15/GB/Month). Once you have your account setup you will need two things to use Amazon S3. Your Amazon access key and your secret key. These are what s3sync will use to authenticate you to Amazon.

Step 2) I’ve packaged together a zip file with all the files you are going to need to get this setup along with SSL. Download the file at http://images.vallery.net/s3backup.zip. You can go to http://s3sync.net/ to see if a newer version if you like but you’ll need to figure some of this out on your own.

Step 3) You need to create a “bucket” in amazon to store your files. A bucket is similar to a folder, however it is globally uniquely named across all Amazon S3 users. In order to create the bucket you are going to need one of the S3 GUI applications that exist. I have included in the zip file the one I have used called “S3 Browser”. You can find the latest version at http://people.no-distance.net/ol/software/s3/. Once you launch S3 browser click on “connection” then “new connection”. You’ll need to provide the access details you got from Amazon in step 1. Once you have connected click the “Add” button which will allow you to create a new bucket. Because the name has to be globally unique I used “vallery-macbookair-backup” where vallery is my last name. Keep track of this bucket name because you need it in the next step.

Step 4) Once you have the zip file I created downloaded it should automatically extract itself into your downloads folder creating a new folder called “s3backup”. Within the s3backup folder are all the files and scripts you will need in order to get this working. There is one key file that needs to be edited in order to make this all work which is called “backup.sh”. Open the file “backup.sh” and replace the place holder access key, secret key, bucket name with the ones you obtained form Amazon and step 3.

Step 5) Now that you have all the files ready to go you need to select a place to store them. The application will run as root at the system level in order to prevent file access issues, therefore I recommend storing the entire s3backup folder in your /Library folder. You should copy the entire folder using finder to /Library. There are a few other paths in “backup.sh” that will need to be updated if you choose to store the file elsewhere.

Step 6) You need to setup your Mac to automatically run the backup shell script on a regular interval. There are a couple ways to do this. Since I am Unix guy I immediately started looking at cron. I discovered however that Apple recommends you use launchd for scheduled tasks. It is fairly complex to setup a scheduled task using launchd but thankfully someone has already created a simple GUI that will let you do it. The application Lingon can be used for this. I’ve included the latest version at the time of writing this in the s3backup directory but you can always obtain the latest version from http://lingon.sourceforge.net/. Once you have launched Lingon you need provide some information. Click the “New” button to start a new agent. Choose “Users Daemons” so that the script will run as root and have access to all of the users on your Mac. Once you have created your new daemon you need to give it a name. I recommend something like com.vallery.s3backup where vallery is your name. You need to give the command line action for what to execute. Again, this assumes that you have stored the s3backup folder in /Library. Enter: “/bin/bash /Library/s3backup/backup.sh > /dev/null”. Lastly you need to give it a schedule as to when to run. I have mine setup to “At a specific date” with “Every day” selected and the time set to 4:00am. This is great if your leave your Mac on all the time. You might select a different option so that you can make sure your Mac isn’t in use when it is doing the backup. Click the “Save” button. It will require you to type in your admin password and then restart your computer.

That is it, your system should run the first backup as schedule. It will take a long time initially as the upload speed is limited to your internet connection. Once the initial upload has taken place it will only upload files that are new or have changed going forward. The script is setup to backup everything in the /Users folder. If you would like to limit what is being backed up you can change this to something else.

In the unfortunate event you actually need to get data out of the s3 store there are a number of applications that you can use to do this. Initially I have been using Panic’s Transmit however it seems to have problems with the way s3sync is storing the data. I found another great free app called “S3 Browser” which has worked well for me. You can also use the Firefox plugin S3 Fox.

I discovered that over the years I haven’t exactly been great about maintaining my addressbook. I’ve lost touch of many of acquaintances that I have had casual communication with. I realized that their email addresses where trapped in the deep bowels of my Gmail account, if only there was some way to extract them. I quickly realized that using the newly released IMAP protocol I could probe every message and then extract out the email addresses from it, and in some cases even additional data like the first and last name. I started playing around with the scripting a bit and came up with what I have now. This tool goes out to the Gmail IMAP server and downloads the message header from ever email that is stored in my Gmail account (except the SPAM folder). It pulls them into a master list, along with the first and last name if available. After all of the emails have been extracted it calculates some basic statistics including frequency of occurrence, which it users to sort them on. All of this information is then exported into CSV files that are compatible with many different applications.

While doing this I discovered an additional use for this data. On many of the social networking sites like Facebook and LinkedIn they will allow you to import a contact list file and find everyone that has registered for the service that you already know. This turned out to be a really killer app for this functionality. The only challenge was that they limit your ability to upload contacts to about 2,000 at one go. I added some additional functionality to my extract that “chunked” the file into several files, each with 2,000 email addresses in it. This allowed me to upload my newly discovered contacts a bit at a time, which worked very well.

If you would like to extract the email addresses from your Gmail or Google Apps for Domains account head over to http://vallery.net/gmail/ and give it a whirl!

If you point your browser over to http://vallery.net/postback/index.php you can see it in action. It will report back to you exactly how your browser is identifying itself, including any cookies you might have received from my word press blog!

Now the next time you are writing an application and you want to debug your outbound posts, just send them over to the above URL and it will respond with exactly what it received.

Pretty cool!

In no particular order my tips are:

1) Directory structures: In your personal document space (Your home directory on a Mac or Linux, “My Documents” on Windows) you need to create a document hierarchy to store your files. Depending on the amount of documents that you store, your system might include date as a component of the hierarchy so that your folders don’t become too large and unmanageable. I first divide my home directory up with a folder for each of the categories of documents that I might have. Word processing, spreadsheet, presentation, and notes. Under that I have it broken down by type. These are categories of the types of projects I might be working on. At work, this is the name of which of my clients for whom the document is relevant. From there, I have sub-directories for each year. 2007, 2006, and so on.

2) File names: All of my files are named with a standard convention. The convention is PROJECT/CLIENT_DESCRIPTION_YYYYMMDD_TYPE.EXT. Including the document type is helpful because sometimes the file extension doesn’t always give it away, or on some machines file extensions might be hidden. So for example if I had a Excel document for a client named Acme that contained a data extract, I would name the file ACME_DATAEXTRACT_20070410_EXCEL.xls

3) Music: I use iTunes to manage my music library. Part by force, since I own 4 iPods and an Apple TV, but part because I enjoy it. I think iTunes is a great interface for music management. I’ve done a few extra things in iTunes that help me stay organized. One of the most important data fields on a music track I find is “Album Artist”. This field is a recent addition to iTunes, but it really makes finding my music easier, especially from the iPod interface. For the music you purchase from the iTunes store, this is typically done for you already. If you’ve ripped your CDs, the CDDB database doesn’t always have this information however. The first thing I do after I’ve ripped a new CD is go and verify if the Album Artist field has info, and that it is correct. If not I manualy update this field based on the artist. This is especially useful on CDs that have a large number of collaborations on them. The second iTunes tip I have is use the rating system. When you are listening to your music, take the few seconds to go and rate the song. If you have a large music library, it is really handy to be able to just put on a playlist of your top rated songs when friends are over. You don’t want them to know you have the Macarena in your library!

4) Feeds: I use Google Reader to organize my feeds. In Google Reader you are able to setup multiple folders that each can contain multiple feeds. I have a folder for friends blogs, a folder for tech news, a folder for blog research, and a folder for each of my other interests. I can quickly see where new articles are and view just the topics I’m interesting in looking at. I also setup keyword searches on Google Blog Search and Google News and add the feed into Google Reader. This lets me get all the information I want, in a single easy to digest location.

5) Photos: We take a lot of digital photos, and we like to publish them online so that family can view them at their leisure. We have literally thousands of digital pictures that we have taken over the years. Keeping them all organized is no easy task. Recently I’ve begun the painstaking process of tagging all of the pictures in iPhoto. I tag the photos based on who is in the picture, the location of the picture, and anything that is particularly unique to the picture. Once the photos are tagged I upload them to our gallery using the iPhotoToGallery plugin. The gallery itself is powered by Gallery, a fantastic open source PHP based web gallery.

6) Address book: Now my address book organization is a bit tricky. As I mentioned above, I live both in the Mac and the Windows universe when it comes to my day job. At work we use Exchange and I carry around a Blackberry that is synced to Exchange. At home, I use my Mac, and I soon hope to have an iPhone. It’s a real challenge to keep my contacts synced between all these devices without spending too much time on it. In AddressBook 4.0 there is a handy feature that allows Exchange users to sync their Mac address book with Outlook Web Edition. Unfortunately for me, my company is still on the very old Exchange 5.5, and so I don’t have outlook for the web yet. While I openly admit this solution is far from perfect, it’s the best I have been able to come up with. Currently, I use my Blackberry as the transport medium for contacts. Exchange and my Blackberry work very well together, so I can always trust them. The problem is getting that same information onto my Mac. PocketMac makes a great utility that syncs your Blackberry with iLife. If I set this up as a one way sync (Blackberry to Mac) it works fairly reliable. The downside is that in order to add a new contact I have to do it with either my Windows PC or my Blackberry. I hope a better solution presents itself soon, or perhaps I might just dump my exchange dependancy all together when the iPhone arrives.

7) Chat contacts: Like most Mac users, I use Adium for my instant messaging. Adium is great because I can merge contacts from different networks into a single entry on my contact list and then set the network priority so I can choose which protocol to use if available. If you have the same person listed twice in Adium under your contact list, drag the contact from the lower priority network on top of the contact for the higher priority network. Adium will then ask you if you want to merge these contacts.
Another handy feature of Adium is having contact information pulled from your Address Book. This way their display name is listed as their real name instead of whatever catchy tag they chose to display as their alias. The last adium tip I have is how I organize my contact list. In Adium you can sort your contact list by status (under the view menu), and then choose to only view online contacts. This greatly reduces the number of contacts displayed and lets me quickly see who is available.

8 ) Email: Again, like Address Book email is a challenge in the hybrid Mac/Windows world I am in. Thankfully my email addresses are fairly separate. Work email stays on Exchange and personal email sits in Mail.app. Since I have my own domain name I use the fantastic Gmail for domains service. Google acts as my mail server and stores all my incoming messages. They are also kind enough to do spam filtering and provide an amazing ~16GB storage limit. I enable pop mail on the account and then fetch all my email down to Mail.app for local viewing and management. Once in Mail.app I use smart mailboxes to automatically sort out messages from the mailing lists I subscribe to. In Mail.app click the cog icon in the lower left, below your list of folders. From there, chose “New Smart Mailbox..”. With this interface you can specify under what criteria the mail message should be moved to the folder. For most mailing lists it is easy because typically a ID tag is prepended to the message subject. You can also use smart folders to create for individuals who email you a lot, so you can immediately see if you have new messages from them. That way you can give them your attention, or ignore them as the case may be.

9) Calendar: Ok, yet again I get bitten by the Exchange vs iLife issue. Similarly to how I manage my contacts I manage my calendar. I use the PocketMac software to import my calendar entries from Exchange/Blackberry into iCal. In iCal I have 2 separate calendars, work and personal. In the work calendar I have everything that is synced from the Blackberry, in the personal calendar I have everything that falls outside of work. This has worked out well because it allows me to stay organized but still keep my personal appointments completely private from my colleagues on our Exchange server. The only downside to this approach is that I do not receive reminders on my Blackberry for my personal calendar appointments. Again, hopefully the iPhone fixes this problem

10) Bookmarks: Now this is an area where the Windows/Mac problem is truly solved. I use del.icio.us to organize all my bookmarks. I can tag them so I know what they are about and I can easily retrieve them later. On Windows there is the new Firefox Extension and on my Mac and Safari I use Pukka combined with a simple javascript entry in my Bookmarks bar

javascript:document.location.href='pukka:url='+encodeURIComponent(location.href)
+'&title='+encodeURIComponent(document.title)+'&extended='+encodeURIComponent(window.getSelection());

When clicked, the above javascript sends your current URL to Pukka, which the gives you a prompt to tag it and upload it to Del.icio.us. The end results is that I have my bookmarks well organized and tagged on all of my computers. To access my bookmarks I just subscribe to the Del.icio.us RSS feed in Safari.

Can you think of any I might have missed? Have a solution to my Windows/Mac Address Book and Calendar problems? Drop a note in the comments.

Recently I had an article hit the front page of Digg. I was lucky because I had already done a number of things to optimize my server so that I survived. My page never went down, and was very responsive for the entire duration of getting dugg. My site is hosted on WordPress on a server that I own and manage. The server hosts about 15 different domains, none of which are exceptionally high traffic sites. The server isn’t anything special either. It’s a single processor P4 3.0GHz with 512MB of memory running CentOS 4.4, Apache 2.0, PHP5, MySQL4. I do have webalizer running on my Apache logs and I have the output of those posted at http://vallery.net/stats/ if you would like to see what the Digg effect can do.

I never exceeded more than a 0.15 load average during the Digg. Load average is a computation of how maxed out a Unix based system is. You multiply the number of processors by 1, and this indicates the maximum load average that you can reasonably sustain. If the load average exceeds this number, than your system is over utilized. In my case, my max server load is 1 given that this is a single processor system. This means I was only using 15% of my system resources during the maximum load that was generated from the Digg.

In order to optimize my system I have done the following things:

1) I’ve installed the WordPress 2.0 plugin wp-cache. This is something that every WordPress blog should have installed. Especially on a shared hosting environment, it will dramatically increase your ability to handle high traffic. The plugin generates the HTML for a given page and then saves it in a cache file. When someone accesses your page, instead of fetching the content from the database it uses the already generated HTML in the cache to send to the browser. This eliminates a number of fetches to the DB and dramatically speeds up page load times.

2) Optimize Apache/MySQL to handle the expected number of database queries. The most common error you receive when viewing a WordPress page that has been owned by the Digg effect is a database connection failed, or timeout. When connecting to a MySQL database you can either have a persistent connection, or generate a new connection for each request. There are a number of schools of thought as to which method is better, but generally speaking using persistent connections utilize more memory, so for me using a new connection for each request makes more sense. A lot of the PHP blog templates including WordPress use these non-persistent connections as well. The rub comes in the maximum number of said connections. MySQL can be configured in your my.cnf file to say the maximum number of connections that can be created to the database. If each instance of Apache (which in turn represents concurrent vistors to your page) has it’s own connection, then you can quickly exceed this maximum number of connections. Apache similarly has a configuration option that dictates the maximum number of threads that can be concurrently running. This number is specified in the httpd.conf file. If the Apache number is larger than the MySQL number (which is the case in most default configurations) when you have more Apache threads running than MySQL is capable of handling, you get our nice “maximum number of connections” error message.

In /etc/my.cnf you want to set the variable “max_connections”. I recommend something reasonably high like 250.
In your httpd.conf file for Apache you want to set the variable “MaxClients”. This should typically be the same number that you selected for MySQL.

3) Set reasonable Apache timeouts. This means that no individual thread/connection can monopolize the system bringing all the other queries to its knees. This protects you from rogue “edge” cases.

In your httpd.conf file set “Timeout” to a low number like 30 (measured in seconds) and “KeepAliveTimeout” to something like 3.

4) If your site makes it on to Digg, use real-time monitoring tools to measure your server’s health. There are a number of command line tools available to help in this regard. The first I would recommend is the tried and true “top” command. This will display the processes running on your machine along with their associated memory and CPU usage. Keep an eye and make sure things aren’t getting out of hand. The second tool that I use is called “Tcptrack“. Tcptrack will need to be installed on your machine, but once it is it will give you a real time view of your incoming connections and bandwidth usage.

Happy Digging!

I’ve been thinking about the best way to handle promoting a story/article from a queued status onto the main page and I’ve had a few thoughts I wanted to share with everyone.

The current scheme for promotion is very simple. Number of votes passing a defined threshold in the config file (and the story is fresher than X days). While this will work for very low volume sites, it doesn’t exactly scale well. In a site with a large number of users, and one would assume proportionately a large number of votes, this breaks down.

Our ideal system would be able to quickly determine that a story is of high value and promote it to the homepage based on the frequency of votes as compared to a large sample population. Using some basic statistics we can determine if a story is an “above average performer” and promote it quickly.

In order to accomplish this we need to take into consideration several variables, including:

1) Number of stories submitted in a given time period
2) The average number of votes over a given time period for all “active” stories in the queue
3) The standard deviation of votes over a given time period for all “active” stories in the queue
4) The target number of stories to be promoted to the home page in a given time period

For the purposes of explaining my ideas I am going to define the time period as one day. We need to calculate and story the answers to some of these questions on a regular basis. Ideally we create a new DB table that stores this information for us to easy lookup. This way we can also track information statistically and show trends. For my plans I plan on implementing a cron job that runs daily that calculates the required and stores them as follows.

1) Calculating the number of stories submitted is trivial. The SQL query I am using is:

select count(*) from links where link_status = ‘queued’ and date(link_date) = ’2006-04-16′;

2&3) Calculating the average and standard deviation can be done with the following query:

select avg(link_votes) as ‘average’, stddev(link_votes) as ‘stdev’ from links where link_status = ‘queued’ and date(link_date) = ’2006-04-16′;

4) This can be set in the config.php file and up to the site administrator.

These values will be used to “predict” the future for our new stories. Each story will have a new variable that stores a floating point number. This number is the number of standard deviations above or below the mean (average). We need an additional check running on a much more frequent interval (I plan on using every 5 minutes) to update the items in the database with their new “score” and promote them once they pass a specific threshold.

I plan on calculating this as follows:

$score = //Z score that indicated if a story is above/below the mean
$numofvotes = //The number of votes that a given story has received
$stddev = //Standard deviation for the given time period as calculated above
$average = //Average number of votes for the given time period as calculated above
$numberofstories = //The number of stories submitted in the given time period as calculated above
$desirednumofstories = //Setting from config.php

The cut-off value determines a “score” threshold, or essentially a percentile that a story must fall into in order to be promoted. I plan on calculating this as follows:

$score = ($numofvotes – $average)/$stddev;
or in SQL

select link_id, ((link_votes – $avg)/$stddev) from links where link_status = ‘queued’ and date(link_date) = ’2006-04-16′;

Once we have this score we need to decide if this story needs to be promoted or not. This is done by first calculating where in the rank order this story is likely to fall for the day using some basic probability statistics.

$rank = round($numofstories/(1+exp(-1.7*$score)),1);
Now we just check and see if that rank falls above our threshold and promote the story accordingly.

$desiredrank = ($numberofstories – $desirednumofstories)
if ($rank >= $desiredrank) then {
// Update story and set to promoted
}

This method should work well assuming that traffic is fairly stable from day to day. Since we are using the previous days data to predict the current days volumes if a sudden traffic spike is hit it will mean that a larger number of stories will be promoted than desired. This can be mitigated on larger volume websites by decreasing the “given period of time” from a day to something shorter. Additionally this whole idea could be retooled to calculate these variables on a per category level. Some sites might have much higher traffic, and therefore votes for one category than another.

I’m currently working on implementing the above for a site that I plan on launching in the near future, and I would love feedback, criticism before I do it.

Here is a simple step-by-step guide to walk you through what I did:

1) Install xvid, divx, and wmv codecs.

These can be found here:

Divx
Xvid
WMV

2) Install the automator actions for compressing and importing into iTunes.

Download Quick Time Compression Actions and Workflow

3) Once installed you will have a directory on your desktop called “QuickTime Compression Workflow resources”. In this folder you will find a sample automator workflow called “Convert videos and add to iTunes”. Open this workflow in automator.

4) Delete the first step of the workflow which is “Ask for Finder Items”. Instead of being prompted for which items to convert, we want to setup a folder action that will automatically convert the files dropped in our folder.

5) Add a new first step to the workflow called “Get Selected Finder Items”. This action can be found under the “Finder” application.

6) Under the “Compress QuickTime Using Most Recent Settings” step change “Choose directory for converted files” to the desktop (or any other temporary folder you want to use).

7) Create a new folder on your desktop. This will be your drop folder, so call it something relevant. I called mine “Convert to AppleTV”.

8 ) Back in automator, click on file and choose “Save as plug-in”, choose “Folder Actions” from the “plug-in for” drop down. Give the plug-in the same name as your folder. Select your newly created folder for the “Attached to folder” option. Click save.

9) Since the script will convert whatever file you drop in your conversion folder using the last settings you used in QuickTime you’ll need to launch QuickTime with a test file and then choose “Export” from the file menu. Assuming you have the most recent version of QuickTime Pro you should have an option “Export Movie to Apple TV”.

10) That’s it!!! Now just close out of everything and drop your files into your new folder and watch as they are converted and imported to iTunes. It works great to leave your Mac on and then drop a bunch of files in the folder before you go to bed. When you get to your PC in the morning everything should be all ready to go.

To find out more about folder actions, check this page out:

Folder Actions