Markus Thielmann is a Berlin based open source enthusiast and developer. Read more...
Use Amazon S3 with Ubuntu Hardy
by Markus Thielmann on 2008-05-10
Amazon S3 is an online storage web service offered by Amazon. I tested it back in 2006, since I was quite surprised that Amazon offered such an innovative product.
I found JungleDisk to be a nice and free utility to access your storage space, since Amazon doesn't offer standardized WebDAV access. While I was still hoping that Jungle Tools would release their software as free and open source, they decided to charge a (reasonable) fee of $20 for this software. I lost interest in continuing testing, since I intended to use S3 as backup space and still prefer open source software for critical tasks.
There were some open source projects trying to replace JungleDisk, but I couldn't get one to work reliable enough for a backup solution.
I was really pleased when I read that Steven Harms reviewed s3fs, a FUSE file system for Amazon S3.
Assumed you already own a Amazon Web Service Account, just make sure you've got Git and Python bindings for fuse installed:
sudo apt-get install python-fuse python-boto git-coreSince there is no Ubuntu package of s3fs available, checkout the source code:
git clone git://git.fedorahosted.org/s3fs ~/s3fsTo play around with s3fs, just start it up with:
# create a bucket, while providing your key pair.
~/s3fs/src/s3fs -C -c <bucketname> -p "<aws access key>" -s "<aws secret access key>"
# format bucket
~/s3fs/src/s3fs -C -f <bucketname> -p "<aws access key>" -s "<aws secret access key>"
# create mount point
mkdir ~/backupS3
# Unfortunately, I couldn't convince s3fs to accept the key pair as a command line option when mounting.
# So, just export them:
export AWS_ACCESS_KEY_ID=<aws access key>
export AWS_SECRET_ACCESS_KEY=<aws secret access key>
# mount it!
~/s3fs/src/s3fs -o bucket=<bucketname> ~/backupS3/Finally you're able to use your S3 space with GUI-tools.
The s3fs project still considers the software as quite beta, so be warned:
"WARNING You should not yet store any data that you do not have otherwise backed up on s3fs! development on this filesystem is early enough that data loss/corruption may occur!! "
It works quite nice for me. I just wished s3fs could provide some visual feedback when it finishes copying. An integration with gvfs would be great, so I could get an simple progress bar for the copy process.
Comments
I used JungleDisk and I really liked it--it worked great--but a few things turned me away from it.
1) It's non-free, which I struggled with the whole time.
2) It was very poorly labeled as a trial, so I ended up putting all my data on s3 via JungleDisk and then the trial ran out. I was then reminded one of the key reasons why I prefer free software: If I did not pay for JungleDisk I would not be able to access my data again. EPIC FAIL!
I ended up emailing them and explaining the situation. They were very professional and let me access my data for a short while so I could retrieve it again. I think that experience however reminded me that non-free software only traps the user it never frees the user. I'll continue to look for a free-software solution, I just hope I can find one as robust as JungleDisk.
Let's hope s3fs will fill that gap. It seems a Ubuntu package is on it's way: "Received some email with some interest from Ubuntu. Looks like there may be .deb packages of s3fs available soon :)"
How do you find the "bucketname"?
Just choose an arbitrary one. Bucket names are unique through the whole S3 system.
After running the "mount" command I get the following error:
Need to specify AWS_ACCESS_KEY_ID
Need to specify AWS_SECRET_ACCESS_KEY
Unable to establish configure this mount
I used the export commands. What did I do wrong?
Thanks
I had the same problem two days before I posted this article. I tried it again with an updated version of s3fs, so I thought it was only a temporary problem. You might want to try it again in a few days (just delete the s3fs directory and get a new copy).
Post new comment