One further question – does your setup allow your database cluster to automatically or “elastically” scale up and down (horizontally, if that makes any sense!) based on demand? it sounds as though increasing or decreasing the number of nodes in the cluster would require manually creating or removing a new server instance and an EBS volume for it to persist to, mounting the EBS volume on the instance, then somehow adding that node into the cluster.

Another hosting solution our team looked at was Google App Engine. GAE provides this kind of database elasticity by default, and can host both Java and Python, but also has its own limitations and the data store isn’t a relational database. I’m guessing the lack of mass file storage on GAE would have been an issue for you too. For us, the problem was not being able to run Apache Jackrabbit using the data store and without Java threading support.

Yes, you’re right: we use EBS to persist data permanently. You’re also right when you state that EC2 instances don’t have non-volatile storage (once you shut down the instance, it’s all wiped clean). EBS works by creating a virtual drive, known as a volume. You decide how large you want the volume (minimum is 1GB, but can go into the tens of TB’s). Once you have created your volume, you can then attach it to any one of your instances with a few commands, or, using the AWS console, you can do it by just pointing and clicking. Once you have attached the volume to your instance, you will need to create a file system on it (if it’s brand new and has never been used).

When you attach your EBS volume, you define a device type, like /dev/sdaf, so on your instance you can simply mount a new drive as if it were a new hard disk. Then, configure your database node (or whatever system you want to persist data to to use that mount point instead of writing to the local file system) and voila! you have non-volatile storage. When you are done with your instance, when you shut it down. Your EBS volume will simply be unmounted and wait to be used again. AWS have also given the ability to create snapshots of your volumes so backups are a breeze! If you configure your AMI to always use your new EBS volume, you will have a pretty robust setup. You can create as many volumes as you like and attach and detach them to any instance at any time (though you can only ever have one volume attached to one instance at a time).

Hope that clears things up.

Let me know if you have any other questions.

Thanks for sharing this information, it’s interesting to see that you took similar decisions to a team I’m working on at the moment.

I’m curious to know how you’re persisting the data in your MySQL cluster. As I understand it, EC2 instances don’t have non-volatile storage so you have to use one of Amazon’s data storage services as a back end. Are you using Elastic Block Storage to store the data from your MySQL cluster, and how does that work exactly?

Thanks

Ben

You could teach Web Oriented Architectures in University!
Your answers where very clear and convincing.

Do you have some resources (web) or books to suggest about solid web architecture strategy?
Ok last question (alternatives) : if someone would decided to choose Python, would you also choose Google Apps engine?

Your service will be a hit, and we know that well scale forever

Cheers

Thanks ever so much for your comment and your feedback. Here are the answers to your questions:

1. We store our client and product data & meta data using MySQL, however we store our raw assets and generated branding on S3. We use both because storing binary data in a database is a bad idea, it can get corrupted easily when restoring / backing up. S3 is lightning fast for serving binary content and infinitely scalable – we never need to worry about runing out of disk space!
2. Rails is great for web sites. It is not so great for background processes or process intensive heavy lifting (one of the reasons why twitter migrated to Scala from Rails). It was a toss up between Python with Django and Java with Struts 2. We chose Java (there was no learning curve for us).
3. You’re correct! AWS does provide the load balancer, we haven’t implemented our own. Sorry if that wasn’t clear.

Hope that clears some questions up, feel free to ask any more!

first of all I wish a huge success because your ideas are brilliant and your service looks disruptive.

I have few and very dumb questions on the architecure (I’m not an experienced engineer and I’m going to create a my first startup) :

1)Why do you need two different storages : Amazon S3 and MYSQL ?

2)Why did you discarded Ruby on Rails in favor of Java ?

3) Load balancing : why do you need to implement your own load balancer , is not Amazon EC2 providing that ?

I know this are really rookie questions, but you inveted to ask , I can’t miss such opportunity

Looking forward to use your service.

And thank you in case you will aswer my questions .

Cheers
Dave

I sat next to you at the Business Link “Start Up” course a month or so back. I have two months in which to develop my business proposition. I’m really keen to use your platform. I was hoping to be part of your closed loop beta testing group.

I’m developing my ideas around a design consultancy for ecological and sustainable land management. In October, I leave for a study tour in central America.

I look forward to hearing from you.
Cheers
Kevin Mascarenhas