Monthly Archives: August 2009

Cloud Tips: Backing Up MySQL on Amazon EC2 to S3

Now that I’m all set up in the Amazon cloud I’m starting to think about backups. Elastic Block Storage (EBS) on Amazon is great and the Snapshots (backups) can be generated with a few clicks from the Management Console, but, for a few reasons I’d like to set up my own backup scripts and here’s why:

  • Amazon EBS snapshots are cool, but there might be a situation where I’d like to restore only one file, or a few rows from a MySQL dump or whatever. EBS works in bundled mode, this means that you store an image of the hard-drive you’re working it, no matter how many files there are. It might be painful setting up an extra hard-drive just for backups and work with its snapshots
  • I don’t believe there’s a feature to schedule or automate EBS snapshots
  • I’d like a simple way to download backed up date onto my local PC
  • Some of my clients like to get their weekly backups by FTP

I don’t really care about my php files, images and other stuff that’s located on my EBS, cause I’m sure I have local copies of all that. The most important part in all my projects is the data stored in my MySQL database, thus I’m going to show you how to setup a simple shell script to generate daily MySQL backups and a simple php script to upload them to a secure S3 bucket.

Take a look at this simple shell script:

filename=mysql.`date +%d.%m.%Y`.sql.gz
echo Generating MySQL Dump: ${filename}
mysqldump -uyour_username -pyour_password --all-databases | gzip -c9 > /ebs/backups/${filename}
echo Uploading ${filename} to S3 bucket
php /ebs/data/s3-php/upload.php ${filename}
echo Removing local ${filename}
rm -f /ebs/backups/${filename}
echo Complete

I’m assuming you’re familiar with shell scripting, thus there’s no need to explain the first few lines. Don’t forget to type in your own username and password for MySQL access, also, I used the path /ebs/backups/ for my daily MySQL backups. You choose your own.

There are a few scripts located in /ebs/data/s3-php/ including upload.php, which takes a single parameter – filename (don’t put your path there, let’s keep things simple). The script simply reads the given file and uploads it to a preset path into your preset S3 bucket. I’m working with the S3-php5-curl class by Donovan Schonknecht. It uses the Amazon S3 REST API to upload files and it’s just one php file called S3.php, which in my case is located in /ebs/data/s3-php right next to my upload.php script.

Before going on to upload.php take a look at this file which I called S3auth.php:

$access_id = 'your_access_id';
$secret_key = 'your_secret_key';
$bucket_name = 'bucket';
$local_dir = '/ebs/backups/';
$remote_dir = 'backups/';

This is the settings file which I use in upload.php. These particular settings assume your backups will be located in /ebs/backups and will be backed up to an Amazon S3 bucket called ‘bucket’ and in the ‘backups’ directory within that bucket. Using single quotes is quite important, especially with the secret_key, as Amazon secret keys often include the backslash symbol. Here’s the upload.php script:

require("S3.php");
require("S3auth.php");

$s3 = new S3($access_id, $secret_key);
$s3->putBucket($bucket_name, S3::ACL_PRIVATE);
$s3->putObjectFile($local_dir.$argv[1], $bucket_name, $remote_dir.$argv[1], S3::ACL_PRIVATE);

All simple here according to the S3.php class documentation. Notice the $argv[1], that’s the first argument passed to the upload.php script, thus the filename of the backup file.

That’s about everything. Try a few test runs with the shell script (remember to chmod +x it otherwise you’ll not be able to execute it) and finally setup a cron running the script daily. Mine works like a charm!



Foller.me: Almost Ready to Remove the Beta Label

With a few core changes to Foller.me (hope you noticed the performance tuning) we’re almost ready to remove the beta label. I wrote on Twitter a few weeks ago that the beta label will be removed in September, so here we are. I’m not sure if it means anything to you, but it does mean a lot to me. You see, a product in beta is used with caution most of the time, that’s what I do. I like to play with betas but I don’t actually use them (except for Yahoo! Pipes and others of course). I think there’s something similar between the words “beta” and “startup”, so yes, I’d like to show the world that we’re stable. No glitches, no more server reboots and ready for massive hi-destructive traffic (thank you Amazon EC2!).

A few words about the core changes. We improved our caching mechanism – the relations are now being cached for no longer than one day. They’re used when generating the followers map, while the topics, mentions and hashtags clouds cache for only an hour. Same caching now applies to the Foller.me API, besides, client-side caching is always welcome. This has speeded up our overall performance and it seems to be running smooth for the past few weeks.

The Foller.me API though is still being improved. We’re coming up with a few more feature improvements to the REST API, most of them will involve styling the clouds and we’re also working on a few more examples to show you how to get the most out of the API goodies.

The beta label removal date hasn’t been announced yet, but we’re thinking September (you know, in Russia kids go to school 1st of September each year, so we’re hoping that their parents buy them brand new PCs and of course a broadband internet connection, therefore kids will discover Twitter and hopefully Foller.me ;).

Thank you for all your support and good luck!

Update: Foller.me is still beta ;) just not yet .. (12/22/09)



Twitter Friendly Links: Now In Alphanumeric Mode

Hello my friends! Yes, I’m finally back from my two-week vacation. It was awesome, I spent a week in Magnitogorsk and another one in Sochi. The weather was great, it only rained on Sunday, when I was on my way back. Everything went fine, I’ve seen all my friends, we had a great time. Sochi is okay, getting ready for year 2014, though their airport in Adler still sucks big time!

Anyways, back to Moscow and back to work, and today I’d like to introduce a new feature in the Twitter Friendly Links plugin for WordPress. I call it the Alphanumeric mode, or format, or whatever. You see, people are used to TinyURL and bit.ly, where links are hashed and alphanumeric most of the time. I used a simple base 32 encoding algorithm in my case, plus added a few thousand, so that the links wouldn’t be shorter than three symbols. Not very nice to see example.com/a I guess.. Right? The Generic mode is still there and switched on by default for all the retro guys like me ;) Yes, I do prefer numbers and I know bit.ly would if they hadn’t hosted thousands of new links every day.

Also, I removed the rev=canonical option cause it’s pretty much outdated and encourage you to use the rel=shortlink HTML and HTTP headers, and rel=canonical too, as encouraged by wordpress.org. By the way, it’s pretty cool to see wordpress.com blogs get shortened links by wp.me as announced here: WP.me – shorten your links and they too support shortlink relations (both HTML and HTTP), which was why I decided to stick to them. Right, as Sam Johnston mentioned in the comments, that made over 7 million blogs accept the shortlink relations. Twitter clients should now be thinking about plugging that into the clients. It would make linking much easier, although may be a little more traffic consuming.

Or what if the guys from Twitter would.. Arrgh, hell no =) Impossible…

P.S. Hope you switched to WordPress 2.8.4 in time before the massive password reset attacks ;) I’m kidding, there were no massive attacks.. I guess =)

I’m so glad to be back. Have a great day everyone!



My First Vacation. Seriously ;)

Okay folks, after years of hard work around the clock, I’ve finally got some time off, so yes, this is my first vacation ever. It all started last Saturday when I took off to Magnitogorsk to see my relatives. I’m actually having a great time here and the weather is pretty cool, but I’m leaving to Sochi this Saturday for one more week and then back to Moscow and of course, back to work ;)

Hope you all miss me and hope to bring you some sunny photos in a few weeks :)



New WordPress Plugin: The Foller.me Widget

Here’s a way for you to show what you’ve been tweeting about lately. Right on your blog, in the sidebar. No Twitter passwords required, cause this thing works via the Foller.me API. This is the first ever app (widget, gadget, or whatever you wanna call that) based on our Foller.me API, so I hope we have a great start here, and you young developers out there, dig through this widget’s source code (which is GPL btw) and play around with the API. Perhaps your app’s the next, and please DO write to us if you do, cause you don’t wanna miss all the credit we offer ;)

The plugin’s located here: Foller.me Widget and hosted at the WordPress.org plugin directory right over here. Thank you for all your votes ;)



And Yet Another WordPress Theme for Inspiration ;)

follerme_new_blog

My brand new creation: The Foller.me Theme for WordPress and I ain’t releasing it public ;)



Multiblog WordPress: Eeek! No Database Connection

I tried browsing to www.kovshenin.com this morning and was pretty sure I’ll get the usual redirect, but no. I got an “Error establishing a database connection”. Right, it seems that www.kovshenin.com is not defined in the wp-config.php file we created earlier this week (Multiple Sites Driven By One WordPress Installation Part II) so here’s a workaround (and we’re switching back to example.org and example-two.org):

$wp_multi = array(
    "example.org" => array(
        "DB_NAME" => "example",
        "DB_USER" => "example-user",
        "DB_PASSWORD" => "example-pass",
        "DB_HOST" => "example-host"
    ),

    "example-two.org" => array(
        "DB_NAME" => "example-two",
        "DB_USER" => "example-two-user",
        "DB_PASSWORD" => "example-two-pass",
        "DB_HOST" => "example-two-host"
    )
);
$server_name = str_replace("www." , "", strtolower($_SERVER["SERVER_NAME"]));
$wp_settings = $wp_multi[$server_name];

There. And we also got rid of an error we’d get if we typed eXamPle.org in the address bar.

I’m thinking of a way to wrap this up in some plugin or a little hack, or perhaps a super-short step-by-step tutorial, so that setting up multiple websites one a single wordpress bundle would be easier than ever. If you have any suggestions feel free to speak ;)



Powered by Delicious: Web Design, Coding & HTML5

This is my first “Powered by Delicious” post, but I plan on writing similar posts at least once a month or so, depending how many links I bookmark. Here’s a list of some of my latest delicious bookmarks, with goodies on web design, coding, html5, photoshop, javascript and a bunch of other stuff.



Multiple Sites Driven By One WordPress Installation Part II: Please Be Careful, You Really Might Mess Up ;)

This is just a little update on what I’ve been doing and what I have acheived so far. So, what we have: two different hosts – example.org and example-two.org both on one webserver right next to eachother, let’s say in the /home/username/www (we’ll just call it www further on) directory. The apache vhosts are configured correctly, the document roots are set. Everything’s working fine. There are two wordpress installations, one in /www/example.org and the second one in /www/example-two.org, which I got rid of. Here’s how.

So, first, create a new directory for the installation somewhere unreachable. I created mine in /www/wordpress_public/ and copied all the files and folders from /www/example.org to /www/wordpress_public. Remember to copy, not move, cause you might mess up. Always keep your backup right by your side in case you destroy everything ;) Also, do not copy the .htaccess file, it has to say in your example.org root folder, while the files in wordpress_public should be accessible by everyone.

Next, you have to create a symbolic link in the example.org directory that links to ../wordpress_public (I called mine “core”, I think it’s fuzzy ;) On a linux system, being in the example.org directory it goes like this:

ln -s ../wordpress_public ./core

So now, /www/example.org/core is pretty much the same as /www/wordpress_public, get it? Note, that if you are running suPHP you’ll get a “is not in document root of Vhost” error when trying to access example.org/core. To get rid of that error, edit the suPHP config file (probably located in /etc/suphp.conf), find the check_vhost_docroot parameter and set it to false. Restart the httpd daemon.

Open up index.php in /www/exaple.org and replace this line:

require('./wp-blog-header.php');

With something like this:

require('./core/wp-blog-header.php');

Before proceeding, delete your .htaccess file in /www/example.org. Then, Open up your usual wordpress admin panel on example.org (yes, it’s still accessible) probably on example.org/wp-admin, browse to your Settings – General screen, then change the WordPress address (URL) to http://example.org/core instead of http://example.org. No trailing slashes! Click save and bang! You’re logged out. Now browse to http://example.org/core/wp-admin and you should be able to login using your old credentials. Browse to your Permalinks settings and refresh them (that should create a new .htaccess file). Done! Almost..

Now, if you’re going to have more than one website (which is actually the point I’m writing all this stuff, duh!) then you should keep a little order in your wp-content directory. I’m talking about the uploads, photos & other post attachments. The generic WordPress attachments are quite easy to change in the settings. Switch them to wp-content/uploads/example.org, next using an FTP client or SSH move all the files and folders in /www/wordpress_public/wp-content/uploads to /www/wordpress_publoic/wp-content/uploads/example.org. That should do the trick. To run a test try creating a new post and uploading a photo or whatever, see where it places it.

If there were already posts on your blog with images attached, then their paths will not change automatically to your new uploads folder, you’d have to do that manually if you didn’t know some basic SQL ;) Run this in phpMyAdmin or whatever:

UPDATE wp_posts w SET `post_content` =
    REPLACE(`post_content`, 'http://example.org/wp-content/uploads',
    'http://example.org/core/wp-content/uploads/example.org')
    WHERE `post_content` LIKE '%http://example.org/wp-content/uploads%';

The WHERE condition is just to make sure other posts aren’t messed up.. Make sure everything’s working and proceed.

Now we’d like the example-two.org website to use the same WordPress installation located in /www/wordpress_public. First, copy themes and plugins from the example-two.org directory to the wordpress_public folder (so that wordpress_public will contain all the plugins and themes that both websites use). In SSH you’d type something like this:

cp -R www/example.org/wp-content/plugins/* www/wordpress_public/wp-content/plugins/
cp -R www/example.org/wp-content/themes/* www/wordpress_public/wp-content/themes/

Then, you have to somehow distinguish in wordpress_public which website we’re gonna show, right? I wrote this little hack you could use for wp-config.php (the one located in /www/wordpress_public) which could handle as many websites as you like:

$wp_multi = array(
	"example.org" => array(
		"DB_NAME" => "example_database",
		"DB_USER" => "examlpe_username",
		"DB_PASSWORD" => "example_password",
		"DB_HOST" => "example_host"
	),

	"example-two.org" => array(
		"DB_NAME" => "example2_database",
		"DB_USER" => "example2_username",
		"DB_PASSWORD" => "example2_password",
		"DB_HOST" => "example2_host"
	)
);
$server_name = $_SERVER["SERVER_NAME"];
$wp_settings = $wp_multi[$server_name];

// ** MySQL settings ** //
define('DB_NAME', $wp_settings["DB_NAME"]);
define('DB_USER', $wp_settings["DB_USER"]);
define('DB_PASSWORD', $wp_settings["DB_PASSWORD"]);
define('DB_HOST', $wp_settings["DB_HOST"]);

I hope you get my point and don’t delete the rest of the wp-config.php file which I didn’t list here ;) Oh and you do have to change example.org and example-two.org, unless you own those domains … Now, save the file and check back on example.org, see if it still works (yes, it should ..)

Back to example-two.org. Create the symbolic link (core) in /www/example-two.org pointing to the wordpress_public folder, just like we did for example.org, remember? Make sure it’s accessible. Browse to your old example-two.org admin panel, switch the WordPress address (URL) to http://example-two.org/core (no trailing slash!), remove your .htaccess file from /www/example-two.org and edit index.php (it should read the same as the index.php of the example.org website). Browse to your new admin panel (http://example-two.org/core/wp-admin) and refresh your permalinks settings. Browse to example-two.org and see if it works. If it does, then you did everything right, and remember to set your upload directory to wp-content/uploads/example-two.org and don’t forget to copy all your previous uploads to the new place.

Now when everything’s working fine, you can go ahead and remove all the files and folders that are somehow related to WordPress from your /www/example.org and /www/example-two.org directories (except from, of course, .htaccess).

Congrats! You now have two websites, driven by one single WordPress installation!

Here’s a list of rules to follow when using this method:

  • You update your WordPress core only once on any of the two (or more) hosts. If there’s a database structure change in the update, then each website will ask about updating its database seperately (when in the admin panel)
  • Same with the plugins. Update only once, and if you do encounter any issues after a certain plugin update, deactive and activate it on each and every site
  • If you’re using the Next Gen Gallery plugin, change the settings to store your gallery in /wp-content/gallery/example.org and /wp-content/gallery/example-two.org, for the two websites, not in one single folder, otherwiste you’ll have a complete mess when you reach 5 or 10 websites
  • I’ve no idea what will happen with cache plugins, still trying to figure out what WP-Super Cache is up to and it’s not my fault if you mess everything up, alright? ;)
  • All this might be pretty dangerous, so you shouldn’t experiment on super popular blogs unless you know what you’re doing

Guess that’s it. I’ll keep running the tests and stuff. I’m really worried about the cache plugin, so I’ll get back at you with that later this week. Also, for your information, kovshenin.com and blog.foller.me are now running this method, so if you notice anything strange, please let me know, okay?