Create an expect script:
/path/to/sshmysql.exp

#!/usr/bin/expect -f
#script by darren cassar
#mysqlpreacher.com

set machine [lindex $argv 0]

set timeout -1

spawn ssh username@$machine
match_max 100000
expect -exact “assword: ”
send — “password\r”
send — “sudo -k; sudo su – mysql\r”
expect -exact “sudo -k; sudo su – mysql”
expect -exact “assword:”
send — “password\r”
interact

# you should change the word password in ’send — “password\r”‘ to your login password
# if you have the same password for each environment you could also script logging into mysql directly from the same expect script BUT that is not recommended.

Create a bash script:
/path/to/login.sh

#!/bin/bash
#script by darren cassar
#mysqlpreacher.com

sm=’/path/to/sshmysql.exp’

menu() {
echo ” 101 – dev.databaseserver1 ”
echo ” 102 – dev.databaseserver2 ”
echo ” 103 – dev.databaseserver3 ”
echo ” 201 – qa.databaseserver1 ”
echo ” 301 – uat.databaseserver1 ”
echo ” 302 – uat.databaseserver2 ”
echo ” 401 – prod.databaseserver1 ”
echo ” ”
}

ARGUMENT=notmenu

if [ -z "$1" ]
then
ARGUMENT=menu
else
choice=$1
fi

if [ $ARGUMENT = "menu" ]
then
menu
else
case “$choice” in
101|dev.databaseserver1 ) $sm dev.databaseserver1;;
102|dev.databaseserver2 ) $sm dev.databaseserver2;;
103|dev.databaseserver3 ) $sm dev.databaseserver3;;
201|qa.databaseserver1 ) $sm qa.databaseserver1;;
301|uat.databaseserver1 ) $sm uat.databaseserver1;;
302|uat.databaseserver2 ) $sm uat.databaseserver2;;
401|prod.databaseserver1 ) $sm prod.databaseserver1;;
* ) echo “Wrong value passed to script”
menu ;;
esac
fi

alias l=’/path/to/login.sh’

Output:

[darrencassar@mymachine ~ ]$ l
101 – dev.databaseserver1
102 – dev.databaseserver2
103 – dev.databaseserver3
201 – qa.databaseserver1
301 – uat.databaseserver1
302 – uat.databaseserver2
401 – prod.databaseserver1

Output:
The below command would log you into the first development database server as mysql user.

[darrencassar@mymachine ~ ]$ l 101

On each machine place aliases for each instance in the .profile

alias use3306=’mysql -u root -p -h 127.0.0.1 -P 3306 –prompt=”mysql \D> “‘

The above setup can be used using any client/server OS: Linux, Solaris, MAC OS or Windows(running Cygwin)

NOTE: If you store the password in clear text inside the expect script, you should at least save the scripts inside an encrypted partition on your machine and make sure that folder is not shared or accessible by anyone. Another way of doing it would be to use either SSHKeys OR save the password inside a file and encrypt it using OpenSSL

Enjoy!


mysql> explain SELECT COUNT(ac.UID) FROM ACTIVE ac JOIN ALL a;
+----+-------------+-------+------+---------------+------+---------+------+---------+-------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+---------------+------+---------+------+---------+-------+
| 1 | SIMPLE | ac | ALL | NULL | NULL | NULL | NULL | 124426 | |
| 1 | SIMPLE | a | ALL | NULL | NULL | NULL | NULL | 7594256 | |
+----+-------------+-------+------+---------------+------+---------+------+---------+-------+
2 rows in set (0.01 sec)

mysql> SELECT COUNT(ac.UID) FROM ACTIVE ac JOIN ALL a ON ac.UID=a.UID;
+---------------+
| COUNT(ac.UID) |
+---------------+
| 17466 |
+---------------+
1 row in set (0.23 sec)

mysql> set @tot = (SELECT COUNT(ac.UID) FROM ACTIVE ac JOIN ALL a ON ac.UID=a.UID);

^CQuery aborted by Ctrl+C


Took more than 60seconds —- what the …..
Why did it take a long?


mysql> set @a=2;
Query OK, 0 rows affected (0.00 sec)

mysql> select @a;
+------+
| @a |
+------+
| 2 |
+------+
1 row in set (0.00 sec)

mysql> set @tot = (SELECT COUNT(*) FROM ACTIVE);
Query OK, 0 rows affected (0.13 sec)

mysql> select @tot;
+--------+
| @tot |
+--------+
| 124426 |
+--------+
1 row in set (0.00 sec)

mysql> set @tot = (SELECT COUNT(ac.UID) FROM ACTIVE ac);
Query OK, 0 rows affected (0.22 sec)

mysql> select @tot;
+--------+
| @tot |
+--------+
| 124426 |
+--------+
1 row in set (0.00 sec)

mysql> SELECT COUNT(ac.UID) FROM ACTIVE ac, ALL a;
+---------------+
| COUNT(ac.UID) |
+---------------+
| 944922897056 |
+---------------+
1 row in set (0.05 sec)

mysql> set @tot=(SELECT COUNT(ac.UID) FROM ACTIVE ac JOIN ALL a);
^CQuery aborted by Ctrl+C
ERROR 1317 (70100): Query execution was interrupted


Reason? …. the query is using the MySQL optimiser rather than the IB one! Why? good question (will have to ask IB devs though).

Work around, use ac temporary table to store the result and setting the variable to the result field, but it’s really ugly isn’t it?

Feel free to send me links you think might be good to add in order to help others.

Remember, SHARING IS CARING!!! …. we get so much for free, why shouldn’t we give some back?

Cheers,
Darren

The tool is downloadable from there and anyone can use it for free in accordance to GPLv2.

I wanted to throw out tutorial about how to install it and use it (Note this tutorial is for version securich version 0.1.2):

Steps:
1. Download it,
2. Install it,
3. Create a role named 'role1' having privileges: select insert update
4. Check roles,
5. Check role privileges,
6. Create a first user john@machine.domain.com (granting privileges on a whole database employees apart from one table),
7. Create a second user paul@10.0.0.2 (granting privileges on all tables in world having word Country in them),
8. Create a third user peter@localhost (granting privileges on the database test),
9. Check user privileges for (paul),
10. Update role created above and see changes (add delete to role 1),
11. Update password (for paul) and see changes,
12. Clone user paul to judas,
13. Check user privileges
14. Check user,
15. Rename user judas to james,
16. Revoke privileges from third user disconnecting any existing connections from that user (useful if a security breach is suspected or if you are a security paranoid thus wanting to make sure the person you are blocking out won't have any more access as from that point onwards).

1. Go to www.securich.com downloads page and download the install script
2. Untar the install script and run it using ./securich_install.sh and it'll install everything automatically

dcassar@ubuntu:~/Desktop$ ./securich_install.sh
Enter version number: 0.1.1
Which kind of installation would you like to do?
1. Install from file on disk
2. Download and install (recommended)
Enter choice (default 2):

Installation starting
–2009-06-19 16:27:56– http://www.securich.com/downloads/securich.0.1.1.tar.gz
Resolving www.securich.com… 64.202.163.10
Connecting to www.securich.com|64.202.163.10|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 29217 (29K) [application/x-tar]
Saving to: `securich.0.1.1.tar.gz'

100%[=====================================================================================================>] 29,217 64.7K/s in 0.4s

2009-06-19 16:27:59 (64.7 KB/s) – 'securich.0.1.1.tar.gz' saved [29217/29217]

Enter mysql root Password (default ):
Enter mysql Hostname/IP (default 127.0.0.1): localhost
Enter mysql Port (default 3306): 3306
Installation complete

3. #log into mysql
use securich;
call create_update_role('role1','select');
call create_update_role('role1','insert');
call create_update_role('role1','update');
4. call check_roles();
5. call check_role_privileges('role1');
6. call grant_privileges('john' , 'machine.domain.com' , 'employees' , '' , 'alltables' , 'role1' , 'john@domain.com');
call revoke_privileges('john' , 'machine.domain.com' , 'employees' , 'salaries' , 'table' , 'role1' , 'N');
7. call grant_privileges('paul' , '10.0.0.2' , 'world' , '^Country' , 'regexp' , 'role1' , 'paul@domain.com');
8. call grant_privileges('peter' , 'localhost' , 'test' , '' , 'all' , 'role1' , 'peter@domain.com');
9. call check_full_user_entries('paul');
10. call create_update_role('role1','delete');
call check_full_user_entries('paul');
11. call set_password('paul' , '10.0.0.2' , 'password123');
12. call clone_user('paul' , '10.0.0.2' , 'judas' , '10.0.0.2' , 'judas@domain.com');
13. call check_full_user_entries('judas');
14. call check_user_privileges('judas' , '10.0.0.2' , 'world' , 'role1');
15. call rename_user('judas' , 'james' , 'james@domain.com');
16. call create_update_role('role2','execute');
17. call grant_privileges('peter' , 'localhost' , 'securich' , 'my_privileges' , 'storedprocedure' , 'role2' , 'peter@domain.com');

18. #connect to mysql using thirduser peter in another session
show databases;
use securich;
show tables;
call my_privileges('test');
show processlist;

19. call revoke_privileges('peter' , 'localhost' , 'test' , '' , '' , 'role1' , 'Y');

20. #as user peter again from 2nd open instance run
show processlist;

dcassar@ubuntu:~/Desktop$ ./securich_install.sh
Enter version number: 0.1.1
Which kind of installation would you like to do?
1. Install from file on disk
2. Download and install (recommended)
Enter choice (default 2):

Installation starting
–2009-06-19 16:27:56– http://www.securich.com/downloads/securich.0.1.1.tar.gz
Resolving www.securich.com… 64.202.163.10
Connecting to www.securich.com|64.202.163.10|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 29217 (29K) [application/x-tar]
Saving to: `securich.0.1.1.tar.gz'

100%[=====================================================================================================>] 29,217 64.7K/s in 0.4s

2009-06-19 16:27:59 (64.7 KB/s) – `securich.0.1.1.tar.gz' saved [29217/29217]

Enter mysql root Password (default ):
Enter mysql Hostname/IP (default 127.0.0.1): localhost
Enter mysql Port (default 3306): 3306
Installation complete
dcassar@ubuntu:~/Desktop$ mysql -u root -p -h 127.0.0.1 -P 3306 Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 429
Server version: 5.1.33 MySQL Community Server (GPL)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> use securich;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> call create_update_role('role1','select');
Query OK, 0 rows affected, 5 warnings (0.03 sec)

mysql> call create_update_role('role1','insert');
Query OK, 0 rows affected (0.04 sec)

mysql> call create_update_role('role1','update');
Query OK, 0 rows affected (0.04 sec)

mysql> call check_roles();
+—-+——-+
| ID | ROLE |
+—-+——-+
| 1 | read |
| 2 | write |
| 3 | role1 |
+—-+——-+
3 rows in set (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

mysql> call check_role_privileges('role1');
+———–+
| PRIVILEGE |
+———–+
| INSERT |
| SELECT |
| UPDATE |
+———–+
3 rows in set (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

mysql> call grant_privileges('john' , 'machine.domain.com' , 'employees' , '' , 'alltables' , 'role1' , 'john@domain.com');
+——————————————————————————————–+
| USER_PASSWORD |
+——————————————————————————————–+
| Password for user — john — contactable at — john@domain.com — is — bfcbd8234d9eb44 — |
+——————————————————————————————–+
1 row in set (0.16 sec)

Query OK, 0 rows affected, 1 warning (0.16 sec)

mysql> call revoke_privileges('john' , 'machine.domain.com' , 'employees' , 'salaries' , 'table' , 'role1' , 'N');
Query OK, 0 rows affected (0.09 sec)

mysql> call grant_privileges('paul' , '10.0.0.2' , 'world' , '^Country' , 'regexp' , 'role1' , 'paul@domain.com');
+——————————————————————————————–+
| USER_PASSWORD |
+——————————————————————————————–+
| Password for user — paul — contactable at — paul@domain.com — is — bc4ab08785e1be6 — |
+——————————————————————————————–+
1 row in set (0.06 sec)

Query OK, 0 rows affected, 1 warning (0.06 sec)

mysql> call grant_privileges('peter' , 'localhost' , 'test' , '' , 'all' , 'role1' , 'peter@domain.com');
+——————————————————————————————-+
| USER_PASSWORD |
+——————————————————————————————-+
| Password for user — peter — contactable at — peter@domain.com — is — 7b3b4746d04b — |
+——————————————————————————————-+
1 row in set (0.04 sec)

Query OK, 0 rows affected (0.04 sec)

mysql> call check_full_user_entries('paul');
+———-+———-+————–+—————–+——-+———–+——-+
| USERNAME | HOSTNAME | DATABASENAME | TABLENAME | ROLE | PRIVILEGE | STATE |
+———-+———-+————–+—————–+——-+———–+——-+
| paul | 10.0.0.2 | world | Country | role1 | INSERT | A |
| paul | 10.0.0.2 | world | Country | role1 | SELECT | A |
| paul | 10.0.0.2 | world | Country | role1 | UPDATE | A |
| paul | 10.0.0.2 | world | CountryLanguage | role1 | INSERT | A |
| paul | 10.0.0.2 | world | CountryLanguage | role1 | SELECT | A |
| paul | 10.0.0.2 | world | CountryLanguage | role1 | UPDATE | A |
+———-+———-+————–+—————–+——-+———–+——-+
6 rows in set (0.01 sec)

Query OK, 0 rows affected, 4 warnings (0.01 sec)

mysql> call create_update_role('role1','delete');
Query OK, 0 rows affected (0.09 sec)

mysql> call check_full_user_entries('paul');
+———-+———-+————–+—————–+——-+———–+——-+
| USERNAME | HOSTNAME | DATABASENAME | TABLENAME | ROLE | PRIVILEGE | STATE |
+———-+———-+————–+—————–+——-+———–+——-+
| paul | 10.0.0.2 | world | Country | role1 | DELETE | A |
| paul | 10.0.0.2 | world | Country | role1 | INSERT | A |
| paul | 10.0.0.2 | world | Country | role1 | SELECT | A |
| paul | 10.0.0.2 | world | Country | role1 | UPDATE | A |
| paul | 10.0.0.2 | world | CountryLanguage | role1 | DELETE | A |
| paul | 10.0.0.2 | world | CountryLanguage | role1 | INSERT | A |
| paul | 10.0.0.2 | world | CountryLanguage | role1 | SELECT | A |
| paul | 10.0.0.2 | world | CountryLanguage | role1 | UPDATE | A |
+———-+———-+————–+—————–+——-+———–+——-+
8 rows in set (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

mysql> call set_password('paul' , '10.0.0.2' , 'password123');
Query OK, 1 row affected (0.02 sec)

mysql> call clone_user('paul' , '10.0.0.2' , 'judas' , '10.0.0.2' , 'judas@domain.com');
+———————————————————————————————-+
| USER_PASSWORD |
+———————————————————————————————-+
| Password for user — judas — contactable at — judas@domain.com — is — 70d5b79d80fab04 — |
+———————————————————————————————-+
1 row in set (0.01 sec)

Query OK, 0 rows affected, 1 warning (0.10 sec)

mysql> call check_full_user_entries('judas');
+———-+———-+————–+—————–+——-+———–+——-+
| USERNAME | HOSTNAME | DATABASENAME | TABLENAME | ROLE | PRIVILEGE | STATE |
+———-+———-+————–+—————–+——-+———–+——-+
| judas | 10.0.0.2 | world | Country | role1 | DELETE | A |
| judas | 10.0.0.2 | world | Country | role1 | INSERT | A |
| judas | 10.0.0.2 | world | Country | role1 | SELECT | A |
| judas | 10.0.0.2 | world | Country | role1 | UPDATE | A |
| judas | 10.0.0.2 | world | CountryLanguage | role1 | DELETE | A |
| judas | 10.0.0.2 | world | CountryLanguage | role1 | INSERT | A |
| judas | 10.0.0.2 | world | CountryLanguage | role1 | SELECT | A |
| judas | 10.0.0.2 | world | CountryLanguage | role1 | UPDATE | A |
+———-+———-+————–+—————–+——-+———–+——-+
8 rows in set (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

mysql> call check_user_privileges('judas' , '10.0.0.2' , 'world' , 'role1');
+———–+
| PRIVILEGE |
+———–+
| DELETE |
| INSERT |
| SELECT |
| UPDATE |
+———–+
4 rows in set (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

mysql> call rename_user('judas' , 'james' , 'james@domain.com');
+———————————————————————————————-+
| USER_PASSWORD |
+———————————————————————————————-+
| Password for user — james — contactable at — james@domain.com — is — 85c2fc100d83884 — |
+———————————————————————————————-+
1 row in set (0.02 sec)

Query OK, 0 rows affected, 1 warning (0.11 sec)

mysql> call create_update_role('role2','execute');
Query OK, 0 rows affected (0.09 sec)

mysql> call grant_privileges('peter' , 'localhost' , 'securich' , 'my_privileges' , 'storedprocedure' , 'role2' , 'peter@domain.com');
Query OK, 0 rows affected (0.08 sec)

mysql> call revoke_privileges('peter' , 'localhost' , 'test' , '' , '' , 'role1' , 'Y');
Query OK, 0 rows affected (0.15 sec)

mysql>

dcassar@ubuntu:~/Desktop$ mysql -u peter -p7b3b4746d04b -h 127.0.0.1 -P 3306
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 437
Server version: 5.1.33 MySQL Community Server (GPL)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> show processlist;
+—–+——-+—————–+——+———+——+——-+——————+
| Id | User | Host | db | Command | Time | State | Info |
+—–+——-+—————–+——+———+——+——-+——————+
| 437 | peter | localhost:49022 | NULL | Query | 0 | NULL | show processlist |
+—–+——-+—————–+——+———+——+——-+——————+
1 row in set (0.00 sec)

mysql> show databases;
+——————–+
| Database |
+——————–+
| information_schema |
| securich |
| test |
+——————–+
3 rows in set (0.00 sec)

mysql> use securich;
Database changed
mysql> show tables;
Empty set (0.00 sec)

**** Note that the only privileges peter has on securich is on the stored procedure 'my_privileges' and definitely no tables

mysql> call my_privileges('test');
+———–+
| PRIVILEGE |
+———–+
| DELETE |
| INSERT |
| SELECT |
| UPDATE |
+———–+
4 rows in set (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

mysql> show processlist;
+—–+——-+—————–+——+———+——+——-+——————+
| Id | User | Host | db | Command | Time | State | Info |
+—–+——-+—————–+——+———+——+——-+——————+
| 437 | peter | localhost:49022 | NULL | Query | 0 | NULL | show processlist |
+—–+——-+—————–+——+———+——+——-+——————+
1 row in set (0.00 sec)

****** – In the meantime the dba revoked rights with terminate live connections from peter@localhost

mysql> show processlist;
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect…
ERROR 1045 (28000): Access denied for user 'peter'@'localhost' (using password: YES)
ERROR:
Can't connect to the server

mysql>

I truly hope you enjoyed this run through. I excuse myself it's a tad too long, but I wished to illustrate some of the cool features of this package.

Cheers,
Darren

PS don't forget to check out Securich HERE

The name is SECURICH and downloads as well as documentation are available at http://www.securich.com/. The bug list is available at http://www.securich.com/mantis/ (you’ll have to register and log in first). Privileges can be granted on a database or table level with a few combinations like:
1. database as one,
2. all tables in database (useful when you need to grant a bunch of privileges to all tables but a few thus revoking the few later),
3. single tables,
4. stored procedure or
5. tables through the use of regular expression.

The user has the facility to create a role and update it on the fly thus propagating the changes onto already existing users having the role in question. There are also loads of other functionalities available and I urge you to check the documentation on http://www.securich.com/.

A cool feature I like is password history which enables dbas to enforce changing of passwords after a certain period of time.

#################################################################################################################################################
## PLEASE NOTE THAT:
## 1. THIS IS AN ALPHA VERSION,
## 2. YOU SHOULD NOT INSTALL THIS PACKAGE ON A PRE-EXISTING MYSQL WHICH ALREADY HAS PRIVILEGES SET UP.
#################################################################################################################################################
##
## This tool will modify current privileges on the “mysql” database tables and should only be used with freshly installed mysqls.
##
##################################################################################################################################################

Features like block user and unblock user as well as many other functionalities are being developed and more features will be added as time goes by.

I would greatly appreciate your feedback about what you think of the package, features you might wish to include etc (feature requests should be done through http://www.securich.com/mantis/ as well.)

Recently I needed to replicate between MySQL and another database technology. You might say, why on earth would you want to do something like that, but believe me there are reasons and definitely not (to go away from MySQL to some other DB technology like Oracle or SQL server). Unsurprisingly there are quite a few different tools to do it from any platform towards MySQL but very few which do it the other way round, just to name a couple: Golden Gate and DSCallards.

Whilst not going into their tools (you can find more information on their websites), HIT from DSCallards needs its software to run on Windows and Golden Gate is an expensive beast which was too much for what I needed, thus I decided to have a look at doing the job myself. Although it might look an overkill to do so, it took me a few hours to find a solution and implement it and a couple more to test it and here is a simple description.

MySQL replication can be SBR (statement based), RBR (row based) or a mixture of both. Now despite the fact that the mixture provides the best performance, it would be the most complicated in order to achieve a home made solution, and SBR in my opinion would have also been a bit of a headache to make sure queries didn’t contain non ansi sql through the use of functions like now(), sysdate() and many others. I therefore decided that RBR would be the option of choice.

Although I wonder how many of you ever read an RBR binlog using `mysqlbinlog mysql-bin.000004`, it would be something like:

#090526 14:09:13 server id 1  end_log_pos 1420  Query   thread_id=192   exec_time=0     error_code=0
SET TIMESTAMP=1243343353/*!*/;
BEGIN
/*!*/;
# at 1420
# at 1464
#090526 14:09:13 server id 1  end_log_pos 1464  Table_map: `test`.`t2` mapped to number 21
#090526 14:09:13 server id 1  end_log_pos 1498  Write_rows: table id 21 flags: STMT_END_F

BINLOG ‘
+ekbShMBAAAALAAAALgFAAAAABUAAAAAAAAABHRlc3QAAnQyAAID/gL+AQM=
+ekbShcBAAAAIgAAANoFAAAQABUAAAAAAAEAAv/+AQAAAA==
‘/*!*/;
# at 1498
#090526 14:09:13 server id 1  end_log_pos 1567  Query   thread_id=192   exec_time=0     error_code=0
SET TIMESTAMP=1243343353/*!*/;
COMMIT
/*!*/;
# at 1567
#090526 14:09:38 server id 1  end_log_pos 1635  Query   thread_id=192   exec_time=0     error_code=0
SET TIMESTAMP=1243343378/*!*/;
BEGIN
/*!*/;
# at 1635
# at 1679
#090526 14:09:38 server id 1  end_log_pos 1679  Table_map: `test`.`t2` mapped to number 21
#090526 14:09:38 server id 1  end_log_pos 1733  Update_rows: table id 21 flags: STMT_END_F

BINLOG ‘
EuobShMBAAAALAAAAI8GAAAAABUAAAAAAAAABHRlc3QAAnQyAAID/gL+AQM=
EuobShgBAAAANgAAAMUGAAAQABUAAAAAAAEAAv///AQAAAABZPwCAAAAAWT+AQAAAP4CAAAA
‘/*!*/;
# at 1733
#090526 14:09:38 server id 1  end_log_pos 1802  Query   thread_id=192   exec_time=0     error_code=0
SET TIMESTAMP=1243343378/*!*/;
COMMIT

Now that isn’t the most readable text you ever seen right? As Giuseppe said “This is more difficult to read than ancient Etruscan. If you are a DBA, you curse and look for help.” But the replication guys at mysql created a nice -v for us to add to mysqlbinlog thus issuing `mysqlbinlog -v mysql-bin.000004` would result in the following:

#090526 14:09:13 server id 1  end_log_pos 1420  Query   thread_id=192   exec_time=0     error_code=0
SET TIMESTAMP=1243343353/*!*/;
BEGIN
/*!*/;
# at 1420
# at 1464
#090526 14:09:13 server id 1  end_log_pos 1464  Table_map: `test`.`t2` mapped to number 21
#090526 14:09:13 server id 1  end_log_pos 1498  Write_rows: table id 21 flags: STMT_END_F

BINLOG ‘
+ekbShMBAAAALAAAALgFAAAAABUAAAAAAAAABHRlc3QAAnQyAAID/gL+AQM=
+ekbShcBAAAAIgAAANoFAAAQABUAAAAAAAEAAv/+AQAAAA==
‘/*!*/;
### INSERT INTO test.t2
### SET
###   @1=1
###   @2=NULL
# at 1498
#090526 14:09:13 server id 1  end_log_pos 1567  Query   thread_id=192   exec_time=0     error_code=0
SET TIMESTAMP=1243343353/*!*/;
COMMIT
/*!*/;
# at 1567
#090526 14:09:38 server id 1  end_log_pos 1635  Query   thread_id=192   exec_time=0     error_code=0
SET TIMESTAMP=1243343378/*!*/;
BEGIN
/*!*/;
# at 1635
# at 1679
#090526 14:09:38 server id 1  end_log_pos 1679  Table_map: `test`.`t2` mapped to number 21
#090526 14:09:38 server id 1  end_log_pos 1733  Update_rows: table id 21 flags: STMT_END_F

BINLOG ‘
EuobShMBAAAALAAAAI8GAAAAABUAAAAAAAAABHRlc3QAAnQyAAID/gL+AQM=
EuobShgBAAAANgAAAMUGAAAQABUAAAAAAAEAAv///AQAAAABZPwCAAAAAWT+AQAAAP4CAAAA
‘/*!*/;
### UPDATE test.t2
### WHERE
###   @1=4
###   @2=’d’
### SET
###   @1=2
###   @2=’d’
### UPDATE test.t2
### WHERE
###   @1=1
###   @2=NULL
### SET
###   @1=2
###   @2=NULL
# at 1733
#090526 14:09:38 server id 1  end_log_pos 1802  Query   thread_id=192   exec_time=0     error_code=0
SET TIMESTAMP=1243343378/*!*/;
COMMIT

The exact same output with some decently readable output. The problem at this point is that the output is not really something any other technology would undestand (not even feeding it to MySQL would work! ….

At this point I needed to do some compromises in order to reduce the complexity of this job, i.e. I will not be creating and altering tables / databases or indexes during runtime (this I can do without as I can do the same things on the slave manually when I need to do anything like that on the master) and the replication won’t be in real time i.e. the slave will be fed the sql periodically through a script. The last compromise wasn’t actually a compromise but a decision based on speed of coding as I’m more proficient in bash than perl and as such I decided to go with bash as a proof of concept that this thing can be done. This would never happen in reality as it would be much slower as compared to coding the same thing in perl or C / C++ (choice is up to you).

Now parsing the binary logs is not rocket science is it? As you can see there are three hashes ‘###’ in front of the readable query so a simple grep is fine. A few subsitutions and text processing and you’ll end up with:

darrencassar@mysqlpreacher $ /home/dcassar/sandbox/5.1.30/bin/mysqlbinlog  | sed ’s/^ *//g’ | tr ‘\015\012′ ‘\020 ‘ | sed ’s/ INSERT/;\nINSERT/g’ | sed ’s/ DELETE/;\nDELETE/g’ | sed ’s/ UPDATE/;\nUPDATE/g’ | sed ‘${/^$/!s/$/;\
> /;}’
INSERT INTO test.t1 SET @1=1 @2=’a';
DELETE FROM test.t1 WHERE @1=1 @2=’a';
INSERT INTO test.t2 SET @1=2 @2=’d';
UPDATE test.t2 WHERE @1=2 @2=’d’ SET @1=4 @2=’d';
INSERT INTO test.t2 SET @1=1 @2=NULL;
UPDATE test.t2 WHERE @1=4 @2=’d’ SET @1=2 @2=’d';
UPDATE test.t2 WHERE @1=1 @2=NULL SET @1=2 @2=NULL ;
darrencassar@mysqlpreacher $

That is more readable but still not correctly formatted for our ANSI SQL slave.

`INSERT INTO test.t1 SET @1=1 @2=’a';` would need to be replaced with `INSERT INTO test.t1 (cola, colb) values(1,’a');` or `INSERT INTO test.t1 values(1,’a');`,
`DELETE FROM test.t1 WHERE @1=1 @2=’a';` would need to be replaced with `DELETE FROM test.t1 WHERE cola=1 AND colb=’a';` and
`UPDATE test.t2 WHERE @1=2 @2=’d’ SET @1=4 @2=’d';` would have to become `UPDATE test.t2 SET cola=4 , colb=’d’ WHERE cola=2 AND colb=’d';`

The above means we need to replace @1 and @2 with the appropriate column names done using a lookup table in my case by placing a files in data/dbname each bearing names of the different tables and listing each column in order:
I.E. for a database named test and table named table1 having columns cola and colb I used the database folder named test in the mysql data folder and placed a file named table1, the contents of which were:

cola
colb

The following piece of code does the job of replacing those dreadful @1, @2 etc with proper table names:

for (( i=1; i<=`wc -l $dbname/$tbname | cut -d ” ” -f 1`; i++ ))
do
columnname=`awk ‘NR==a’ a=$i $dbname/$tbname`
LINE_TEMP=`echo $LINE_TEMP | sed ’s/@’$i’=/AND ‘$columnname’=/g’ | sed ’s/SET AND/SET/g’ | sed ’s/WHERE AND/WHERE/g’`
done

If you were thinking how I extracted dbname and tbname, here it is:

dbname=`echo $LINE | cut -d ” ” -f$col | cut -d “.” -f 1`
tbname=`echo $LINE | cut -d ” ” -f$col | cut -d “.” -f 2`

where $LINE is each line extracted using the first command parsing mysqlbinary output and the variable col is 3 for delete and 2 for update which reflects the positioning of the database name and table name inside the extracted lines.

A rather ugly way of re-ordering the update command yet functional is:

echo $LINE_TEMP | cut -d “.” -f2 | sed ’s/’$tbname’ //’ | sed ’s/;//’ > templinefile
cat templinefile | sed ’s/ SET /\nSET /g’ > templinefile2
LINE_TEMP=”UPDATE $dbname.$tbname `tail -1 templinefile2  | sed ’s/ is NULL/=NULL/g’ | sed ’s/AND/,/g’` `head -1 templinefile2`;”

The last thing to remember is taking care of `=NULL` and replace it by `is NULL`.

LINE_TEMP=`echo $LINE_TEMP | sed ’s/=NULL/ is NULL/g’`

As I said this was a fast proof of concept rather than a full fledged optimized script doing the job! The total length of code is 37 lines (excluding comments but with correct and nead formatting).

Enjoy
Darren

Today we’re going through the steps required for a typical MySQL installation from a .tar.gz package on a *nix based platform, including the download, installation, configuration and securing.

Steps involved:
1. Download MySQL binary tarball from mysql.com
2. Create a folder structure where the installation will be held.
3. Install the package
4. Secure the installation

Step 1: Download MySQL

Go to http://dev.mysql.com/downloads/ and choose the particular version that suits your platform (OS and machine type). Do note that installing a 32 bit version of MySQL on a 64 bit machine will not give you the full power available. There is quite a bit of performance impact when going for 64 bit, so if possible install the 64 bit version.

Step 2: Set up a folder structure to hold the MySQL related files

Normally in any database server I like to create a /mysql which will be holding any MySQL related info including, the releases, installation, configuration, data, logs etc. The below is a just personal choice and is subject to your liking. A typical folder structure would be

/mysql/ ->
     applications ->
          maatkit
          sandbox
          etc
     doc ->
          MySQL books
          RefMans
          etc

     installations ->
          mysql_cnf ->
               mysqlpreacher_master_malta_3306
               mysqlpreacher_slave_malta_3406
               etc

          mysql_data ->
               mysqlpreacher_master_malta_3306
               mysqlpreacher_slave_malta_3406

          mysql_dist ->
               ..
          mysql_inst ->
               ..
          mysql_logs ->
               ..

     releases ->
          5.0
          5.1
          6.0

     testing ->
          perl
          sandbox
          scripts
          etc

Step 3: Installing MySQL

The installation part takes just a few standard commands which follow:

#Execute the following only if you don’t already have a mysql user and mysql group. If you are not sure run: `cat /etc/group | grep -i mysql` and `cat /etc/group | grep -i mysql`. The ‘grep -i’ permits you to find just the user mysql or variant of it including those of a different case.

mysqlpreacher:~ darrencassar$ cat /etc/group | grep -i mysql
_mysql:*:74:
mysqlpreacher:~ darrencassar$ cat /etc/passwd | grep -i mysql
_mysql:*:74:74:MySQL Server:/var/empty:/usr/bin/false

#groupadd mysql
#useradd -g mysql mysql

#Let’s assume you followed the above folder structure

cd /mysql/releases/5.1
ls
gunzip mysql-5.1.xx-xxx-xx.tar.gz
tar -xf mysql-5.1.xx-xxx-xx.tar
ls

mysqlpreacher:~ darrencassar$ cd /mysql/releases/5.1/
mysqlpreacher:5.1 darrencassar$ ls
5.1.26                    mysql-5.1.26-rc-osx10.5-x86_64.dmg    mysql-5.1.28-rc-osx10.5-x86.tar.gz
5.1.28                    mysql-5.1.26-rc-osx10.5-x86_64.tar.gz    mysql-5.1.30-osx10.5-x86_64.tar.gz
mysqlpreacher:5.1 darrencassar$ gunzip mysql-5.1.30-osx10.5-x86_64.tar.gz
mysqlpreacher:5.1 darrencassar$ tar -xf mysql-5.1.30-osx10.5-x86_64.tar
mysqlpreacher:5.1 darrencassar$ ls
5.1.26                    mysql-5.1.26-rc-osx10.5-x86_64.dmg    mysql-5.1.28-rc-osx10.5-x86.tar.gz    mysql-5.1.30-osx10.5-x86_64.tar
5.1.28                    mysql-5.1.26-rc-osx10.5-x86_64.tar.gz    mysql-5.1.30-osx10.5-x86_64

mv mysql-5.1.xx-xxx-xx ../../installations/mysql_inst/
cd /mysql/installations/mysql_inst/
mv mysql-5.1.xx-xxx-xx mysql-5.1.xx-xxx-xx_001

mysqlpreacher:5.1 darrencassar$ mv mysql-5.1.30-osx10.5-x86_64 ../../installations/mysql_inst/
mysqlpreacher:5.1 darrencassar$ cd /mysql/installations/mysql_inst/
mysqlpreacher:mysql_inst darrencassar$ mv mysql-5.1.30-osx10.5-x86_64 mysql-5.1.30-osx10.5-x86_64_001

#set up your naming convention, this is a way I find useful to work with myself
ln -s mysql-5.1.xx-xxx-xx_001 mysql_51xx_001
cd mysql_51xx_001

mysqlpreacher:mysql_inst darrencassar$ ln -s mysql-5.1.30-osx10.5-x86_64_001 mysql_5130_001
mysqlpreacher:mysql_inst darrencassar$ cd mysql_5130_001

#set ownerships and permissions
sudo chown -R mysql .
sudo chgrp -R mysql .

mysqlpreacher:mysql_5130_001 darrencassar$ sudo chown -R mysql .
mysqlpreacher:mysql_5130_001 darrencassar$ sudo chgrp -R mysql .

#Install the MySQL using mysql_install_db
sudo scripts/mysql_install_db –user=mysql

mysqlpreacher:mysql_5130_001 darrencassar$ sudo scripts/mysql_install_db --user=mysql
WARNING: The host 'mysqlpreacher' could not be looked up with resolveip.
This probably means that your libc libraries are not 100 % compatible
with this binary MySQL version. The MySQL daemon, mysqld, should work
normally with the exception that host name resolving will not work.
This means that you should use IP addresses instead of hostnames
when specifying MySQL privileges !
Installing MySQL system tables...
OK
Filling help tables...
OK

To start mysqld at boot time you have to copy
support-files/mysql.server to the right place for your system

PLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER !
To do so, start the server, then issue the following commands:

./bin/mysqladmin -u root password 'new-password'
./bin/mysqladmin -u root -h mysqlpreacher password 'new-password'

Alternatively you can run:
./bin/mysql_secure_installation

which will also give you the option of removing the test
databases and anonymous user created by default.  This is
strongly recommended for production servers.

See the manual for more instructions.

You can start the MySQL daemon with:
cd . ; ./bin/mysqld_safe &

You can test the MySQL daemon with mysql-test-run.pl
cd ./mysql-test ; perl mysql-test-run.pl

Please report any problems with the ./bin/mysqlbug script!

The latest information about MySQL is available at http://www.mysql.com/
Support MySQL by buying support/licenses from http://shop.mysql.com/

sudo chown -R root .
sudo chown -R mysql data

mysqlpreacher:mysql_5130_001 darrencassar$ sudo chown -R root .
mysqlpreacher:mysql_5130_001 darrencassar$ sudo chown -R mysql data

#provide a link from the data folder onto the mysql_data folder
cd /mysql/installations/mysql_data/
ln -s  /mysql/installations/mysql_inst/mysql_51xx_001/data/ data_51xx_001

cd /mysql/installations/mysql_inst/mysql_51xx_001/
sudo su

mysqlpreacher:mysql_5130_001 darrencassar$ cd /mysql/installations/mysql_data/
mysqlpreacher:mysql_data darrencassar$ ln -s /mysql/installations/mysql_inst/mysql_5130_001/data/ data_5130_001
mysqlpreacher:mysql_data darrencassar$ cd /mysql/installations/mysql_inst/mysql_5130_001
mysqlpreacher:mysql_5130_001 darrencassar$ mkdir /mysql/installations/mysql_cnf/mysqlpreacher_master_malta_3306/
mysqlpreacher:mysql_5130_001 darrencassar$ cp -rp support-files/my-small.cnf /mysql/installations/mysql_cnf/mysqlpreacher_master_malta_3306/my.cnf
mysqlpreacher:mysql_5130_001 darrencassar$ sudo su
Password:
sh-3.2# pwd
/mysql/installations/mysql_inst/mysql-5.1.30-osx10.5-x86_64_001

#Depending upon the size of your dataset, a generic my.cnf to use (options file where you set specific parameters which are read by MySQL before starting up) can be obtained from the support-files folder. This file normally requires quite some tweaking depending on the hardware, dataset, type of application querying the database, etc. Configuration is not a one time task either since things change with time, and so must the parameters set in this file.

#The generic samples of my.cnf are:
my-small.cnf, my-medium.cnf, my-large.cnf, and my-huge.cnf

#Unless the location of the options file described above is hardcoded in the startup script (mysql.server), the order by which MySQL looks for a my.cnf is:

Filename                            Purpose

/etc/my.cnf                       Options
/etc/mysql/my.cnf            Global options (as of MySQL 5.1.15)
SYSCONFDIR/my.cnf          Global options
$MYSQL_HOME/my.cnf      Server-specific options
defaults-extra-file             The file specified with –defaults-extra-file=path, if any
~/.my.cnf                          User-specific options

#Once the installation is complete, just start MySQL using:
sudo bin/mysqld_safe  –defaults-file=/mysql/installations/mysql_cnf/mysqlpreacher_master_malta_3306/my.cnf –user=mysql &
#note that the –defaults-file needs to be the first parameter in this command otherwise you’ll see an error “Too many arguments (first extra is ‘–defaults-file=/mysql/installations/mysql_cnf/mysqlpreacher_master_malta_3306/my.cnf’).”

#you can check if MySQL did indeed start using the following
ps -ef | grep -i mysql

sh-3.2# sudo bin/mysqld_safe  --defaults-file=/mysql/installations/mysql_cnf/mysqlpreacher_master_malta_3306/my.cnf --user=mysql &
[1] 88880
sh-3.2# 090105 03:19:09 mysqld_safe Logging to '/mysql/installations/mysql_inst/mysql-5.1.30-osx10.5-x86_64_001/data/mysqlpreacher.err'.
090105 03:19:09 mysqld_safe Starting mysqld daemon with databases from /mysql/installations/mysql_inst/mysql-5.1.30-osx10.5-x86_64_001/data

sh-3.2# ps -ef | grep -i mysql
0 88880 88282   0   0:00.02 ttys001    0:00.04 /bin/sh bin/mysqld_safe --defaults-file=/mysql/installations/mysql_cnf/mysqlpreacher_master_malta_3306/my.cnf --user=mysql
74 88992 88880   0   0:00.04 ttys001    0:00.12 /mysql/installations/mysql_inst/mysql-5.1.30-osx10.5-x86_64_001/bin/mysqld --defaults-file=/mysql/installations/mysql_cnf/mysqlpreacher_master_malta_3306/my.cnf --basedir=/mysql/installations/mysql_inst/mysql-5.1.30-osx10.5-x86_64_001 --datadir=/mysql/installations/mysql_inst/mysql-5.1.30-osx10.5-x86_64_001/data --user=mysql --log-error=/mysql/installations/mysql_inst/mysql-5.1.30-osx10.5-x86_64_001/data/mysqlpreacher.err --pid-file=/mysql/installations/mysql_inst/mysql-5.1.30-osx10.5-x86_64_001/data/mysqlpreacher.pid --socket=/tmp/mysql.sock --port=3306
0 89007 88282   0   0:00.00 ttys001    0:00.00 grep -i mysql

Step 4: Secure the installation

When a database is placed online it does require some careful securing since it would otherwise be vulnerable to malicious acts such as alterations and denial of service.

MySQL uses Access Controlled Lists in order to discriminate between users, that means each user is given access to perform certain tasks while prohibited from doing others. It is quite flexible but managing fine grained rights allocation to a lot of users can also become quite a nightmare (consult maatkit mk_show_grants at http://www.maatkit.org/doc/mk-show-grants.html for this kind of setup). A MySQL command helpful in the latter task is:
SHOW GRANTS FOR ‘user’@'host’;

Using GRANT and REVOKE will permit you to set privileges for individual users e.g.
In order to grant select to john on database FOO while connected via localhost, the command would be:
GRANT SELECT ON FOO.* TO ‘john’@'localhost’ IDENTIFIED BY ‘password’;
FLUSH PRIVILEGES;

Note that omitting the ‘identified by’ would cause a user to have access without a password, something which should never be permitted for an online live server.

A further security measure, apart from ACL described above is to place MySQL server behind a firewall, preferably inside a DMZ (demilitarized zone) and should the MySQL server be accessible only through the socket file, then it would be a good idea to insert ’skip-networking’ in the configuration file my.cnf thus disabling any client from accessing the instance from any machine other than the machine on which it is installed.

One last suggestion as regards the security topic, consult MySQL updates and security patches in case there is something which directly impacts your current setup so that you can upgrade immediately if you are vulnerable in any way.

#access MySQL directly can be done using
bin/mysql -u root -p
SHOW DATABASES;

#If running a live MySQL system, run `drop database test`, you can always execute `create database test` later

#Create a password for root
USE MYSQL;

UPDATE user
SET Password = PASSWORD(“password”)
WHERE User=’root’;

FLUSH PRIVILEGES;
EXIT

#checking MySQL password set earlier
bin/mysql -u root -p #don’t insert a password to test it out
bin/mysql -u root -p #…. insert the new password (“password” in our case)

#It is good practice to set up a monitoring script which queries MySQL for any user without a password or for anonymous users.

USE MYSQL;
SELECT User, Host, Password
FROM user
WHERE User=” OR Host=” OR Password=”;

#This should be remedied using:
DELETE FROM user
WHERE User=”;

sh-3.2# bin/mysql -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.1.30 MySQL Community Server (GPL)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> SHOW DATABASES;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| test               |
+--------------------+
3 rows in set (0.02 sec)

mysql> USE MYSQL;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> UPDATE user
-> SET Password = PASSWORD("password")
-> WHERE User='root';
Query OK, 3 rows affected (0.04 sec)
Rows matched: 3  Changed: 3  Warnings: 0

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.02 sec)

mysql> EXIT
Bye
sh-3.2# bin/mysql -u root -p
Enter password:
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)
sh-3.2# bin/mysql -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.1.30 MySQL Community Server (GPL)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> USE MYSQL;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> SELECT User, Host, Password
-> FROM user
-> WHERE User='' OR Host='' OR Password='';
+------+---------------+----------+
| User | Host          | Password |
+------+---------------+----------+
|      | localhost     |          |
|      | mysqlpreacher |          |
+------+---------------+----------+
2 rows in set (0.00 sec)

mysql> DELETE FROM user
-> WHERE User='';
Query OK, 2 rows affected (0.02 sec)

mysql> SELECT User, Host, Password
-> FROM user
-> WHERE User='' OR Host='' OR Password='';
Empty set (0.00 sec)

mysql> \q
Bye
sh-3.2#

This section does not exhaust all security guidelines of course, but helps making your instance more secure.

#DONE

———– o ———–

#A few general notes:

#Note that not all the MySQL commands need a ‘;’ at the end, an example is `\s` and `SHOW STATUS \G`
#Exiting from MySQL can be done using `EXIT` or `QUIT` or `\q`
#A helpful command to get you started is `?` which provides a list of commands available

#There are many other commands and combination of syntaxes which you can write in order to obtain results

#It is good practice to insert MySQL path in the env variable PATH in order to avoid typing the whole path each time you want to use MySQL. In bash you’d do:

export PATH=/path/to/mysql/bin:$PATH

o insert the MySQL’s path before all the others, otherwise just interchange the positions of $PATH and the actual /path/to/mysql/bin

#Killing the MySQL is done using the following command:
bin/mysqladmin -u root -p shutdown

#Starting and stopping MySQL automatically can be done using mysql.server, found in support-files folder. The file needs to be copied in /etc/rc.d and edited to insert the custom parameters of data directory, base directory etc.

#If the storage engine used is innodb, a good my.cnf option would be innodb_file_per_table since otherwise innodb will store all it’s table data in a single file which is not optimal when having big tables which need to be optimzed / analyzed.

#Remember to set up monitoring scripts should this be a live system in order to avoid downtime.