Say you have the following date: MAR 16 2008 09:12:51:893AM
SELECT DATE_FORMAT(STR_TO_DATE('MAR 16 2008 09:12:51:893AM','%M %d %Y %h:%i:%s:%f%p'),'%Y%m%d%k%i%s.%f'); --> 2008031691251.893000


What if its PM rather than AM
SELECT DATE_FORMAT(STR_TO_DATE('MAR 16 2008 09:12:51:893PM','%M %d %Y %h:%i:%s:%f%p'),'%Y%m%d%k%i%s.%f'); --> 20080316211251.893000


Ok so this is just simple string manipulation where:
%M is the month name
%d is day number
%Y is the year
%h is the hour
%i is the minute
%s is the second
%f is the microsecond
%p is the period: ante or post meridiem
In the DATE_FORMAT part we se a %k which is in 24hr format in order to loose the period.

A more detailed list is found here

Here is a demo:

mysql Tue Jun 15 12:32:37 2010 > CREATE TABLE test.abc(a DECIMAL(17,3)) ENGINE=MYISAM;
Query OK, 0 rows affected (0.03 sec)

mysql Tue Jun 15 12:32:45 2010 > INSERT INTO abc VALUES ( DATE_FORMAT(STR_TO_DATE('MAR 16 2008 09:12:51:893PM','%M %d %Y %h:%i:%s:%f%p'),'%Y%m%d%k%i%s.%f') );
Query OK, 1 row affected (0.01 sec)

mysql Tue Jun 15 12:32:51 2010 > SELECT * FROM abc;
+--------------------+
| a                  |
+--------------------+
| 20080316211251.893 |
+--------------------+
1 row in set (0.00 sec)

mysql Tue Jun 15 12:32:56 2010 > SELECT TIMESTAMP(a) FROM abc;
+----------------------------+
| TIMESTAMP(a)               |
+----------------------------+
| 2008-03-16 21:12:51.893000 |
+----------------------------+
1 row in set (0.00 sec)

Create an expect script:
/path/to/sshmysql.exp

#!/usr/bin/expect -f
#script by darren cassar
#mysqlpreacher.com

set machine [lindex $argv 0]

set timeout -1

spawn ssh username@$machine
match_max 100000
expect -exact “assword: ”
send — “password\r”
send — “sudo -k; sudo su – mysql\r”
expect -exact “sudo -k; sudo su – mysql”
expect -exact “assword:”
send — “password\r”
interact

# you should change the word password in ’send — “password\r”‘ to your login password
# if you have the same password for each environment you could also script logging into mysql directly from the same expect script BUT that is not recommended.

Create a bash script:
/path/to/login.sh

#!/bin/bash
#script by darren cassar
#mysqlpreacher.com

sm=’/path/to/sshmysql.exp’

menu() {
echo ” 101 – dev.databaseserver1 ”
echo ” 102 – dev.databaseserver2 ”
echo ” 103 – dev.databaseserver3 ”
echo ” 201 – qa.databaseserver1 ”
echo ” 301 – uat.databaseserver1 ”
echo ” 302 – uat.databaseserver2 ”
echo ” 401 – prod.databaseserver1 ”
echo ” ”
}

ARGUMENT=notmenu

if [ -z "$1" ]
then
ARGUMENT=menu
else
choice=$1
fi

if [ $ARGUMENT = "menu" ]
then
menu
else
case “$choice” in
101|dev.databaseserver1 ) $sm dev.databaseserver1;;
102|dev.databaseserver2 ) $sm dev.databaseserver2;;
103|dev.databaseserver3 ) $sm dev.databaseserver3;;
201|qa.databaseserver1 ) $sm qa.databaseserver1;;
301|uat.databaseserver1 ) $sm uat.databaseserver1;;
302|uat.databaseserver2 ) $sm uat.databaseserver2;;
401|prod.databaseserver1 ) $sm prod.databaseserver1;;
* ) echo “Wrong value passed to script”
menu ;;
esac
fi

alias l=’/path/to/login.sh’

Output:

[darrencassar@mymachine ~ ]$ l
101 – dev.databaseserver1
102 – dev.databaseserver2
103 – dev.databaseserver3
201 – qa.databaseserver1
301 – uat.databaseserver1
302 – uat.databaseserver2
401 – prod.databaseserver1

Output:
The below command would log you into the first development database server as mysql user.

[darrencassar@mymachine ~ ]$ l 101

On each machine place aliases for each instance in the .profile

alias use3306=’mysql -u root -p -h 127.0.0.1 -P 3306 –prompt=”mysql \D> “‘

The above setup can be used using any client/server OS: Linux, Solaris, MAC OS or Windows(running Cygwin)

NOTE: If you store the password in clear text inside the expect script, you should at least save the scripts inside an encrypted partition on your machine and make sure that folder is not shared or accessible by anyone. Another way of doing it would be to use either SSHKeys OR save the password inside a file and encrypt it using OpenSSL

Enjoy!


mysql> explain SELECT COUNT(ac.UID) FROM ACTIVE ac JOIN ALL a;
+----+-------------+-------+------+---------------+------+---------+------+---------+-------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+---------------+------+---------+------+---------+-------+
| 1 | SIMPLE | ac | ALL | NULL | NULL | NULL | NULL | 124426 | |
| 1 | SIMPLE | a | ALL | NULL | NULL | NULL | NULL | 7594256 | |
+----+-------------+-------+------+---------------+------+---------+------+---------+-------+
2 rows in set (0.01 sec)

mysql> SELECT COUNT(ac.UID) FROM ACTIVE ac JOIN ALL a ON ac.UID=a.UID;
+---------------+
| COUNT(ac.UID) |
+---------------+
| 17466 |
+---------------+
1 row in set (0.23 sec)

mysql> set @tot = (SELECT COUNT(ac.UID) FROM ACTIVE ac JOIN ALL a ON ac.UID=a.UID);

^CQuery aborted by Ctrl+C


Took more than 60seconds —- what the …..
Why did it take a long?


mysql> set @a=2;
Query OK, 0 rows affected (0.00 sec)

mysql> select @a;
+------+
| @a |
+------+
| 2 |
+------+
1 row in set (0.00 sec)

mysql> set @tot = (SELECT COUNT(*) FROM ACTIVE);
Query OK, 0 rows affected (0.13 sec)

mysql> select @tot;
+--------+
| @tot |
+--------+
| 124426 |
+--------+
1 row in set (0.00 sec)

mysql> set @tot = (SELECT COUNT(ac.UID) FROM ACTIVE ac);
Query OK, 0 rows affected (0.22 sec)

mysql> select @tot;
+--------+
| @tot |
+--------+
| 124426 |
+--------+
1 row in set (0.00 sec)

mysql> SELECT COUNT(ac.UID) FROM ACTIVE ac, ALL a;
+---------------+
| COUNT(ac.UID) |
+---------------+
| 944922897056 |
+---------------+
1 row in set (0.05 sec)

mysql> set @tot=(SELECT COUNT(ac.UID) FROM ACTIVE ac JOIN ALL a);
^CQuery aborted by Ctrl+C
ERROR 1317 (70100): Query execution was interrupted


Reason? …. the query is using the MySQL optimiser rather than the IB one! Why? good question (will have to ask IB devs though).

Work around, use ac temporary table to store the result and setting the variable to the result field, but it’s really ugly isn’t it?

In fairness sake, MySQL have indeed supplied a way to retain milli and micro seconds in a decimal field `DECIMAL(17,3)`, and it is also queryable as if it were a timestamp BUT why isn’t it possible to store in a `DATETIME` or `TIMESTAMP` field? Why can’t we run a ’select now()’ or ’select curtime()’ etc and get a full timestamp including milli / micro seconds?

I have counted 37 different usernames asking for this feature, spanning from 15th Feb 2005 to recently. (list found below)

Some have suggested UDFs, others suggested using log4j while others, pretty annoyed, allegedly went to MS SQL and some to Postgres :). The comments were varied but one thing was quite common … a feeling of frustration and or disappointment, probably because despite all the requests for the feature in question to be implemented, it is still missing.

Back in 2007, Kevin Burton also tried to understand why MySQL doesn’t support milliseconds here. Brian Aker gave his view on the matter (back in 2007) in the first comment on the same blog post.

My point is (yeah about time :)) …. this is quite an important feature!! and I’m hoping a few MySQL developers / project managers / community people might read this blog post and maybe give it some more importance.

Now from the technical side, I am going to describing a way, `someone from MySQL has suggested it to me this week` to store milli and micro seconds. Maybe it’ll help you work your milliseconds requirement out (or maybe not). As to the way to generate milli and micro seconds, there is a UDF written by Mr Wadimoff here which you might use.

Storing:


mysql> create table ttable
-> (
-> timecol decimal(17,3)
-> ) Engine=MyISAM;
Query OK, 0 rows affected (0.04 sec)

mysql> insert into ttable (timecol) values( 20081223094234.572);
Query OK, 1 row affected (0.00 sec)

mysql> insert into ttable (timecol) values( 20090806133004.132);
Query OK, 1 row affected (0.00 sec)

mysql> insert into ttable (timecol) values( 20100411011258.985);
Query OK, 1 row affected (0.00 sec)

mysql> insert into ttable (timecol) values( 20110411011258.985);
Query OK, 1 row affected (0.00 sec)

mysql> select timestamp(timecol) from ttable;
+----------------------------+
| timestamp(timecol) |
+----------------------------+
| 2008-12-23 09:42:34.572000 |
| 2009-08-06 13:30:04.132000 |
| 2010-04-11 01:12:58.985000 |
| 2011-04-11 01:12:58.985000 |
+----------------------------+
4 rows in set (0.00 sec)

mysql> select timecol from ttable where timecol < "20090901000000";
+--------------------+
| timecol |
+--------------------+
| 20081223094234.572 |
| 20090806133004.132 |
+--------------------+
2 rows in set (0.00 sec)

mysql> select timecol from ttable where timecol > "20090901000000";
+--------------------+
| timecol |
+--------------------+
| 20100411011258.985 |
| 20110411011258.985 |
+--------------------+
2 rows in set (0.00 sec)

mysql> select timecol from ttable where timecol > now();
+--------------------+
| timecol |
+--------------------+
| 20100411011258.985 |
| 20110411011258.985 |
+--------------------+
2 rows in set (0.00 sec)

mysql> select datediff(20110411011258.985,20100411011258.985);
+-------------------------------------------------+
| datediff(20110411011258.985,20100411011258.985) |
+-------------------------------------------------+
| 365 |
+-------------------------------------------------+
1 row in set (0.00 sec)

mysql> select timecol, timestamp(timecol) as timestamp ,year(timecol) as year, month(timecol) as month, day(timecol) as day,quarter(timecol) as quarter from ttable;
+--------------------+----------------------------+------+-------+------+---------+
| timecol | timestamp | year | month | day | quarter |
+--------------------+----------------------------+------+-------+------+---------+
| 20081223094234.572 | 2008-12-23 09:42:34.572000 | 2008 | 12 | 23 | 4 |
| 20090806133004.132 | 2009-08-06 13:30:04.132000 | 2009 | 8 | 6 | 3 |
| 20100411011258.985 | 2010-04-11 01:12:58.985000 | 2010 | 4 | 11 | 2 |
| 20110411011258.985 | 2011-04-11 01:12:58.985000 | 2011 | 4 | 11 | 2 |
+--------------------+----------------------------+------+-------+------+---------+
4 rows in set (0.00 sec)

Bug link: http://bugs.mysql.com/bug.php?id=8523

List of people asking for this feature:

[15 Feb 2005 21:47] Christopher Miller
[03 May 2005 19:46] Boris Burtin
[30 Jun 2005 10:52] Yoshiaki Tajika
[06 Mar 2006 22:14] Tim Sheehy
[14 Mar 2006 21:50] Oriol Garrote
[17 Aug 2006 22:15] Sami Shalabi
[01 Apr 2007 05:22] Ben Valentine
[02 Apr 2007 23:25] Eric George
[27 Apr 2007 06:10] Verghese Mappillai
[23 May 2007 20:47] Andrew McLaughlin
[25 May 2007 18:24] Stephen Pietrowicz
[31 May 2007 10:51] harjeev chug
[22 Jun 2007 01:17] Peter McCulloch
[20 Jul 2007 16:42] Daren Schwenke
[03 Aug 2007 04:45] Jean-Guy Mossu
[20 Feb 2008 09:32] bodri bodri
[11 Mar 2008 20:12] Nathan Atkinson
[24 Mar 2008 08:35] Sourav Sipani
[02 Jul 2008 05:03] Michael Haselton
[22 Aug 2008 10:32] Aniruddha Shival
[28 Aug 2008 17:37] Jorge Urdaneta
[29 Aug 2008 22:19] Ben Wern
[05 Sep 2008 10:41] Pavel Alexeev
[19 Dec 2008 14:31] Felix Dierich
[09 Feb 2009 07:58] Gabriele Tozzi
[17 Feb 2009 23:20] Paul Craven
[21 Feb 2009 12:55] Thomas Ene
[27 Feb 2009 12:13] Pascal Calovini
[09 Mar 2009 09:03] Siu Ching Pong (Asuka Kenji)
[12 Mar 2009 19:41] Yu Chen
[02 Apr 2009 02:35] Noel Akins
[02 Apr 2009 13:09] Lars Monsees
[19 Apr 2009 17:30] Ryan Shillington
[22 Apr 2009 09:00] Jeff Peff
[04 May 2009 09:30] alastair knowles
[28 May 2009 11:42] Yuri Kirilin
[31 May 2009 04:37] Peter Thairu

The name is SECURICH and downloads as well as documentation are available at http://www.securich.com/. The bug list is available at http://www.securich.com/mantis/ (you’ll have to register and log in first). Privileges can be granted on a database or table level with a few combinations like:
1. database as one,
2. all tables in database (useful when you need to grant a bunch of privileges to all tables but a few thus revoking the few later),
3. single tables,
4. stored procedure or
5. tables through the use of regular expression.

The user has the facility to create a role and update it on the fly thus propagating the changes onto already existing users having the role in question. There are also loads of other functionalities available and I urge you to check the documentation on http://www.securich.com/.

A cool feature I like is password history which enables dbas to enforce changing of passwords after a certain period of time.

#################################################################################################################################################
## PLEASE NOTE THAT:
## 1. THIS IS AN ALPHA VERSION,
## 2. YOU SHOULD NOT INSTALL THIS PACKAGE ON A PRE-EXISTING MYSQL WHICH ALREADY HAS PRIVILEGES SET UP.
#################################################################################################################################################
##
## This tool will modify current privileges on the “mysql” database tables and should only be used with freshly installed mysqls.
##
##################################################################################################################################################

Features like block user and unblock user as well as many other functionalities are being developed and more features will be added as time goes by.

I would greatly appreciate your feedback about what you think of the package, features you might wish to include etc (feature requests should be done through http://www.securich.com/mantis/ as well.)

Recently I needed to replicate between MySQL and another database technology. You might say, why on earth would you want to do something like that, but believe me there are reasons and definitely not (to go away from MySQL to some other DB technology like Oracle or SQL server). Unsurprisingly there are quite a few different tools to do it from any platform towards MySQL but very few which do it the other way round, just to name a couple: Golden Gate and DSCallards.

Whilst not going into their tools (you can find more information on their websites), HIT from DSCallards needs its software to run on Windows and Golden Gate is an expensive beast which was too much for what I needed, thus I decided to have a look at doing the job myself. Although it might look an overkill to do so, it took me a few hours to find a solution and implement it and a couple more to test it and here is a simple description.

MySQL replication can be SBR (statement based), RBR (row based) or a mixture of both. Now despite the fact that the mixture provides the best performance, it would be the most complicated in order to achieve a home made solution, and SBR in my opinion would have also been a bit of a headache to make sure queries didn’t contain non ansi sql through the use of functions like now(), sysdate() and many others. I therefore decided that RBR would be the option of choice.

Although I wonder how many of you ever read an RBR binlog using `mysqlbinlog mysql-bin.000004`, it would be something like:

#090526 14:09:13 server id 1  end_log_pos 1420  Query   thread_id=192   exec_time=0     error_code=0
SET TIMESTAMP=1243343353/*!*/;
BEGIN
/*!*/;
# at 1420
# at 1464
#090526 14:09:13 server id 1  end_log_pos 1464  Table_map: `test`.`t2` mapped to number 21
#090526 14:09:13 server id 1  end_log_pos 1498  Write_rows: table id 21 flags: STMT_END_F

BINLOG ‘
+ekbShMBAAAALAAAALgFAAAAABUAAAAAAAAABHRlc3QAAnQyAAID/gL+AQM=
+ekbShcBAAAAIgAAANoFAAAQABUAAAAAAAEAAv/+AQAAAA==
‘/*!*/;
# at 1498
#090526 14:09:13 server id 1  end_log_pos 1567  Query   thread_id=192   exec_time=0     error_code=0
SET TIMESTAMP=1243343353/*!*/;
COMMIT
/*!*/;
# at 1567
#090526 14:09:38 server id 1  end_log_pos 1635  Query   thread_id=192   exec_time=0     error_code=0
SET TIMESTAMP=1243343378/*!*/;
BEGIN
/*!*/;
# at 1635
# at 1679
#090526 14:09:38 server id 1  end_log_pos 1679  Table_map: `test`.`t2` mapped to number 21
#090526 14:09:38 server id 1  end_log_pos 1733  Update_rows: table id 21 flags: STMT_END_F

BINLOG ‘
EuobShMBAAAALAAAAI8GAAAAABUAAAAAAAAABHRlc3QAAnQyAAID/gL+AQM=
EuobShgBAAAANgAAAMUGAAAQABUAAAAAAAEAAv///AQAAAABZPwCAAAAAWT+AQAAAP4CAAAA
‘/*!*/;
# at 1733
#090526 14:09:38 server id 1  end_log_pos 1802  Query   thread_id=192   exec_time=0     error_code=0
SET TIMESTAMP=1243343378/*!*/;
COMMIT

Now that isn’t the most readable text you ever seen right? As Giuseppe said “This is more difficult to read than ancient Etruscan. If you are a DBA, you curse and look for help.” But the replication guys at mysql created a nice -v for us to add to mysqlbinlog thus issuing `mysqlbinlog -v mysql-bin.000004` would result in the following:

#090526 14:09:13 server id 1  end_log_pos 1420  Query   thread_id=192   exec_time=0     error_code=0
SET TIMESTAMP=1243343353/*!*/;
BEGIN
/*!*/;
# at 1420
# at 1464
#090526 14:09:13 server id 1  end_log_pos 1464  Table_map: `test`.`t2` mapped to number 21
#090526 14:09:13 server id 1  end_log_pos 1498  Write_rows: table id 21 flags: STMT_END_F

BINLOG ‘
+ekbShMBAAAALAAAALgFAAAAABUAAAAAAAAABHRlc3QAAnQyAAID/gL+AQM=
+ekbShcBAAAAIgAAANoFAAAQABUAAAAAAAEAAv/+AQAAAA==
‘/*!*/;
### INSERT INTO test.t2
### SET
###   @1=1
###   @2=NULL
# at 1498
#090526 14:09:13 server id 1  end_log_pos 1567  Query   thread_id=192   exec_time=0     error_code=0
SET TIMESTAMP=1243343353/*!*/;
COMMIT
/*!*/;
# at 1567
#090526 14:09:38 server id 1  end_log_pos 1635  Query   thread_id=192   exec_time=0     error_code=0
SET TIMESTAMP=1243343378/*!*/;
BEGIN
/*!*/;
# at 1635
# at 1679
#090526 14:09:38 server id 1  end_log_pos 1679  Table_map: `test`.`t2` mapped to number 21
#090526 14:09:38 server id 1  end_log_pos 1733  Update_rows: table id 21 flags: STMT_END_F

BINLOG ‘
EuobShMBAAAALAAAAI8GAAAAABUAAAAAAAAABHRlc3QAAnQyAAID/gL+AQM=
EuobShgBAAAANgAAAMUGAAAQABUAAAAAAAEAAv///AQAAAABZPwCAAAAAWT+AQAAAP4CAAAA
‘/*!*/;
### UPDATE test.t2
### WHERE
###   @1=4
###   @2=’d’
### SET
###   @1=2
###   @2=’d’
### UPDATE test.t2
### WHERE
###   @1=1
###   @2=NULL
### SET
###   @1=2
###   @2=NULL
# at 1733
#090526 14:09:38 server id 1  end_log_pos 1802  Query   thread_id=192   exec_time=0     error_code=0
SET TIMESTAMP=1243343378/*!*/;
COMMIT

The exact same output with some decently readable output. The problem at this point is that the output is not really something any other technology would undestand (not even feeding it to MySQL would work! ….

At this point I needed to do some compromises in order to reduce the complexity of this job, i.e. I will not be creating and altering tables / databases or indexes during runtime (this I can do without as I can do the same things on the slave manually when I need to do anything like that on the master) and the replication won’t be in real time i.e. the slave will be fed the sql periodically through a script. The last compromise wasn’t actually a compromise but a decision based on speed of coding as I’m more proficient in bash than perl and as such I decided to go with bash as a proof of concept that this thing can be done. This would never happen in reality as it would be much slower as compared to coding the same thing in perl or C / C++ (choice is up to you).

Now parsing the binary logs is not rocket science is it? As you can see there are three hashes ‘###’ in front of the readable query so a simple grep is fine. A few subsitutions and text processing and you’ll end up with:

darrencassar@mysqlpreacher $ /home/dcassar/sandbox/5.1.30/bin/mysqlbinlog  | sed ’s/^ *//g’ | tr ‘\015\012′ ‘\020 ‘ | sed ’s/ INSERT/;\nINSERT/g’ | sed ’s/ DELETE/;\nDELETE/g’ | sed ’s/ UPDATE/;\nUPDATE/g’ | sed ‘${/^$/!s/$/;\
> /;}’
INSERT INTO test.t1 SET @1=1 @2=’a';
DELETE FROM test.t1 WHERE @1=1 @2=’a';
INSERT INTO test.t2 SET @1=2 @2=’d';
UPDATE test.t2 WHERE @1=2 @2=’d’ SET @1=4 @2=’d';
INSERT INTO test.t2 SET @1=1 @2=NULL;
UPDATE test.t2 WHERE @1=4 @2=’d’ SET @1=2 @2=’d';
UPDATE test.t2 WHERE @1=1 @2=NULL SET @1=2 @2=NULL ;
darrencassar@mysqlpreacher $

That is more readable but still not correctly formatted for our ANSI SQL slave.

`INSERT INTO test.t1 SET @1=1 @2=’a';` would need to be replaced with `INSERT INTO test.t1 (cola, colb) values(1,’a');` or `INSERT INTO test.t1 values(1,’a');`,
`DELETE FROM test.t1 WHERE @1=1 @2=’a';` would need to be replaced with `DELETE FROM test.t1 WHERE cola=1 AND colb=’a';` and
`UPDATE test.t2 WHERE @1=2 @2=’d’ SET @1=4 @2=’d';` would have to become `UPDATE test.t2 SET cola=4 , colb=’d’ WHERE cola=2 AND colb=’d';`

The above means we need to replace @1 and @2 with the appropriate column names done using a lookup table in my case by placing a files in data/dbname each bearing names of the different tables and listing each column in order:
I.E. for a database named test and table named table1 having columns cola and colb I used the database folder named test in the mysql data folder and placed a file named table1, the contents of which were:

cola
colb

The following piece of code does the job of replacing those dreadful @1, @2 etc with proper table names:

for (( i=1; i<=`wc -l $dbname/$tbname | cut -d ” ” -f 1`; i++ ))
do
columnname=`awk ‘NR==a’ a=$i $dbname/$tbname`
LINE_TEMP=`echo $LINE_TEMP | sed ’s/@’$i’=/AND ‘$columnname’=/g’ | sed ’s/SET AND/SET/g’ | sed ’s/WHERE AND/WHERE/g’`
done

If you were thinking how I extracted dbname and tbname, here it is:

dbname=`echo $LINE | cut -d ” ” -f$col | cut -d “.” -f 1`
tbname=`echo $LINE | cut -d ” ” -f$col | cut -d “.” -f 2`

where $LINE is each line extracted using the first command parsing mysqlbinary output and the variable col is 3 for delete and 2 for update which reflects the positioning of the database name and table name inside the extracted lines.

A rather ugly way of re-ordering the update command yet functional is:

echo $LINE_TEMP | cut -d “.” -f2 | sed ’s/’$tbname’ //’ | sed ’s/;//’ > templinefile
cat templinefile | sed ’s/ SET /\nSET /g’ > templinefile2
LINE_TEMP=”UPDATE $dbname.$tbname `tail -1 templinefile2  | sed ’s/ is NULL/=NULL/g’ | sed ’s/AND/,/g’` `head -1 templinefile2`;”

The last thing to remember is taking care of `=NULL` and replace it by `is NULL`.

LINE_TEMP=`echo $LINE_TEMP | sed ’s/=NULL/ is NULL/g’`

As I said this was a fast proof of concept rather than a full fledged optimized script doing the job! The total length of code is 37 lines (excluding comments but with correct and nead formatting).

Enjoy
Darren

Before starting to read, I am hereby assuming the MySQL list of MCDBAs is on a residence basis not citizenship.

There are at the time of this writing a total of 796 MCDBAs in the world, living in 57 countries.

When analyzing the number of MCDBAs / country, it is pretty obvious the scene is totally dominated by the US:

+----------------------+-------------+
| Country              | No of Certs |
+----------------------+-------------+
| UnitedStates         |         346 | 
| Unitedkingdom        |          47 | 
| Germany              |          46 | 
| India                |          34 | 
| Italy                |          28 | 
| Japan                |          26 | 
| Canada               |          23 | 
| China                |          20 | 
| Brazil               |          18 | 
| Australia            |          16 | 
| France               |          16 | 
| Netherlands          |          13 | 
| Switzerland          |          13 | 
| SouthKorea           |          12 | 
| Spain                |          11 | 
| HongKong             |          10 | 
| Sweden               |          10 | 
| SouthAfrica          |          10 | 
| Singapore            |           9 | 
| Taiwan               |           8 | 
| Poland               |           7 | 
| Philippines          |           6 | 
| Norway               |           5 | 
| Russia               |           5 | 
| Finland              |           5 | 
| Ireland              |           4 | 
| Israel               |           3 | 
| Srilanka             |           3 | 
| Ukraine              |           3 | 
| Austria              |           3 | 
| Belgium              |           3 | 
| Denmark              |           2 | 
| Egypt                |           2 | 
| Mexico               |           2 | 
| Malta                |           2 | 
| Thailand             |           2 | 
| Unitedarabemirates   |           2 | 
| Latvia               |           2 | 
| Bangladesh           |           1 | 
| Croatia              |           1 | 
| TrinidadandTobago    |           1 | 
| BosniaandHerzegovina |           1 | 
| Malaysia             |           1 | 
| Slovenia             |           1 | 
| SaudiArabia          |           1 | 
| Indonesia            |           1 | 
| Jordan               |           1 | 
| Kenya                |           1 | 
| Lithuania            |           1 | 
| Luxembourg           |           1 | 
| Guatemala            |           1 | 
| Nepal                |           1 | 
| Gibraltar            |           1 | 
| Newzealand           |           1 | 
| Pakistan             |           1 | 
| Portugal             |           1 | 
| Virginislands        |           1 | 
+----------------------+-------------+ 


Typical solutions consist of:
Heartbeat with DRBD on Linux. Although HB was introduced to Linux, it can be used on Solaris, FreeBSD and others without the DRBD but with other solutions.
Sun Cluster,
Veritas (commercial),
MySQL Proxy (still in alpha at the time of writing this blog but on testing proved to be a sweet solution),
Sequoia,
Tungsten (commercial),
MySQL cluster (which sadly doesn’t fit my scenario) and others

Whist researching the above list of load balancers / auto failover solutions I decided to give Sequoia (a load balancer and auto failover tier) a shot. I documented the commands and configuration steps and am listing them here under.
Used windows for this one (shameful I know :P) but should work the same on *nix.

After downloading and unzipping Sequoia I went ahead and configured the controller.xml file. Apart from several other parameters available, this is the place where you should set the port where sequoia listens to requests and forwards them to the MySQL instances it is configured to in return connect to.

------------------------------> CONTROLLER.XML to be stored in sequoia.x.xx/config/controller

<?xml version=”1.0″ encoding=”UTF-8″ ?>

<!DOCTYPE SEQUOIA-CONTROLLER PUBLIC “-//Continuent//DTD SEQUOIA-CONTROLLER 2.10.10//EN”  “http://sequoia.continuent.org/dtds/sequoia-controller-2.10.10.dtd”>

<SEQUOIA-CONTROLLER>
<Controller port=”25322″>
<Report hideSensitiveData=”true” generateOnShutdown=”true” generateOnFatal=”true” enableFileLogging=”true” />
<JmxSettings>
<RmiJmxAdaptor/>
</JmxSettings>
</Controller>
</SEQUOIA-CONTROLLER>

Next came the distribution file which contains several general parameters for which you should read the documentation if you would like to know better what each does. I personally followed a standard file and tuned it for my needs.

------------------------------> TEST-MYSQL-DISTRIBUTION.XML to be stored in sequoia.x.xx/config/virtualdatabase

<?xml version=”1.0″ encoding=”UTF-8″?>
<!DOCTYPE SEQUOIA PUBLIC “-//Continuent//DTD SEQUOIA 2.10.10//EN” “http://sequoia.continuent.org/dtds/sequoia-2.10.10.dtd”>

<SEQUOIA>

<VirtualDatabase name=”test”>

<Monitoring>
<SQLMonitoring defaultMonitoring=”off”>
<SQLMonitoringRule queryPattern=”^select” caseSensitive=”false” applyToSkeleton =”false” monitoring=”on”/>
</SQLMonitoring>
</Monitoring>

<Backup>
<Backuper backuperName=”Octopus”
className=”org.continuent.sequoia.controller.backup.backupers.OctopusBackuper”
options=”zip=true”/>
</Backup>

<AuthenticationManager>
<Admin>
<User username=”admin” password=”"/>
</Admin>
<VirtualUsers>
<VirtualLogin vLogin=”user” vPassword=”"/>
</VirtualUsers>
</AuthenticationManager>

<DatabaseBackend name=”localhost1″ driver=”com.mysql.jdbc.Driver”
url=”jdbc:mysql://localhost:3306/test”
connectionTestStatement=”select 1″>
<ConnectionManager vLogin=”user” rLogin=”test” rPassword=”">
<VariablePoolConnectionManager initPoolSize=”10″ minPoolSize=”5″
maxPoolSize=”50″ idleTimeout=”30″ waitTimeout=”10″/>
</ConnectionManager>
</DatabaseBackend>

<DatabaseBackend name=”localhost2″ driver=”com.mysql.jdbc.Driver”
url=”jdbc:mysql://localhost:3307/test”
connectionTestStatement=”select 1″>
<ConnectionManager vLogin=”user” rLogin=”test” rPassword=”">
<VariablePoolConnectionManager initPoolSize=”10″ minPoolSize=”5″
maxPoolSize=”50″ idleTimeout=”30″ waitTimeout=”10″/>
</ConnectionManager>
</DatabaseBackend>

<RequestManager>
<RequestScheduler>
<RAIDb-1Scheduler level=”passThrough”/>
</RequestScheduler>

<RequestCache>
<MetadataCache/>
<ParsingCache/>
<!–       <ResultCache granularity=”table”/> –>
</RequestCache>

<LoadBalancer>
<RAIDb-1>
<WaitForCompletion policy=”first”/>
<RAIDb-1-LeastPendingRequestsFirst/>
</RAIDb-1>
</LoadBalancer>
<RecoveryLog driver=”com.mysql.jdbc.Driver”
url=”jdbc:mysql://localhost:3306/recovery” login=”test” password=”">
<RecoveryLogTable tableName=”RECOVERY”
logIdColumnType=”BIGINT NOT NULL”
vloginColumnType=”VARCHAR NOT NULL”
sqlColumnType=”VARCHAR NOT NULL”
extraStatementDefinition=”,PRIMARY KEY (log_id)”/>
<CheckpointTable tableName=”CHECKPOINT”
checkpointNameColumnType=”VARCHAR(255) NOT NULL”/>
<BackendTable tableName=”BACKEND”
databaseNameColumnType=”VARCHAR(255) NOT NULL”
backendNameColumnType=”VARCHAR(255) NOT NULL”
checkpointNameColumnType=”VARCHAR(255) NOT NULL”/>
<DumpTable tableName=”DUMP” dumpNameColumnType=”VARCHAR(255) NOT NULL”
dumpDateColumnType=”TIMESTAMP”
dumpPathColumnType=”VARCHAR(255) NOT NULL”
dumpFormatColumnType=”VARCHAR(255) NOT NULL”
checkpointNameColumnType=”VARCHAR(255) NOT NULL”
backendNameColumnType=”VARCHAR(255) NOT NULL”
tablesColumnType=”VARCHAR(255) NOT NULL”/>
</RecoveryLog>

</RequestManager>

</VirtualDatabase>

</SEQUOIA>

Given the above recovery log table contains illegal names for MySQL, it complains and therefore I went along and created the table directly using SQL. List of MySQL reserved words can be found here: http://dev.mysql.com/doc/mysqld-version-reference/en/mysqld-version-reference-reservedwords-5-1.html

------------------------------> INSTALLATION STEPS

1) Run the following create table before loading virtualdatabase configuration C:\test-mysql-distribution.xml on the console window:
CREATE DATABSE recovery

CREATE TABLE recovery (
`log_id` BIGINT NOT NULL UNIQUE,
`vlogin` TEXT NOT NULL,
`sql` VARCHAR(255) NOT NULL,
`sql_param` TEXT,
`auto_conn_tran` CHAR(1) NOT NULL,
`transaction_id` BIGINT,
`request_id` BIGINT,
`exec_status` CHAR(1) NOT NULL,
`exec_time` BIGINT,
`update_count` BIGINT,
PRIMARY KEY (`log_id`)
)

Edit controller.bat and console.bat to include the mysqlconnector in the CLASSPATH
SET CLASSPATH=;”C:\Program Files\sequoia-2.10.10-bin/drivers/mysql-connector-java-3.1.14-bin.jar”;$CLASSPATH

------------------------------> MySQL INSTALLATION STEPS

Install two mysql instances on 3306 and 3307 respectively, on the same machine (can be done on different machines and test-mysql-distribution.xml changed to suit the IPs. Set master – master replication and prepare startup / stop script. (For help installing master – master replication you can either use sandbox or install it customly using my.cnf replication settings and remember to set auto_increment_increment and auto_increment_offset according to how many masters you have.

On a two node setup they would be:

First Master the values:

auto_increment_increment = 2
auto_increment_offset = 1

Second Master:

auto_increment_increment = 2
auto_increment_offset = 2

If you are not knowledgable about the subject I recommend you use MySQL Sandbox or check out MySQL Documentation at: http://dev.mysql.com/doc/refman/5.1/en/replication.html

------------------------------> EXECUTION STEPS

Run controller.bat then console.bat and issue `load virtualdatabase configuration C:\test-mysql-distribution.xml` when ready

------------------------------> OUTPUT on CONSOLE.BAT window

NOTE if stuck in console, just hit `tab` and the commands will be displayed

Running the tools and software:

Launching the Sequoia controller console
Initializing Controller module...
Initializing VirtualDatabase Administration module...
Initializing SQL Console module...
Sequoia driver (Sequoia core v2.10.10) successfully loaded.
localhost:1090 > load virtualdatabase configuration C:\test-mysql-distribution.xml
XML file C:\test-mysql-distribution.xml successfully sent to Sequoia controller.
localhost:1090 > admin test
Virtual database Administrator Login > admin
Virtual database Administrator Password >
Ready to administrate virtual database test
test(admin) > initialize localhost1
Virtual Database test has been successfully initialized from backend localhost1
test(admin) > initialize localhost2
Backend localhost1 has a last known checkpoint (Initial_empty_recovery_log)
test(admin) > show backend localhost1
+--------------------------+----------------------------------+
| Backend Name | localhost1 |
| Driver | com.mysql.jdbc.Driver |
| URL | jdbc:mysql://localhost:3306/test |
| Active transactions | 0 |
| Pending Requests | 0 |
| Read Enabled | false |
| Write Enabled | false |
| Is Initialized | false |
| Static Schema | false |
| Connection Managers | 1 |
| Total Active Connections | 50 |
| Persistent Connections | 0 |
| Total Requests | 0 |
| Total Transactions | 0 |
| Last known checkpoint | Initial_empty_recovery_log |
+--------------------------+----------------------------------+

test(admin) > show backend localhost2
+--------------------------+----------------------------------+
| Backend Name | localhost2 |
| Driver | com.mysql.jdbc.Driver |
| URL | jdbc:mysql://localhost:3307/test |
| Active transactions | 0 |
| Pending Requests | 0 |
| Read Enabled | false |
| Write Enabled | false |
| Is Initialized | false |
| Static Schema | false |
| Connection Managers | 1 |
| Total Active Connections | 50 |
| Persistent Connections | 0 |
| Total Requests | 0 |
| Total Transactions | 0 |
| Last known checkpoint | |
+--------------------------+----------------------------------+

test(admin) > enable localhost1 force
Enabling backend localhost1 from its last known checkpoint
test(admin) > show backend localhost1
+--------------------------+----------------------------------+
| Backend Name | localhost1 |
| Driver | com.mysql.jdbc.Driver |
| URL | jdbc:mysql://localhost:3306/test |
| Active transactions | 0 |
| Pending Requests | 0 |
| Read Enabled | true |
| Write Enabled | true |
| Is Initialized | true |
| Static Schema | false |
| Connection Managers | 1 |
| Total Active Connections | 10 |
| Persistent Connections | 0 |
| Total Requests | 0 |
| Total Transactions | 0 |
| Last known checkpoint | |
+--------------------------+----------------------------------+

test(admin) > initialize localhost2 force
Virtual Database test has been successfully initialized from backend localhost2
test(admin) > enable localhost2
Enabling backend localhost2 from its last known checkpoint
test(admin) > show backend localhost1
+--------------------------+--------------------------------------------------------------+
| Backend Name | localhost1 |
| Driver | com.mysql.jdbc.Driver |
| URL | jdbc:mysql://localhost:3306/test |
| Active transactions | 0 |
| Pending Requests | 0 |
| Read Enabled | false |
| Write Enabled | false |
| Is Initialized | false |
| Static Schema | false |
| Connection Managers | 2 |
| Total Active Connections | 25 |
| Persistent Connections | 0 |
| Total Requests | 0 |
| Total Transactions | 0 |
| Last known checkpoint | backup localhost1-172.22.16.103:25322-20090311114054271+0000 |
+--------------------------+--------------------------------------------------------------+

test(admin) > disable localhost1
Disabling backend localhost1 with automatic checkpoint.
test(admin) > initialize localhost1 force
Virtual Database test has been successfully initialized from backend localhost1
test(admin) > enable localhost1
Enabling backend localhost1 from its last known checkpoint
test(admin) > show backend localhost1
+--------------------------+----------------------------------+
| Backend Name | localhost1 |
| Driver | com.mysql.jdbc.Driver |
| URL | jdbc:mysql://localhost:3306/test |
| Active transactions | 0 |
| Pending Requests | 0 |
| Read Enabled | true |
| Write Enabled | true |
| Is Initialized | true |
| Static Schema | false |
| Connection Managers | 2 |
| Total Active Connections | 30 |
| Persistent Connections | 0 |
| Total Requests | 0 |
| Total Transactions | 0 |
| Last known checkpoint | |
+--------------------------+----------------------------------+

test(admin) > show backend localhost2
+--------------------------+----------------------------------+
| Backend Name | localhost2 |
| Driver | com.mysql.jdbc.Driver |
| URL | jdbc:mysql://localhost:3307/test |
| Active transactions | 0 |
| Pending Requests | 0 |
| Read Enabled | true |
| Write Enabled | true |
| Is Initialized | true |
| Static Schema | false |
| Connection Managers | 2 |
| Total Active Connections | 25 |
| Persistent Connections | 0 |
| Total Requests | 3 |
| Total Transactions | 0 |
| Last known checkpoint | |
+--------------------------+----------------------------------+

test(admin) > show backend localhost2
+--------------------------+----------------------------------+
| Backend Name | localhost2 |
| Driver | com.mysql.jdbc.Driver |
| URL | jdbc:mysql://localhost:3307/test |
| Active transactions | 0 |
| Pending Requests | 0 |
| Read Enabled | true |
| Write Enabled | true |
| Is Initialized | true |
| Static Schema | false |
| Connection Managers | 2 |
| Total Active Connections | 25 |
| Persistent Connections | 0 |
| Total Requests | 3 |
| Total Transactions | 0 |
| Last known checkpoint | |
+--------------------------+----------------------------------+

test(admin) > disable localhost2
Disabling backend localhost2 with automatic checkpoint.
test(admin) > initialize localhost2 force
Virtual Database test has been successfully initialized from backend localhost2
test(admin) > enable localhost2
Enabling backend localhost2 from its last known checkpoint
test(admin) > disable localhost1
Disabling backend localhost1 with automatic checkpoint.
test(admin) > initialize localhost1
Backend localhost2 is not in a disabled state (current state is enabled)
test(admin) > disable localhost1
Disabling backend localhost1 with automatic checkpoint.
test(admin) > initialize localhost1
Backend localhost2 is not in a disabled state (current state is enabled)
test(admin) > initialize localhost1 force
Virtual Database test has been successfully initialized from backend localhost1
test(admin) > enable localhost1
Enabling backend localhost1 from its last known checkpoint
test(admin) >



------------------------------> OUTPUT on CMD JAVA PROCESS EXECUTION WINDOW

C:\Program Files\MySQL>java -cp .;"c:\Program Files\MySQL\sequoia-driver.jar" Connect
Database connection established
id = 2, name = snake, category = reptile
id = 4, name = frog, category = amphibian
id = 6, name = tuna, category = fish
id = 8, name = racoon, category = mammal
4 rows were retrieved
Database connection terminated



-------------------------------> Here I inserted some rows directly in the database and reran the java program


C:\Program Files\MySQL>java -cp .;"c:\Program Files\MySQL\sequoia-driver.jar" Connect
Database connection established
id = 2, name = snake, category = reptile
id = 4, name = frog, category = amphibian
id = 6, name = tuna, category = fish
id = 8, name = racoon, category = mammal
id = 10, name = darren, category = human
id = 12, name = michael, category = alien
6 rows were retrieved
Database connection terminated



------------------------------> The JAVA PROGRAM used, obtained from: http://www.kitebird.com/articles/jdbc.html :


import java.sql.*;

public class Connect
{
public static void main (String[] args)
{
Connection conn = null;

try
{
String userName = "admin";
String password = "";
String url = "jdbc:sequoia://127.0.0.1:25322/test?user=root";
Class.forName ("org.continuent.sequoia.driver.Driver").newInstance ();
conn = DriverManager.getConnection (url,userName,password);
System.out.println ("Database connection established");

/* Uncomment the following at the first run so that it creates a table called animal and inserts a few rows in it.
*/
/* Statement s = conn.createStatement ();
int count;
s.executeUpdate ("DROP TABLE IF EXISTS animal");
s.executeUpdate (
"CREATE TABLE animal ("
+ "id INT UNSIGNED NOT NULL AUTO_INCREMENT,"
+ "PRIMARY KEY (id),"
+ "name CHAR(40), category CHAR(40))");
count = s.executeUpdate (
"INSERT INTO animal (name, category)"
+ " VALUES"
+ "('snake', 'reptile'),"
+ "('frog', 'amphibian'),"
+ "('tuna', 'fish'),"
+ "('racoon', 'mammal')");
s.close ();
System.out.println (count + " rows were inserted"); : */

Statement s = conn.createStatement ();
s.executeQuery ("SELECT id, name, category FROM animal");
ResultSet rs = s.getResultSet ();
int count = 0;
while (rs.next ())
{
int idVal = rs.getInt ("id");
String nameVal = rs.getString ("name");
String catVal = rs.getString ("category");
System.out.println (
"id = " + idVal
+ ", name = " + nameVal
+ ", category = " + catVal);
++count;
}
rs.close ();
s.close ();
System.out.println (count + " rows were retrieved");



}
catch (Exception e)
{
System.err.println ("Cannot connect to database server");
// System.err.println (e);
e.printStackTrace();
}
finally
{
if (conn != null)
{
try
{
conn.close ();
System.out.println ("Database connection terminated");
}
catch (Exception e) { /* ignore close errors */ }
}
}
}
}



------------------------------> COMPILING / EXECUTING JAVA PROGRAM

download and install JDK
compile: javac Connect.java
execute: java -cp .;”c:\Program Files\MySQL\sequoia-driver.jar” Connect