Some good suggestions form Mark Matthews - Bug #5020

Having encountered the problem again myself today, trying to make jdbc for mysql reconnect any terminated connections using autoreconnect=true I figured out a way to work it out from the pooling side.

Introduction to the problem:

On the mysql side wait_timeout is set to default 8hrs and any connections idle for longer than that were beomg terminated despite setting the connection string to: url=jdbc:mysql://localhost:3306/dbname?autoReconnect=true. The application was thence throwing an exception.

The solution was to introduce a ping from the pooler which for “Ibatis”, the pooler technology used in this case, was:

This covers eventualities of network glitches or connections exceeding the wait_timeout having the pooler ping the database if the connection was idle for more than 10 seconds (the value is in milliseconds).

The configuration if Ibatis in this case is something like:

Ibatis


For every pooling technology there will be a different “Pool.PingConnectionsNotUsedFor” term, but the concept is still there. It worked like a charm.


JDBC driver=com.mysql.jdbc.Driver
url=jdbc:mysql://localhost:3306/dbname
username=username
password=password

PS … autoreconnect is not recommended when using mysql jdbc!

Say you have the following date: MAR 16 2008 09:12:51:893AM
SELECT DATE_FORMAT(STR_TO_DATE('MAR 16 2008 09:12:51:893AM','%M %d %Y %h:%i:%s:%f%p'),'%Y%m%d%k%i%s.%f'); --> 2008031691251.893000


What if its PM rather than AM
SELECT DATE_FORMAT(STR_TO_DATE('MAR 16 2008 09:12:51:893PM','%M %d %Y %h:%i:%s:%f%p'),'%Y%m%d%k%i%s.%f'); --> 20080316211251.893000


Ok so this is just simple string manipulation where:
%M is the month name
%d is day number
%Y is the year
%h is the hour
%i is the minute
%s is the second
%f is the microsecond
%p is the period: ante or post meridiem
In the DATE_FORMAT part we se a %k which is in 24hr format in order to loose the period.

A more detailed list is found here

Here is a demo:

mysql Tue Jun 15 12:32:37 2010 > CREATE TABLE test.abc(a DECIMAL(17,3)) ENGINE=MYISAM;
Query OK, 0 rows affected (0.03 sec)

mysql Tue Jun 15 12:32:45 2010 > INSERT INTO abc VALUES ( DATE_FORMAT(STR_TO_DATE('MAR 16 2008 09:12:51:893PM','%M %d %Y %h:%i:%s:%f%p'),'%Y%m%d%k%i%s.%f') );
Query OK, 1 row affected (0.01 sec)

mysql Tue Jun 15 12:32:51 2010 > SELECT * FROM abc;
+--------------------+
| a                  |
+--------------------+
| 20080316211251.893 |
+--------------------+
1 row in set (0.00 sec)

mysql Tue Jun 15 12:32:56 2010 > SELECT TIMESTAMP(a) FROM abc;
+----------------------------+
| TIMESTAMP(a)               |
+----------------------------+
| 2008-03-16 21:12:51.893000 |
+----------------------------+
1 row in set (0.00 sec)

Say you were trying to run a query which returned a heap of warnings and you want to go through the list ….

mysql> load data infile '/aaa/bbb/ccc' into table xyz LINES TERMINATED BY '\r\n';
Query OK, 0 rows affected, 65535 warnings (4 min 25.12 sec)

mysql> show warnings;
+---------+------+------------------------------------------------+
| Level   | Code | Message                                        |
+---------+------+------------------------------------------------+
| Warning | 1366 | Incorrect value: '' for column 'xx' at row 1   |
...
| Warning | 1366 | Incorrect value: '' for column 'xx' at row 9   |
| Warning | 1366 | Incorrect value: '' for column 'xx' at row 10  |
+---------+------+------------------------------------------------+
64 rows in set (0.00 sec)

As per above only 64 rows are show. Why?

mysql> show variables like 'max_error_count';
+-----------------+-------+
| Variable_name   | Value |
+-----------------+-------+
| max_error_count | 64    |
+-----------------+-------+
1 row in set (0.00 sec)

Ok so … what is the solution?

mysql> set max_error_count=65535;
Query OK, 0 rows affected (0.00 sec)

mysql> pager more
PAGER set to 'more'

Here I am setting the max_error_count to a high value only for the session and not globally.

Details here

The “pager more” will work only on *nix systems where the output will be “piped” to the `more` command on your OS and displayed in batches so you can pause the scrolling of the long output.

Having said the above, from my experience, when you have that many warnings, it’ll be a repetition of the same error or group of errors.

Create an expect script:
/path/to/sshmysql.exp

#!/usr/bin/expect -f
#script by darren cassar
#mysqlpreacher.com

set machine [lindex $argv 0]

set timeout -1

spawn ssh username@$machine
match_max 100000
expect -exact “assword: ”
send — “password\r”
send — “sudo -k; sudo su – mysql\r”
expect -exact “sudo -k; sudo su – mysql”
expect -exact “assword:”
send — “password\r”
interact

# you should change the word password in ’send — “password\r”‘ to your login password
# if you have the same password for each environment you could also script logging into mysql directly from the same expect script BUT that is not recommended.

Create a bash script:
/path/to/login.sh

#!/bin/bash
#script by darren cassar
#mysqlpreacher.com

sm=’/path/to/sshmysql.exp’

menu() {
echo ” 101 – dev.databaseserver1 ”
echo ” 102 – dev.databaseserver2 ”
echo ” 103 – dev.databaseserver3 ”
echo ” 201 – qa.databaseserver1 ”
echo ” 301 – uat.databaseserver1 ”
echo ” 302 – uat.databaseserver2 ”
echo ” 401 – prod.databaseserver1 ”
echo ” ”
}

ARGUMENT=notmenu

if [ -z "$1" ]
then
ARGUMENT=menu
else
choice=$1
fi

if [ $ARGUMENT = "menu" ]
then
menu
else
case “$choice” in
101|dev.databaseserver1 ) $sm dev.databaseserver1;;
102|dev.databaseserver2 ) $sm dev.databaseserver2;;
103|dev.databaseserver3 ) $sm dev.databaseserver3;;
201|qa.databaseserver1 ) $sm qa.databaseserver1;;
301|uat.databaseserver1 ) $sm uat.databaseserver1;;
302|uat.databaseserver2 ) $sm uat.databaseserver2;;
401|prod.databaseserver1 ) $sm prod.databaseserver1;;
* ) echo “Wrong value passed to script”
menu ;;
esac
fi

alias l=’/path/to/login.sh’

Output:

[darrencassar@mymachine ~ ]$ l
101 – dev.databaseserver1
102 – dev.databaseserver2
103 – dev.databaseserver3
201 – qa.databaseserver1
301 – uat.databaseserver1
302 – uat.databaseserver2
401 – prod.databaseserver1

Output:
The below command would log you into the first development database server as mysql user.

[darrencassar@mymachine ~ ]$ l 101

On each machine place aliases for each instance in the .profile

alias use3306=’mysql -u root -p -h 127.0.0.1 -P 3306 –prompt=”mysql \D> “‘

The above setup can be used using any client/server OS: Linux, Solaris, MAC OS or Windows(running Cygwin)

NOTE: If you store the password in clear text inside the expect script, you should at least save the scripts inside an encrypted partition on your machine and make sure that folder is not shared or accessible by anyone. Another way of doing it would be to use either SSHKeys OR save the password inside a file and encrypt it using OpenSSL

Enjoy!

It seems this user can’t connect to the mysql gets access denied:
Access denied for user ‘user’@'machine_bbbbb’ (using password: YES)

Note that the machine the user is being seen from is totally different from the one I set up in the grant!! WHY?

run a reverse lookup on the ip of machine_aaaaa, turns out it shows machine_bbbbb. So I figure a big bad guy messed up /etc/hosts, I was right! `cat /etc/hosts` just to find an entry for machine_aaaaa blehh

Ok, solution is to remove the entry from /etc/hosts (after finding out it wasn’t even necessary and wasn’t even supposed to be there in the first place), restart nscd.

Retry
AGAIN – Access denied for user ‘user’@'machine_bbbbb’ (using password: YES)

What the ….

What’s wrong now? — yeah silly me forgot to `flush hosts` :)

Retry
YOHOO I’m in!

I’ve seen quite a few blogs about disabling name resolve in mysql with skip-name-resolve and granting privileges using IPs – something which would also have avoided the above (but still not found the root of the problem):

http://jeremy.zawodny.com/blog/archives/011421.html
http://www.mysqlperformanceblog.com/2008/05/31/dns-achilles-heel-mysql-installation/

Irony (from the Ancient Greek εἰρωνεία eirōneía, meaning hypocrisy, deception, or feigned ignorance) is a situation, literary technique or rhetorical device, in which there is an incongruity, discordance or unintended connection that goes beyond the most evident meaning. It is the expression of one’s meaning by using language that normally signifies the opposite.

Types of irony
Modern theories of rhetoric distinguish among verbal, dramatic and situational irony.
▪ Verbal irony is a disparity of expression and intention: when a speaker says one thing but means another, or when a literal meaning is contrary to its intended effect. An example of this is sarcasm.
▪ Dramatic irony is a disparity of expression and awareness: when words and actions possess a significance that the listener or audience understands, but the speaker or character does not.
▪ Situational irony is the disparity of intention and result: when the result of an action is contrary to the desired or expected effect. Likewise, cosmic irony is disparity between human desires and the harsh realities of the outside world (or the whims of the gods). By some definitions, situational irony and cosmic irony are not irony at all.


mysql> explain SELECT COUNT(ac.UID) FROM ACTIVE ac JOIN ALL a;
+----+-------------+-------+------+---------------+------+---------+------+---------+-------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+---------------+------+---------+------+---------+-------+
| 1 | SIMPLE | ac | ALL | NULL | NULL | NULL | NULL | 124426 | |
| 1 | SIMPLE | a | ALL | NULL | NULL | NULL | NULL | 7594256 | |
+----+-------------+-------+------+---------------+------+---------+------+---------+-------+
2 rows in set (0.01 sec)

mysql> SELECT COUNT(ac.UID) FROM ACTIVE ac JOIN ALL a ON ac.UID=a.UID;
+---------------+
| COUNT(ac.UID) |
+---------------+
| 17466 |
+---------------+
1 row in set (0.23 sec)

mysql> set @tot = (SELECT COUNT(ac.UID) FROM ACTIVE ac JOIN ALL a ON ac.UID=a.UID);

^CQuery aborted by Ctrl+C


Took more than 60seconds —- what the …..
Why did it take a long?


mysql> set @a=2;
Query OK, 0 rows affected (0.00 sec)

mysql> select @a;
+------+
| @a |
+------+
| 2 |
+------+
1 row in set (0.00 sec)

mysql> set @tot = (SELECT COUNT(*) FROM ACTIVE);
Query OK, 0 rows affected (0.13 sec)

mysql> select @tot;
+--------+
| @tot |
+--------+
| 124426 |
+--------+
1 row in set (0.00 sec)

mysql> set @tot = (SELECT COUNT(ac.UID) FROM ACTIVE ac);
Query OK, 0 rows affected (0.22 sec)

mysql> select @tot;
+--------+
| @tot |
+--------+
| 124426 |
+--------+
1 row in set (0.00 sec)

mysql> SELECT COUNT(ac.UID) FROM ACTIVE ac, ALL a;
+---------------+
| COUNT(ac.UID) |
+---------------+
| 944922897056 |
+---------------+
1 row in set (0.05 sec)

mysql> set @tot=(SELECT COUNT(ac.UID) FROM ACTIVE ac JOIN ALL a);
^CQuery aborted by Ctrl+C
ERROR 1317 (70100): Query execution was interrupted


Reason? …. the query is using the MySQL optimiser rather than the IB one! Why? good question (will have to ask IB devs though).

Work around, use ac temporary table to store the result and setting the variable to the result field, but it’s really ugly isn’t it?

will all manage to get into mysql given the way mysql authenticates users is against your username and client host from where you are connecting.

This verification is done versus the following columns in the mysql.user table, i.e., User,Host and Password columns.
An entry in the mysql.user table with the following values User=”, Host=’%’ will accept ANY user connecting from ANYWHERE in the world, thus disabling ANY security. Hence the reason for this blog post highlighting the importance of dropping such accounts, at least in all environments apart from dev.

Further information at:

http://dev.mysql.com/doc/refman/5.1/en/connection-access.html