Secured MySQL InnoDB Cluster with Certificate creation using OpenSSL

This is a demo tutorial to show how we can create InnoDB Cluster with newly installed Certificate and having X509 certificate verification via MySQL Router connection.   

Recorded Video

The full process is recorded on Youtube - showing creating InnoDB Cluster with newly installed CA, Server Certificates.  The Router creation is configured with SSL Server certificate from the same CA certificate across Server nodes.   The creation of User (create user my509user identified by '....' require X509) using X509 certificate PASSTHROUGH verification via Router connection.

https://www.youtube.com/watch?v=w1xgpjw0VTw

Environment

The following environment was tested 

Oracle Linux Server release 8.6

MySQL Server 8.0.31

MySQL Shell 8.0.31

MySQL Router 8.0.31

Github Script

The github script provides the steps to guide thru Installation of 3 nodes 

https://github.com/ivanxma/mylab/tree/main/13-InnoDBCluster/99-SSL-cert-IC

To configure the node1/node2/node3 hostname under ./comm.sh   (Change the hostname based on your environment)

```

export HOST1=workshop20

export HOST2=workshop22

export HOST3=workshop23

```

Background

The creation of InnoDB Cluster creates Internal User (mysql_innodb_cluster_<server_id>@'%') 

Here is an example :

mysql> select user,host from mysql.user;

+--------------------------+-------------+

| user                     | host        |

+--------------------------+-------------+

| gradmin                  | %           |

| mysql_innodb_cluster_101 | %           |

| mysql_innodb_cluster_201 | %           |

| mysql_innodb_cluster_301 | %           |

| mysql.infoschema         | localhost   |

| mysql.session            | localhost   |

| mysql.sys                | localhost   |

| root                     | localhost   |

+--------------------------+-------------+

8 rows in set (0.00 sec)

Note : the RENAME USER does not work.

 (e.g.  mysql > rename user mysql_innodb_cluster_101@'%' to mysql_innodb_cluster_101@'10.0.%' )  

"%" as host may can be restricted to specific IP subnet via replicationAllowedHost option with MySQL Shell Admin API.   

https://dev.mysql.com/doc/mysql-shell/8.0/en/create-cluster.html#mysql-innodb-cluster-replicationallowedhost

As an example to restrict user creation with Host from subnet 192.0.2.0/24:

mysql-js> dba.createCluster('testCluster', {replicationAllowedHost:'192.0.2.0/24'})

Group Replication with communication stack "MYSQL" goes through the standard MySQL port (for example 3306.  The encrypted communication / recovery channel is established with SSL certificate.    

https://dev.mysql.com/doc/refman/8.0/en/group-replication-secure-socket-layer-support-ssl.html

group_replication_ssl_mode  is configured as REQUIRED to ensure secured SSL between nodes.

REQUIRED

Establish a secure connection if the server supports secure connections.

group_replication_ssl_mode can also be configured as VERIFY_CA to ensure certificate being used and verified.

VERIFY_CA

Like REQUIRED, but additionally verify the server TLS certificate against the configured Certificate Authority (CA) certificates.

The following sections describe the steps to setup the VERIFY_CA option and creation of certificates using OpenSSL command.

1. On each node to initialize MySQL Data Directory [ Script ] 

2. Create CA certificate on Node1 and using the same CA certificate to create server-cert.pem for each server 

Refer to CA creation [ Script ]

Refer to Server certificate creation [ Script ] 

3. Startup the server with unique $SERVER_ID on each HOST [ Script ] 

4.On each host, configure Group Replication Admin User [ Script ] 

5. Create 3 nodes Innodb Cluster with MySQL Shell with specific options to secure the SSL connection

 [ Create Cluster Script ] [ Add Instances Script ]

6. Bootstrap MySQL Router [ Script ]

7. Create Router Certificate with same CA certificate and Configure the mysqlrouter.conf  

[ Create Router Script ] [ Config PASSTHROUGH ]

8. Start Router [ Script ] 

9 Finally, creating a user with X509 and Login via MySQL Router

    Login to Primary Server with 'root'

mysql > create user my509user@'%' identified by 'my509pass' REQUIRE X509;

     Alternatively, user creation can follow more constraint with 

REQUIRE SUBJECT "/...."

 REQUIRE ISSUER "/..."

    For example (Check with the client certificate for the ISSUER or SUBJECT content  [ openssl x509 -text -in <certificate file> ] )

    example 1 : mysql > create user my509user@'%' identified by 'my509pass' REQUIRE ISSUER "/O=MySQL/CN=MySQL" ;

    example 2 : mysql > create user my509user@'%' identified by 'my509pass' REQUIRE SUBJECT "/CN=hostname";

    example 3 : mysql > create user my509user@'%' identified by 'my509pass' REQUIRE SUBJECT "/CN=hostname" AND ISSUER "/O=MySQL/CN=MySQL;

 

    Login using certificate via the Router  For example : (Given that the 'client-key.pem' 'client-cert.pem' is created on each host)

mysql -umy509user -pmy509pass -h127.0.0.1 -P6446 --ssl-mode=VERIFY_CA --ssl-ca=~/data/3310/ca.pem --ssl-cert=~/data/3310/client-cert.pem --ssl-key=~/data/3310/client-key.pem 

References

https://github.com/ivanxma/mylab/tree/main/13-InnoDBCluster/99-SSL-cert-IC

https://dev.mysql.com/doc/mysql-shell/8.0/en/mysql-innodb-cluster-setting-options.html

https://dev.mysql.com/doc/mysql-shell/8.0/en/create-cluster.html#mysql-innodb-cluster-replicationallowedhost

MySQL Performance Schema to identify SQL Statements which the execution duration exceeds average timing by 30%

 

Statement Digest in MySQL 

https://dev.mysql.com/doc/refman/8.0/en/performance-schema-statement-digests.html

The parser is also used by the STATEMENT_DIGEST_TEXT() and STATEMENT_DIGEST() functions, which applications can call to compute a normalized statement digest and a digest hash value, respectively, from an SQL statement.

events_statement_summary_by_digest in Performance Schema contains information

+-----------------------------+-----------------+------+-----+---------+-------+

| Field                       | Type            | Null | Key | Default | Extra |

+-----------------------------+-----------------+------+-----+---------+-------+

| SCHEMA_NAME                 | varchar(64)     | YES  | MUL | NULL    |       |

| DIGEST                      | varchar(64)     | YES  |     | NULL    |       |

| DIGEST_TEXT                 | longtext        | YES  |     | NULL    |       |

| COUNT_STAR                  | bigint unsigned | NO   |     | NULL    |       |

| SUM_TIMER_WAIT              | bigint unsigned | NO   |     | NULL    |       |

| MIN_TIMER_WAIT              | bigint unsigned | NO   |     | NULL    |       |

| AVG_TIMER_WAIT              | bigint unsigned | NO   |     | NULL    |       |

| MAX_TIMER_WAIT              | bigint unsigned | NO   |     | NULL    |       |

| SUM_LOCK_TIME               | bigint unsigned | NO   |     | NULL    |       |

| SUM_ERRORS                  | bigint unsigned | NO   |     | NULL    |       |

| SUM_WARNINGS                | bigint unsigned | NO   |     | NULL    |       |

| SUM_ROWS_AFFECTED           | bigint unsigned | NO   |     | NULL    |       |

| SUM_ROWS_SENT               | bigint unsigned | NO   |     | NULL    |       |

| SUM_ROWS_EXAMINED           | bigint unsigned | NO   |     | NULL    |       |

| SUM_CREATED_TMP_DISK_TABLES | bigint unsigned | NO   |     | NULL    |       |

| SUM_CREATED_TMP_TABLES      | bigint unsigned | NO   |     | NULL    |       |

| SUM_SELECT_FULL_JOIN        | bigint unsigned | NO   |     | NULL    |       |

| SUM_SELECT_FULL_RANGE_JOIN  | bigint unsigned | NO   |     | NULL    |       |

| SUM_SELECT_RANGE            | bigint unsigned | NO   |     | NULL    |       |

| SUM_SELECT_RANGE_CHECK      | bigint unsigned | NO   |     | NULL    |       |

| SUM_SELECT_SCAN             | bigint unsigned | NO   |     | NULL    |       |

| SUM_SORT_MERGE_PASSES       | bigint unsigned | NO   |     | NULL    |       |

| SUM_SORT_RANGE              | bigint unsigned | NO   |     | NULL    |       |

| SUM_SORT_ROWS               | bigint unsigned | NO   |     | NULL    |       |

| SUM_SORT_SCAN               | bigint unsigned | NO   |     | NULL    |       |

| SUM_NO_INDEX_USED           | bigint unsigned | NO   |     | NULL    |       |

| SUM_NO_GOOD_INDEX_USED      | bigint unsigned | NO   |     | NULL    |       |

| SUM_CPU_TIME                | bigint unsigned | NO   |     | NULL    |       |

| COUNT_SECONDARY             | bigint unsigned | NO   |     | NULL    |       |

| FIRST_SEEN                  | timestamp(6)    | NO   |     | NULL    |       |

| LAST_SEEN                   | timestamp(6)    | NO   |     | NULL    |       |

| QUANTILE_95                 | bigint unsigned | NO   |     | NULL    |       |

| QUANTILE_99                 | bigint unsigned | NO   |     | NULL    |       |

| QUANTILE_999                | bigint unsigned | NO   |     | NULL    |       |

| QUERY_SAMPLE_TEXT           | longtext        | YES  |     | NULL    |       |

| QUERY_SAMPLE_SEEN           | timestamp(6)    | NO   |     | NULL    |       |

| QUERY_SAMPLE_TIMER_WAIT     | bigint unsigned | NO   |     | NULL    |       |

+-----------------------------+-----------------+------+-----+---------+-------+

The events_statements_history_long[events_statement_current / events_statement_history] contains information about a SQL statement and Digest together with the execution time.  With referencing the events_statement_history_long, we can track the execution timing compared to the average execution time from table "events_statements_summary_by_digest".

1. To enable 'performance_schema" to collect events_statements_history_long, we need to enable the consumer (for the *history_long).  The default value is OFF which means the events_statements_history_long is empty without any data.   Enabling the events_statements_history_long allows more SQL statements to be captured and compared with the average execution time.

mysql > update setup_consumers set enabled='YES' where name = 'events_statements_history_long';

2. Retrieving the list of SQL Statements where the digest is the same and the execution time is having 30% over the average timing.

mysql > select a.sql_text,  a.digest, (a.timer_end - a.timer_start) duration, b.avg_timer_wait 

          > from events_statements_history_long a, events_statements_summary_by_digest b 

          > where a.digest = b.digest and b.avg_timer_wait > (a.timer_end - a.timer_start) *1.3

Performance schema / sys schema contains valuable information which we can use them to track / tune our system.  It can be the SQL.  It can be storage or files.  

You can share more with others about how you can reuse the performance_schema tables to improve your application.

Enjoy reading!

China Encryption（国密算法）with MySQL

China Encryption Standards 

There are SM1,SM2, SM3, SM4, SM7, SM9, ...etc with China encryption standards.  With international encryption standards, we can leverage encryption libraries from publicly available source(s). For example, one of the most common and popular use encryption library is OpenSSL. 

This tutorial is written to provide steps with one of the "SSL" library having SMx (China Encryption Library) namely BabaSSL with MySQL.  It does not serve as a purpose of the only way adopting China Encryption Standards with MySQL. Applying compatible / Latest OpenSSL library from China Encryption vendor(s)/source(s) may possibly work in similar way as documented in this tutorial.

Environment (for the tutorial sharing) 

1. Compute Resource (VM) 

2. Operating System - Compute VM is provisioned with Oracle Linux 8 

3. MySQL community with 8.0.xis installed via public yum repository 

4. BabaSSL 8.3 [ BabaSSL 8.3.2-dev ] is used. It is based on OpenSSL 1.1.1h dated 22 Sep 2020

Introduction 

MySQL leverages OpenSSL library to provide ciphers with TLS encryption over communication channel.  To allow switching the OpenSSL library with China Standards and MySQL can dynamically use the underlying library feature to enforce the communication using China Encryption Standard.

Steps 

1. Provision VM and OS/packages update 

2. Download BabaSSL 8.3.2 stable source build for compilation 

3. Compile BabaSSL and Install BabaSSL 

4. Install MySQL 8.0.30 (from yum repository) 

5. Configure TLS with SMx for MySQL 

6. Change system service for mysqld with alternate BabaSSL library path 

7. Reload and restart mysqld service 

8. MySQL (mysql) client with BabaSSL library over TLS with SMx connection 

Provision VM and OS / Packages update

Compute Instance (VM) is provisioned with Oracle Linux 8.  Once the VM is provisioned and it is ready for connection, login to the shell terminal and apply update and package updates

Login from terminal as  

```

ssh -i <privatekey>  opc@<public IP> 

# sudo yum update

# sudo yum install wget

```

Download BabaSSL 8.3.2 stable source build for compilation 

To download the 8.3 stable source zip file, execute the following commands and extract the zip file 

```

# wget https://github.com/Tongsuo-Project/Tongsuo/archive/refs/heads/8.3-stable.zip

# unzip 8.3-stable.zip

```

To compile the source, change directory to the unzipped folder 'Tongsuo-8.3-stable" and execute commands as follows :

```

# cd Tongsuo-8.3-stable

# mkdir bld

# ../config

# make

```

The installation will put the BabaSSL into /usr/local/bin and /usr/local/lib64 and corresponding default installation path(s).   

Note : The default installation does not replace any standard OS system own OpenSSL but it puts into /usr/local as alternate installation.

```

# sudo make install

```

Finally, make change to /etc/profile and append the following

```

# export LD_LIBRARY_PATH=/usr/local/lib64:$LD_LIBRARY_PATH

```

Exit the terminal and re-login to the VM 

Launch a SSH connection to the Compute VM with new library path appended (from updated /etc/profile).   

To check BabaSSL and SMx ciphers, execute the following commands and validate if BabaSSL is installed.

```

# openssl version

# openssl -v ciphers|grep SM

```

Output should be displayed as follows :

MySQL Router Restful API

MySQL Router with InnoDB Cluster allows transparent routing access for application to backend MySQL Servers.  

Restful API (Http access) feature was added to MySQL Router since 8.0.17.

This article is written to provide a tutorial with Router Restful API.   It is tested with MySQL 8.0.29 release.