Simplest solution is to avoid this undesired situation by adding locking to your backup script and prevent script to start second time when it’s already running.

Here is working sample. You will need to replace “sleep 10″ string with actual backup script call:

#!/bin/bash

LOCK_NAME="/tmp/my.lock"
if [[ -e $LOCK_NAME ]] ; then
        echo "re-entry, exiting"
        exit 1
fi

### Placing lock file
touch $LOCK_NAME
echo -n "Started..."

### Performing required work
sleep 10

### Removing lock
rm -f $LOCK_NAME

echo "Done."

It works perfectly most of the times. Problem is that you could still theoretically run two scripts at the same time so both will pass lock file checks and will be running together. To avoid that you would need to place unique lock file just before check and make sure no other processes did the same.

Here is improved version:

#!/bin/bash

UNIQSTR=$$
LOCK_PREFIX="/tmp/my.lock."
LOCK_NAME="$LOCK_PREFIX$UNIQSTR"

### Placing lock file
touch $LOCK_NAME
if [[ -e $LOCK_NAME && `ls -la $LOCK_PREFIX* | wc -l` == 1 ]] ; then
        echo -n "Started..."
        ### Performing required work
        sleep 10
        ### Removing lock
        rm -f $LOCK_NAME
        echo "Done."
else

### another process is running, removing lock
        echo "re-entry, exiting"
        rm -f $LOCK_NAME
        exit 1
fi

Now even if you managed to run two scripts at the same time only one script could actually start backup. In very rare situation both scripts will refuse to start (because of two lock files existing at the same time) but you could catch this issue by simply monitoring script exit code. Anyway – as soon you receive backup exit code different than zero it’s time to review your backup structure and make sure it works as desired.

Please note – when you terminate this script manually you will also need to remove lock file as well so script will pass check on startup. You could also use this script for any periodic tasks you have like Sphinx indexing, merging or index consistency checking.

For your convenience this script is available for download directly or using wget:

wget http://astellar.com/downloads/backup-wrapper.sh

You could also find more about MySQL backup solutions here.

Keep your data safe and have a nice day!

First meeting organized by NYPHP meetup on Tuesday, September 25th at IBM, 590 Madison Avenue, New York. I’ll be speaking about search services in cloud environment and distributed search tips and tricks. Event is free, please RSVP.

One week later on October 1st, I’ll be doing tutorial about MySQL and Sphinx “Full-text based services with Sphinx and MySQL” at greatest MySQL event in East Cost, Percona Live NY. Use “FlashSale” code to get exceptional 25% discount (valid until Sept 23rd).

Looking forward to meet you in New York City!

Checking index consistency

One of the most important functions of indextool is checking index consistency. You will need to have sphinx config file and index files.
/path/to/indextool -c sphinx.conf --check my_sphinx_index

This will perform checking of my_sphinx_index for consistency between document list, hit list, positions and other internal sphinx index structures. Please note that indextool is only checking disk indexes (starting from 2.0.2 it could also check on-disk part of Real-Time indexes, but not a memory part). Usual output for healthy index looks likes this:


using config file 'sphinx.conf'...
checking index 'my_sphinx_index'...
checking dictionary...
checking data...
checking rows...
checking attribute blocks index...
checking kill-list...
check passed, 4.4 sec elapsed


indextool doesn’t fix issues itself it’s only telling you if index okay or not. In case of troubles you will need to rebuild broken index. Usually you could do that with indexer [--rotate] my_sphinx_index where –rotate is used to rebuild index on the fly, while searchd is running.

Getting number of documents from index

indextool is providing you an option to reverse engineer index to see internal structures and settings along with global information like number of documents stored, number of bits per document identifier (32 or 64), tokenizer, morphology type and some other index setting. Usage:
indextool --dumpheader my_sphinx_index.sph

Shortened results would look like this:
dumping header for index 'my_sphinx_index'...
dumping header file 'my_sphinx_index.sph'...
version: 23
idbits: 64
docinfo: extern
fields: 5
field 0: page
field 1: title
field 2: description
field 3: tags
attrs: 23
attr 0: user_id, uint, bitoff 0
attr 1: url_crc32, uint, bitoff 32
[...]
attr 12: deleted, uint, bitoff 384
attr 13: private, uint, bitoff 416
attr 14: external, uint, bitoff 448
attr 15: enabled, uint, bitoff 480
attr 16: lastactivity, timestamp, bitoff 512
attr 17: url, ordinal, bitoff 544
attr 18: has_picture, bool, bitoff 576
[...]
total-documents: 3438767
total-bytes: 21164723870
min-prefix-len: 0
min-infix-len: 0
exact-words: 0
html-strip: 1
[...]


Lots of interesting internal index information as you could see. Besides total-documents and total-bytes you can find names and internal types for all non full-text attributes with their sizes. bitoff (offset) field for last attr record will give you an idea of attributes memory consumption. For full-text fields you can find names, text processing settings like prefix/infix indexing, stemming length, stopwords, zones support and some other settings.

Are there documents in index that matched my keyword?

Easy! Even when daemon is not running:

# indextool -c sphinx.conf --dumphitlist <my_sphinx_index> <keyword>

or in some more convenient way, just to get the number of docs:

# indextool -c sphinx.conf --dumphitlist <my_sphinx_index> <keyword> | grep docs | wc -l

You could find indextool in any Sphinx distribution since version 0.9.9, it’s usually located at the same directory as searchd.

php: error while loading shared libraries: libmysqlclient.so.16: cannot open shared object file: No such file or directory

while using Percona MySQL Server 5.5 just go ahead and install Percona-Server-shared-compat package from Percona Repo:

yum install Percona-Server-shared-compat

sort_buffer_size is a per-connection variable and do not belongs to any specific storage engine. It doesn’t matter do you use MyISAM or InnoDB – MySQL will allocate sort_buffer_size for every sort (required most of the times for ORDER BY and GROUP BY queries) so increasing it’s value might help speeding up those queries however I would not recommend to change it from the default value unless you are absolutely sure about all the drawbacks. Value for out-of-the-box MySQL-5.1.41 installation on Ubuntu is 2Mb and it’s recommended to keep it that way.

On the other side myisam_sort_buffer_size used by MyISAM to perform index sorting on relatively rare table-wide modifications like ALTER/REPAIR TABLE. Stock value is 8Mb so if you are using MyISAM tables intensively (please refer to my other post to see how to know your tables type) I would recommend to set it to some higher value close or even more than key_buffer_size but still small enough to keep it in memory and prevent MySQL from swapping.

Looking forward to meet you at FOSDEM 2012!

MySQL server tuning and query slowness hunting are always step by step process and without knowing all the data (SHOW GLOBAL VARIABLES, SHOW GLOBAL STATUS, SHOW TABLE STATUS LIKE ‘tablename’, EXPLAIN details for slow query is just some of the required information) it would be generally a blind guess. But there are still few things which is related to newly installed MySQL server.

If you are using stock MySQL you might need to check memory pool size which MySQL used to load index data to avoid slow IO requests and increase queries speed. Connect to MySQL and fire two queries:

SHOW VARIABLES LIKE 'key_buffer_size';
SHOW VARIABLES LIKE 'innodb_buffer_pool_size';

Default values are very small. If you see something less than 128MB (8Mb for older versions) that might be an issue. key_buffer_size used by MyISAM tables while innodb_buffer_pool_size required for InnoDB tables. I would recommend to set one of them to some relatively big value (generally from 25% to 80% of total memory on the box depends on the case).

How do you know which table type you are using? While connected to mysql please type:

SHOW CREATE TABLE 'table_name';

and find ENGINE=MyISAM or ENGINE=InnoDB on the last line. Increase key_buffer_size in my.cnf (/etc/my.cnf or /etc/mysql/my.cnf) if you see MyISAM or innodb_buffer_pool_size for InnoDB and restart MySQL.

Have fun!

Say we have table called <my_table> with `title` and `content` text fields. In MySQL you have to fire query like this:

SELECT * FROM <my_table> WHERE MATCH(`title`,`content`) AGAINST ('I love Sphinx');

Let’s see how could we do the same query with Sphinx.

There are two steps to run the Sphinx as a MySQL FT replacement. First you will need to pull all needed text data from MySQL to Sphinx. For that you have to configure source and index definition in Sphinx search config. Second step is to run indexer program which will connect to MySQL and fetch all the desired data. Then fire up the search daemon which will serve queries. Simplified Sphinx configuration example is below:

You need to let Sphinx know where to look for the data (source configuration):

source my_source
{
    type      = mysql
    sql_host  = localhost
    sql_user  = sphinx
    sql_pass  = ********
    sql_db    = <my_database_name>
    sql_port  = 3306
    sql_query = SELECT id, title, description FROM <my_table>
}

Please note that id field in sql_query. This field MUST be positive integer and have to be unique across all the documents in collection. Auto_incremented integer primary key from MySQL table with work in this case like a charm.

Now we need to tell Sphinx where to store all that data and configure index:

index my_first_sphinx_index
{
    source        = my_source
    path          = <path_to_sphinx_home>/var/index1
    docinfo       = extern
    charset_type  = utf-8
}

That’s it. Let’s add few more required sections to complete configuration:

Indexer settings:

indexer
{
    mem_limit    = 256M
    write_buffer = 8M
}

and daemon configuration:

searchd
{
    listen                  = 9312
    listen                  = 9306:mysql41
    pid_file                = <sphinx_path>/var/searchd.pid
    max_matches             = 1000
}

Put blocks above to <sphinx_path>/etc/sphinx.conf file which will be your main Sphinx configuration file.
Please also make sure that <sphinx_path>/var directory is writable for user you planning to run sphinx daemon.

Now we have to perform initial indexing by running indexer binary

<sphinx_path>/bin/indexer my_first_sphinx_index -c <sphinx_path>/etc/sphinx.conf

indexer my_first_sphinx_index tells indexer to create index called my_first_sphinx_index described in sphinx config. To create all the indexes at once (if you have two or more) run indexer –all -c <sphinx_path>/etc/sphinx.conf

Now you have to run the search daemon apparently called searchd

<sphinx_path>/bin/searchd -c <sphinx_path>/etc/sphinx.conf

Now Sphinx should be able to answer queries.

Fire up mysql client and connect to brand new Sphinx installation:

$ mysql -h0 -P 9306
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 2.0.3-id64-dev (rel20-r3043)

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

Please note server version. It is Sphinx! Now you can hire all power of Sphinx full-text query language!

mysql> SELECT * FROM my_first_sphinx_index WHERE
MATCH('I love Sphinx') LIMIT 0,5; SHOW META;
+---------+--------+
| id      | weight |
+---------+--------+
| 7637682 |   2652 |
| 6598265 |   2612 |
| 6941386 |   2612 |
| 6913297 |   2584 |
| 7139957 |   1667 |
+---------+--------+
5 rows in set (0.01 sec)

+---------------+--------+
| Variable_name | Value  |
+---------------+--------+
| total         | 51     |
| total_found   | 51     |
| time          | 0.013  |
| keyword[0]    | love   |
| docs[0]       | 227990 |
| hits[0]       | 472541 |
| keyword[1]    | sphinx |
| docs[1]       | 114    |
| hits[1]       | 178    |
+---------------+--------+
9 rows in set (0.00 sec)
mysql>

Please note – Sphinx returns document IDs, not a document content, so you need to query MySQL to fetch additional fields: SELECT * FROM <my_table> WHERE id IN (7637682, 6598265, …, 7139957)

Config above is fully working but very simple and provided as an example. You can download it from this website directly or using wget:

wget http://astellar.com/downloads/sphinx.conf

Another way to create initial Sphinx configuration is to adopt Sphinx configuration sample called sphinx-min.conf.dist bundled to the Sphinx RPM and Deb packets.

You could also learn more about Sphinx tips and tricks from my talks on various conferences and meetups, read blog posts about Sphinx and follow me on twitter.

If you are looking for help with Sphinx installation and integration, troubleshooting and fine tuning please contact me for a quote with your problem description.

Enjoy!

P.S. If you found this article useful please share it!

Enjoy!

Say if you are searching for “when is jane’s birthday” you are actually looking to find documents with “jane’s birthday”, and you don’t really care about lot’s of documents (blog posts, news articles, etc) with only “when” and “is” inside.

Remove those high frequency words from search index is usually smart move and ages ago I’ve created two stopword file samples which I’m using by now.

stopwords.txt is a top 100 most frequent words in my blog post collection while stopwords-500.txt as you might expect is a 500 top frequent words. They are old, but not yet included in sphinx distribution so I would suggest to start with stopwords.txt and add it using stopwords option to your sphinx config file as below:

     stopwords = /path/to/stopwords.txt

You could also download stopword files using wget:

wget http://astellar.com/downloads/stopwords.txt
wget http://astellar.com/downloads/stopwords-500.txt

Learn more about Sphinx tips and tricks from my talks on various conferences and meetups, read blog posts about Sphinx and follow me on twitter.

If you are looking for help with Sphinx installation and integration, troubleshooting and fine tuning please contact me for a quote with your problem description.

Enjoy!

P.S. If you found this article useful please share it!