Vlad Fedorkov

Performance consulting for MySQL and Sphinx

Tips and tricks while working with Production DBs

From time to time we have to work with live environments and production databases. For some of us this is day-to-day job. And most of the time cost of a mistake is way higher than expected improvement especially on the databases. Because issue on the database side will affect everything else.

I heard enough war stories about ruined productions and can imagine well enough speed of DROP DATABASE command replicating across the cluster. So I’m scared to make changes in production. The more loss expected if things go wrong the more I’m going to be scared planning every change. But I still love to make improvements so the only question is how to make them safer.

This post is not intended to be a guide or best practices on how to avoid issues at all, it’s more invitation to discussion that started between me and @randomsurfer in twitter on how to avoid production failures. It was hard for me to fit to 150 characters so I’m switching to more comfortable environment.

So here is a few practices that I use while working with remote servers:

  • Do not use rm -rf with leading wildcards. Ever. If you need to delete all subdirectories go with rm -rf dir1 dir1 dir2 etc. I’m also trying to avoid even trailing slashes near to rm -rf to be honest, just because.
  • Commandline history is convenient but could be dangerous if your connection is slow (it happens often when you work coast to coast or trans-Atlantic) so you could press enter on wrong command. To avoid this I’m hitting <space> at the end of command line before hit enter and wait for terminal response to make sure that command appearing on the command line will not scroll away.
  • I do my best to not make changes while sick. Sometimes it’s hard to reschedule a change, but most of the time it’s worth to move it to later time (even few hours could help sometimes) or hand off to someone else than put customer’s environment to risk.
  • Work with a single server at a time. If you have several open SSH windows with different server there is possibility to mess up the window and issue right command in wrong place. Especially when one window is a master and another is slave – same DB names, same tables, easy to mess up. If you still need to work on different hosts at the same time – change terminal’s background. I usually use red color (128,0,0) for master SSH window.

If you have your own tricks on how to not run DROP DATABASE in wrong place (yeah, I know pt-slave-delay helps in this case, but still!) please feel free to share them!

Thank you!

Category: General, Operations
  • Daniël van Eeden says:

    Setting a good prompt in MySQL helps

    September 23, 2013 at 8:07 pm
    • Rolf says:

      I second this. Hostname should be in every prompt, db or os. Its also important to have very distinctive names per environment. Avoid naming servers after transformers, star wars or other characters, this does not scale well. Datacenter, descriptive name or abbr, incremental number. Preface then name with agile or qa for lower environments, even higher.

      Don’t ever drop a database until it’s empty.

      September 24, 2013 at 12:33 am
  • Todor says:

    chattr +ai filename
    I am always protecting the most sensitive files and directories with chattr in order to avoid accidents.

    September 24, 2013 at 7:19 am
  • Mark Grennan says:

    Be very careful with GUI tools.

    For some insane resin the all put the Create table right next to the Truncate and Drop table. I would put these a few menus away from each other.

    September 25, 2013 at 4:27 pm
  • Clyde Crimes says:

    It’s recommended that children and young people have their own green room separate from the adult green room, cast trailers and crew areas. This is gives them a place away from the working environment to relax during rest periods and is also a safeguarding procedure. Plus it’s helps to have any and all children in one place with their parents or chaperones, where they will be safe and happy, and where you can findthem when it’s time to bring them to make up, costume or set. You should ensure the space is child-friendly and that no equipment, hot water urns or any other potentially dangerous items are kept in the same place.

    October 20, 2019 at 1:01 pm
  • EOTechzkb says:

    Testaru. Best known

    December 11, 2020 at 4:28 pm
  • Drywalladr says:

    “Julia’s Garland” (fr. Guirlande de Julie)

    December 26, 2020 at 2:35 pm
  • Squierkug says:

    Europe, and in Ancient Russia

    January 10, 2021 at 12:29 am
  • Artisancwi says:

    “Julia’s Garland” (fr. Guirlande de Julie)

    January 14, 2021 at 10:05 am
  • Flukeguy says:

    (palimpsests). In the XIII-XV centuries in

    January 19, 2021 at 2:14 pm
  • Garminzmxq says:

    Since manuscripts are subject to deterioration

    February 15, 2021 at 4:29 pm
  • raiden says:

    My name is Raiden. And I am a professional academic writer with many years of experience in writing.

    My goal is to solve problems related to writing. And I have been doing it for many years. I have been with several groups as a volunteer and have assisted in many ways.
    My love for writing has no end. It is like the air we breathe, something I cherish with all my being. I am a passionate writer who started at an early age.
    I’m happy that I`ve already sold several copies of my works in different countries like Canada and China and others too numerous to mention.
    I also work in an organization that provides assistance to many clients from different parts of the world. People always come to me because I work no matter how difficult their projects are. I help them to save money, because I feel fulfilled when people come to me for professional help.

    Ghost Writer – Raiden – studentenkamerbrugge.com Team

    June 24, 2021 at 10:05 pm
  • Furrionwqj says:

    European glory, and even after

    July 28, 2021 at 8:06 pm
  • elouise says:

    Hey guys, my name is Elouise Wagstaff!

    I`m an academic writer and I`m going to change your lifes onсe and for all
    Writing has been my passion for a long time and now I can`t imagine my life without it.
    Most of my poems were sold throughout Canada, USA, Old England and even Australia. Also I`m working with services that help people to save their nerves.
    People ask me “Mr, Elouise, I need your professional help” and I always accept the request, `cause I know, that only I can solve all their problems!

    Academic Writer – Elouise – http://www.euradiamultilateral.com Corps

    September 1, 2021 at 8:31 am

Your email address will not be published. Required fields are marked *

*