keepconf-manual.txt

                         ----------------------
                                 Keepconf
                          Configuration Manual
                         ----------------------
                             version 2.2.0
                                Ricardo F.
                             16/January/2021



============
| Abstract |
============

Keepconf is a agentless tool for backup and track files from remote hosts. It
uses rsync and git for the purpouse. Indeed, it can:
  - process lists of files/folders for retrieve it from hosts
  - limit size of the files fetched
  - store content in different defined directories
  - trigger hooks for execute whatever after/before fetching/committing
  - use a local or remote git repository
  - report the final status for monitoring the results in csv format


=================
| Prerequisites |
=================


Almost these version of the following software:

   python 3 with this modules (included in python core by default):
      sys, optparse, os, glob, time, string, re, configparser, tempfile, subprocess, distutils
   rsync 3.0
   git 1.7

A ssh-keys connection to the target hosts. Password prompt is not supported.
** NOTE: Please, be sure that the configured user can connect to the target
host without any prompt.

A configuration file, recomended in "/etc/keepconf/keepconf.cfg"

Some free space for store the remote files.

In the remote hosts, only rsync and a ssh client.



============================
| Command line  parameters |
============================

There are only a few command line options :

    -h, --help            show this help message and exit
    -f PATH, --conf=PATH  configuration file/folder path
    -i, --init            initialize git repostory only
    -n, --nocommit        avoid commit changes
    -c, --commitonly      only commit changes
    -m TEXT, --message=TEXT
                          commit message
    -s, --silent          silent fetching output
    -x, --xilent          silent commit output
    -v, --verbose         verbose output
    -V, --version         show version


-h, --help
  Print a quick reference of the command line parameters.

  Examples:
         keepconf -h
         keepconf --help


-f <PATH>, --conf=<PATH>
  Define the configuration file or path to use. If it is a folder, each file
  inside it with .cfg extension will be parsed, wich is good if you want to separate
  hosts by different enviroments with different properties.

  Examples:
         keepconf -f /etc/keepconf/  # (Default)(Each .cfg file will be parsed)
         keepconf -f /etc/keepconf/keepconf.cfg 
         keepconf -f /etc/keepconf/keepconf-client1.cfg
         keepconf --conf /home/user1/myservers/production.cfg


-i, --init
  Initialize repository only, wich will be located at "d_dest" variable defined
  in the configuration file. Depends if "repo" variable is defined with local or
  remote repository, a fresh init or a clone will be done. If is not defined
  or is "False", any action will be done. With multiple configuration files, 
  choose one of them with -f option.

  Examples:
         keepconf -i (For one con figuration file)
         keepconf -i -f /etc/keepconf/keepconf.cfg
         keepconf -i -f /etc/keepconf/keepconf-client1.cfg
         keepconf -init (For one con figuration file)


-n, --nocommit
  Avoid committing changes, executing "pre_commit" and "post_commit" scripts.
  Only fetching files and executing "pre_get" and "post_get" will be done.

  Examples:
         keepconf -n
         keepconf --nocommit


-c, --commitonly
  Avoid fetching files, executing "pre_get" and "post_get". Only commit
  operation and execution of "pre_commit" and "post_commit" will be done.

  Examples:
         keepconf -c
         keepconf --commitonly


-m, --message
  Substitute the default commit message with the text defined inside this
  option.

  Examples:
         keepconf -m 'Fixing issue with repo'
         keepconf --message 'Deleting old files from old hosts'


-s, --silent
  When rsync fetchs files, some information about their running state are
  printed trought the terminal. For silent the rsync stdout and stderr
  use this option.

  Examples:
         keepconf -s
         keepconf --silent

-x, --xilent
  When git is execute, some information about their behaviour are printed
  trought the terminal. For silent git stdout and stderr use this option.

  Examples:
         keepconf -x
         keepconf --xilent


-v, --verbose
  Print a lot of information about the internals of keepconf. Good for
  debugging how it gets the variables from files and the backgroud behaviour.

  Examples:
         keepconf -v
         keepconf --verbose


-V, --version:
  Print version number and exit.

  Examples:
         keepconf -V
         keepconf --version



======================
| Configuration file |
======================

Inside the configuration file, anything following a sharp ('#') or semicolon
(';') is ignored. Space caracter is allowed, but if is possible, avoid it.
All names of sections and variables must be only in lowercase.
Take care of the tabs and spaces at the beginning of each line, dont add any
of them.

[main]
  Section for general variables. The following are available inside here:

  d_dest=
    Directory for store the remote files and locate the repository. It is a
    requirement for the basic behaviour of keepconf. Rsync put there all
    files fetched inside a folder with the name of each host. A valid and
    empty dir is needed. If not exist, the initialize option creates it
    automatically. Require a full path definition.

    Example:
           d_dest = /var/keepconf/hosts/

  d_monitor=
    Directory for store the .csv monitor files. The name of the files is
    the same that the configuration files except the extension.

    Examples:
           d_monitor = /var/tmp/keepconf/
           d_monitor = False  # (Default)


[hosts]
  Section for define hosts to fetch. Two options are available here, list_hosts
  variable content and/or list of hosts alone:

  list_hosts=
    Path to file, or comma separated list of files, with the list of hosts to
    fetch. The content of all files is merged together.

    Example:
          list_hosts = /etc/keepconf/hosts/hosts_prod.lst
          list_hosts = /etc/mydir/hosts_prod.lst, /etc/mydir/hosts_dev.lst

  (list of hosts)
    Define hostsnames or IPs directly, one per line. The format is the same as
    described in "list_hosts and list_files format". If list_host is already
    present, all will be combined.

    Example:
          webserver
          ftpserver
          192.168.1.50


[files]
  Section for define lists of files to fetch. Two options are available here,
  list_files variable content and/or list of paths alone:

  list_files=
    Path to file, or comma separated list of files, with the list of files to
    fetch. The content of all files is merged together.

    Example:
          list_files = /etc/keepconf/files/files_default.lst
          list_files = /etc/mydir/files_default.lst, /etc/mydir/files_env1.lst

  (list of paths)
    Define paths for fetch directly, one per line. The format is the same as
    described in "list_hosts and list_files format". If list_files is already
    present, all will be combined.

    Example:
          /etc/resolv.conf
          /opt/app1/*
          !/opt/app1/logs


[sync]
  Section for fetching variables. The following are available inside here:

  fetch=
    Enable or disable fetching behaviour.

    Examples:
           fetch = True  # (Default)
           fetch = False

  pre_get=
    Directory for store the files that will be executed before fetching the
    files from the remote hosts.
    See "External scripts" section for more information.

    Examples:
           pre_get = /etc/keepconf/pre-get.d/
           pre_get = False  # (Default)

  post_get=
    Directory for store the files that will be executed after fetching the
    files from remote hosts.
    See "External scripts" section for more information.

    Examples:
           post_get = /etc/keepconf/post-get.d/
           post_get = False  # (Default)

  max_size=
    Size limit for files to fetch. It must be a number followed by the
    initial of the size name.

    Examples:
           max_size = 25M  # (Default)
           max_size = 100K
           max_size = 1024B

  rsync_user=
    User for connect to remote hosts via ssh.

    Examples:
           rsync_user = backup  # (Default)
           rsync_user = root

  rsync_key=
    Ssh key for use in the rsync connection to the hosts. Define a path to the
    key. If none is defined or this variable is not used, the default ssh
    behaviour for the user who execute keepconf will be used.

    Examples:
           rsync_key = None  # (Default)
           rsync_key = /home/backup/.ssh/servers-prod-key.rsa

  rsync_port=
    Ssh port for use in the rsync connection tho the remote hosts.

    Examples:
           rsync_port = 22  # (Default)
           rsync_port = 2222

  rsync_opt=
    Common options for pass to rsync command. They must be a compact list
    like the example. By default:
      archive, recursive, preserve hard links, verbose, compress, copy links

    Examples:
           rsync_opt = arHvzL  # (Default)
           rsync_opt = ar
           rsync_opt = v

[vcs]
  Section for version control system variables. The following are available
  inside here:

  commit=
    Enable or disable commit behaviour.

    Examples:
           commit = True  # (Default)
           commit = False

  pre_commit=
    Directory for store the files that will be executed before committing
    files located inside "d_dest" directory. See "External scripts" section
    for more information.

    Examples:
           pre_commit = /etc/keepconf/pre-commit.d/
           pre_commit = False  # (Default)

  post_commit=
    Directory for store the files that will be executed after committing
    files located inside "d_dest" directory. See "External scripts" section
    for more information.

    Examples:
           post_commit = /etc/keepconf/pre-commit.d/
           post_commit = False  # (Default)

  repo=
    Type of git repository:
      "False" disable commit behaviour.
      "local" local repository located inside "d_dest" directory.
      "ssh://user@host/rute/to/repo" remote repository cloned into "d_dest".

    Examples:
           repo = False
           repo = "local"
           repo = "ssh://user@host/rute/to/repo"


====================================
| list_hosts and list_files format |
====================================

Each file listed inside "list_hosts" and "list_files" variables (and hosts and
files listed directly inside [hosts] and [files] sections)  will have the 
following properties.

Anything following a sharp ('#') a semicolon (';') or a comma (',') is ignored.
Space character is allowed, but if is possible, avoid it use. Take care of the
tabs and spaces at the beginning of each line, dont add any of them.

For the files, any extension is valid, .lst .txt etc...

"list_hosts"
  One host per line.

  Complete dns hostnames and IP are allowed.

  For willcard match its only available the use of ranges defined into
  brackets [1-25]. A third argument can be passed to the range brackets
  [1-25-2] for change the step.

  Examples:
         host1.prod
         192.168.5.12
         db.local.company
         web-tier[1-10].prod.company

"list_files"
  One path per line.

  Folder must ends with slash '/' character.

  It's available the use of asterisk '*', interrogation '?' and ranges defined
  into brackets [1-25] for willcard match.

  It's possible to include, exclude or force paths:
      Include - Normal line path.
      Exclude - A '!' character must be before the path.
      Force - Based on the behaviour of rsync and how keepconf interact with
      it, exclude rules have precedence over includes. For force the include
      of a path inside a exclude match, add a '&' character before the path.

  Examples:
         /etc/  # Include all /etc/ but not folders inside it
         /etc/* # Include folders inside it recursively
         /etc/apache2/sites-enabled/*.conf  # Include all files ended in conf
         !/etc/cron.d/  # Exclude cron.d folder
         &/etc/cron.d/myfile  # Force include inside a exclude rule match
         /home/user1/conf[1-15]*.conf  # Get a range of files ended in conf


====================
| External Scripts |
====================

Variables "pre_get", "post_get", "pre_commit" and "post_commit" defines a
folder wich all files into it with execution permissions are executed. But
executed in diferent stages of the flow:

pre-get -> fetch files -> post-get -> pre-commit -> commit -> post-commit

Two arguments are passed to each file when its executed, first is a comma 
separated list with the hosts applicable and second is "d_dest" variable.

The definition of the directory must be a complete route starting from the
root path.

Are a good point for include diferent particular tasks. For example:

- Locate a script inside "pre_get" or "post_get" for backup a swich, firewall,
appliance, whatever  device... wich need a particular process. Put the files
fetched inside a folder into "d_dest" directory and it will be tracked with
git as all other content.

- Execute keepconf with no commit option '-n' and put a script inside
"post_get" for call an other version control system, like subversion.

- Put a script inside "post_commit" directory for do some task after each
execution. Sending an email as example.

- Delete especial and binary files for avoid commit them. Actually this script
it is included into "pre-commit.d" directory at the github repo.

Full path definition is required for this variables.

Note: Example bash scripts shipped with the proyect and located in post-get.d/ 
and pre-commit.d/ only work on Linux systems.


===========
| Logging |
===========

For get a log of the behaviour, redirect sdout and/or stderr to a file in
execution time:

  Examples:
        keepconf > /var/log/keepconf.log  # Redirect stdout
        keepconf 2> /var/log/keepconf.log  # Redirect stderr
        keepconf &> /var/log/keepconf.log  # Redirect stdout and stderr


======================
| Schedule execution |
======================

For automate the backup process with a log, this line can be added to the cron
daemon:
  With traditional syslog:
    15 06 * * * keepconf &> /var/log/keepconf.log

  With systemd journalctl:
    15 06 * * * systemd-cat keepconf

And for rotate the log generated, add a file called "keepconf" to
/etc/logrotate.d with this content:

/var/log/keepconf.log {
  rotate 12
  monthly
  compress
  missingok
}


===========================
| Using multiple ssh keys |
===========================

Rely on ".ssh/config" to connect to different hosts with different ssh keys.

As example:
# cat .ssh/authorized_keys

Host web1.rack1
  IdentityFile /root/.ssh/id_rsa_001

Host *.rack2
  IdentityFile /root/.ssh/id_rsa_002


================
| Some Toughts |
================

For avoid problems with section definition, range brackets are not allowed
at the beginning of a line.

By default, paths ended in '/' fetch all content inside it only, if ends with
'/*' fetch all content inside it recursively.

The monitor csv files can be converted with a script to Prometheus (textfile
collector), there is a Grafana dashboard for graph the values, look for
both into the grafana-dashboard folder in this repo.

Inside the rsync call, the arguments StrictHostKeyChecking=no and 
UserKnownHostsFile=/dev/null are passed for avoid annoying startup issues
when hosts are new and any ssh connection is done before (avoid prompt for 
accept ssh fingerprint).

If you found any bugs, please report it with all information as you can
(a verbose output its helpfull). Thanks.