keepconf-hands-on.txt

                         ----------------------
                                Keepconf
                                Hands On
                         ----------------------
                               Ricardo F.
                            16/January/2021




1.- Abstract
2.- Ssh configuration, for any host to backup
3.- Ssh configuration, for backup storage host 
4.- Keepconf configuration, for backup storage host
5.- Keepconf case, adding new host
6.- Keepconf case, removing old host
7.- Keepconf case, using two configuration files




================
| 1.- Abstract |
================

This document is a hands on guide for starting from scratch, covering the base
ssh and rsync configuration and various keepconf scenarios.

The following hosts are used for the examples:

backups.example.com - Backup storage  
web1.example.com    - Webserver
web2.example.com    - Webserver



=========================-------------------------
| 2.- Ssh configuration | For any host to backup | 
=========================-------------------------

+ Prerequisites:

 - ssh server
 - rsync > 3.0
 - a user, for example "backman", for connect into.


+ "backman" a normal user:

backman@web1:~$ id backman 
uid=1001(backman) gid=1001(backman) groups=1001(backman)


+ Generate the ssh key for the user "backman" without passphrase:

backman@web1:~$ ssh-keygen -t rsa -P ''  -f ~/.ssh/id_rsa
Generating public/private rsa key pair.
Your identification has been saved in /home/backman/.ssh/id_rsa.
Your public key has been saved in /home/backman/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:EW/PLNgH/ZmD3PxOn84W8AxhpJkPJzyF5DVk+oQ2/ws backman@lnode3
The key's randomart image is:
+---[RSA 2048]----+
|        . ..+B   |
|         oooXo.  |
|        . +#oo.  |
|         =.*@B o |
|        S o B+%  |
|           o  .* |
|             E .+|
|              o++|
|              o=o|
+----[SHA256]-----+


+ Authorize the key for connect into it without password:

backman@web1:~$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys


+ Ensure that ssh is running and rsync installed.




=========================--------------------------
| 3.- Ssh configuration | For backup storage host | 
=========================--------------------------

Prerequisites:

 - ssh client
 - rsync > 3.0
 - git > 1.7
 - python 3
 - keepconf, latest release.
 - a user, in this case "root" will launch the backup.


+ Copy the private key from the user "backman" at "web1.example.com" into .ssh/ folder:

root@backups:~# cd .ssh/
root@backups:~/.ssh# scp root@web1.example.com:/home/backman/.ssh/id_rsa id_rsa-backman
root@web1.example.com's password: 
id_rsa                                                                                      100%  668     0.7KB/s   00:00    
root@backups:~/.ssh# chmod 700 id_rsa-backman


+ Configure ssh for connect to "web1" with the user "backman" and its key.

root@backups:~/.ssh# cat >> ~/.ssh/config << EOF
Host web1.example.com
  IdentityFile /root/.ssh/id_rsa-backman
EOF


+ Test it!, don't continue until you can connect to "web1.example.com" with the
user "backman" without password:
(Only the first time you will need to ask "yes" to the question)

root@backups:~/.ssh# ssh backman@web1.example.com
The authenticity of host 'web1.example.com (192.168.1.5)' can't be established.
ECDSA key fingerprint is 97:59:3f:36:ae:0e:5e:67:5d:4c:cc:91:db:b9:76:c3.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'web1.example.com,192.168.1.5' (ECDSA) to the list of known hosts.
backman@web1:~$ 




==============================--------------------------
| 4.- Keepconf configuration | For backup storage host | 
==============================--------------------------


+ Install keepconf:

root@backups:~# bash <(curl -Ls https://raw.githubusercontent.com/rfmoz/keepconf/master/keepconf-install.sh)


+ Be sure that git is configured correctly with your name an email. Maybe you have already set them:

root@backups:~# git config user.email || git config --global user.email "you@example.com"
root@backups:~# git config user.name || git config --global user.name "Your Name"


+ Set the default backup user for keepconf, uncomment the rsync_user line and change the user to "backman":

root@backups:~# vi /etc/keepconf/keepconf.cfg
 # Default global user for connect to other hosts
 rsync_user = backman


+ Remove the example conf files, you won't need it:

root@backups:~# rm /etc/keepconf/hosts/*


+ Create a file called "/etc/keepconf/hosts/web-servers.lst" and add the following example lines:

web1.example.com


+ Create a file called "/etc/keepconf/files/web-files.lst" and add the following example lines:

/etc/passwd
/var/lib/dpkg/status


+ Set both files on the variables inside "/etc/keepconf/keepconf.cfg" and remove 
'# Or / And sample .. list' content next to them too:

list_hosts = /etc/keepconf/hosts/web-servers.lst

list_files = /etc/keepconf/files/web-files.lst



+ Run it! (In this example, with "-s" for reduce output): 

root@backups:~# keepconf -s
- - -
- -
- Processing: /etc/keepconf/keepconf.cfg
- Initialize local repository on: /var/keepconf/hosts/
Initialized empty Git repository in /var/keepconf/hosts/.git/
- -
- Fetching start
- Reading hosts list: /etc/keepconf/hosts/web-servers.lst
- Reading files list: /etc/keepconf/files/web-files.lst
- Finding exec files in: /etc/keepconf/pre-get.d/
- Fetching host: web1.example.com
- Finding exec files in: /etc/keepconf/post-get.d/
- Processing: /etc/keepconf/post-get.d/01-remove-binary
  Script -> Removing binary files
- Fetching done
- -
- Committing start
- Finding files in /etc/keepconf/pre-commit.d/
- Processing: /etc/keepconf/pre-commit.d/01-add-git-ignore
  Script -> Adding files to .gitignore
- Committing into local repository
[master (root-commit) f8cc8aa] Keepconf commit at 13:27:21 - Tuesday/August/2015 Hosts ok: [1] web1.example.com Hosts bad: [0]
 Committer: root <root@backups.example.com>
 3 files changed, 8359 insertions(+)
 create mode 100644 .gitignore
 create mode 100644 web1.example.com/etc/passwd
 create mode 100644 web1.example.com/var/lib/dpkg/status
- Finding files in /etc/keepconf/post-commit.d/
- Committing done
- 
- Hosts rsync'ed: [1] web1.example.com
- Hosts not rsync'ed: [0] 
- Fetched in: 0.44 seconds
- Committed in: 0.04 seconds
- Completed in: 0.48 seconds
- Monitor file: /var/tmp/keepconf/keepconf.csv


+ Now, the files from "web1.example.com" are yet inside "backups.example.com" located where 
keepconf "d_dest" variable define, as you can see:

root@backups:~# cd /var/keepconf/hosts/web1.example.com/
root@backups:/var/keepconf/hosts/web1.example.com# tree
.
├── etc
│   └── passwd
└── var
    └── lib
        └── dpkg
            └── status

4 directories, 2 files


+ And tracked into a git repository:

root@backups:/var/keepconf/hosts/web1.example.com# git log
commit f8cc8aa01cbe9d95b1ed0da8d7b8dcf465ab7d80
Author: root <root@backups.example.com>
Date:   Tue Aug 4 13:27:21 2015 +0200

    Keepconf commit at 13:27:21 - Tuesday/August/2015
    Hosts ok: [1] web1.example.com
    Hosts bad: [0]


+ Finally, add a cron entry for automate the process:

root@backups:~# vi /etc/cron.d/keepconf
 # /etc/cron.d/keepconf: crontab entrie for keepconf update 
 MAILTO=root
 15 5    * * *   root    if [ -x /usr/bin/keepconf ]; then /usr/bin/keepconf > /dev/null; fi





=====================------------------
| 5.- Keepconf case | Adding new host | 
=====================------------------

+ Configure "web2" as described in section "2.- Ssh configuration, For any host to backup."

+ Add the new host to the configuration file, in this case "web2" to "/etc/keepconf/hosts/web-servers.lst":

  Previous state:
    web1.example.com

  Posterior state, option with full name:
    web1.example.com
    web2.example.com

  Posterior state, option using wildcard:
    web[1-2].example.com


+ Execute keepconf normally, and check that "web2" is fetched:

root@backups:~# keepconf


+ If all went ok, inside "d_dest" directory there are a new folder for the new server with their files:

root@backups:~# cd /var/keepconf/hosts/
root@backups:/var/keepconf/hosts/# ls
web1.example.com  web2.example.com


+ And the corresponding commit with the new host:

root@backups:/var/keepconf/hosts/# git log
commit  dff0b7add12856bcfa8ae00d0bc573a841235f37
Author: root <root@backups.example.com>
Date:   Tue Aug 5 12:22:01 2015 +0200

    Keepconf commit at 12:22:01 - Thursday/August/2015
    Hosts ok: [2] web1.example.com, web2.example.com
    Hosts bad: [0]




=====================--------------------
| 6.- Keepconf case | Removing old host | 
=====================--------------------


+ Remove the host from the configuration file, in this case "web2" from "/etc/keepconf/hosts/web-servers.lst": 

  Previous state:
    web1.example.com
    web2.example.com

  Posterior state:
    web1.example.com


+ Remove the fetched files from the repo directory "/var/keepconf/hosts/":

root@backups:/var/keepconf/hosts/# git rm -r web2.example.com


+ Execute keepconf with commit only and an informative message:

root@backups:/var/keepconf/hosts/# keepconf -c -m "Deleting old host"'


+ Wich produce the following commit:

root@backups:/var/keepconf/hosts/# git log
commit df0ce9a83f3d7e509dfcaae3895318ff7db61b41
Author: root <root@backups.example.com>
Date:   Thu Aug 6 10:19:50 2015 +0200

    Keepconf: Deleting old host




=====================--------------------------------
| 7.- Keepconf case | Using two configuration files | 
=====================--------------------------------


Is it possible to have more than one configuration file, for example, each one for one enviroment.

+ Copy default configuration file to the same folder with an other name:

root@backups:~# cp /etc/keepconf/keepconf.cfg /etc/keepconf/keepconf-dev.cfg

+ Change whatever you consider for the new enviroment. Avoid mix folders and repositories.

+ Run "keepconf", both files will be parsed and process using their configured values.