[New feature] Client/Server model and pull backup

[creativeprojects]

Today, all the backup operations are initiated by the client (push).
What if you would want a backup server asking for files from the clients? (pull).

I'm opening this issue as a discussion around the client/server model: a resticprofile server could wait for commands to trigger a backup, via a simple REST API.

Is it a good idea?

[jkellerer]

That's funny, because before stepping over resticprofile, I was thinking that it would be nice to have a server to manage restic configuration and clients that securely connect to the server to apply it on endpoints. Combined with a management web UI (for admins) to control auth, clients and settings, and a self-service UI (for endpoints) to restore data from snapshots.
From my perspective this is a good idea, but it has to be done very carefully as it can lead to serious loss of information.

Some thoughts:

• This would simplify backup on many endpoints by creating generic profiles that apply to many hosts.
• A server could manage password-file and authorization against the repository (and itself) and bind this to clients so that access can also be revoked easily.
• In some cases (REST repositories) every client could get random auth-credentials that are valid just for one backup when using the server as reverse proxy in front of the repo.
• A client-server model would allow to simplify using --append-only mode of restic-rest-server to protect from ransomware (or misconfiguration). E.g. clients can only append but never delete. Retention & checks are handled by the server (that has full access).
• In the very long term web UIs could be added for viewing status, management and self-service (leveraging the REST-API exposed by the server).

Design considerations:

• Client & server communication must be implemented with very high security standards. E.g. TLS-1.3+ and exchange of public keys like in SSH or Wireguard.
• Client & server communication should be implemented by a separate executable resticprofile-service that dedicates only to this task (e.g. securely transferring profiles, passwords, status, logs and triggering actions).
• resticprofile should remain dedicated to one task of executing profiles only and not do client & server communication by itself.
• Profiles could be enhanced to support conditionals (e.g. only-if-host-is: [...], only-if-os-is: [...], only-if-path-exists: [...], etc.) that way they can be created generically.
• resticprofile show could be enhanced to create machine readable output (while evaluating conditionals) so that a resticprofile-service can create profiles that are dedicated to a certain endpoint.

[jkellerer]

While it is a side effect when backup schedules are controlled by a central unit, I'd like to add that push based scheduling allows to use backup queues per repository that control backup concurrency and limit load. E.g. a profile with a schedule of "Every day at 12:00" could run on a larger amount of endpoints when only some endpoints may run the backup concurrently.

This can even make sense with a hand-full of endpoints when the repo is something like a consumer NAS.
Current solution is to have different schedules for different endpoints.

[creativeprojects]

It looks like we do have similar ideas 😄

Client & server communication must be implemented with very high security standards. E.g. TLS-1.3+ and exchange of public keys like in SSH or Wireguard.

This one is tricky though:

• if we give the impression it's super secure, some users might just leave it publicly accessible, and that means we'll have to provide zero-day security fixes.
• or we can provide minimum security only, forcing the user to encapsulate in some sort of VPN. But then if the user network is compromised, it's also nice to know the hackers might not have easy access to the backup system.

[jkellerer]

I'd not strip down security just because someone could have the impression that it can be used securely without VPN. Personally I do not trust my own network since any desktop (or server) could get infected and attack others to some extent. If it can be done in a secure way and still work quite simple (e.g. public key exchange using a key that is not too long when string encoded), I'd use that but still mention that it is not a good idea to use it without VPN or similar.

[creativeprojects]

Winter is coming so I'll have a bit more time at the week-ends: I'll start preparing a design document 😄

[jkellerer]

Same here (regarding time :)). Wanted to create a quick prototype on HTTP(S)+Websockets+SSH (SSH via Websockets for identification, commands & logs, HTTP/S for everything else). Was thinking about the communication part a bit and while client-certs can be a solution to identify agents uniquely, using HTTPS only means that middle boxes or proxies may create issues (technical and data protection). Embedding SSH with websockets allows to use something proven that is secure even when outer transfer is only HTTP. Nevertheless my aim is just to test whether this works well without heavy externals dependencies. It should not mean that this has to be the way to go.

[creativeprojects]

Funny enough I was doing a quick PoC the other day: transparent AES encryption over HTTP. I'm not sure how usable it actually is, but it's very simple: https://github.com/creativeprojects/encryption

[creativeprojects]

Randomly, I just found this discussion 🤔 restic/restic#299

[jkellerer]

Thanks a lot for reference. My (somewhat biased) takeaways are: There is a demand for a pull based system and it may be sufficient to use a remote shell for this. While both client and backup-server must be protected its important that a client has minimal access to the backup so that malicious software cannot easily do harm. For the same reasons the repository should not require to be reachable in the same (e.g. public) network as where the client sits.

I'd add that it is also very crucial that the client has to be protected to avoid take over or dataloss so a too permissive remote shell or even a homegrown remote-fs doesn't sound like a good plan.

While it seems there is something going on @ restic/restic with regards to pulling from remote sources, I think for resticprofile it still is good to consider supporting pull based operation. This could be done with remote shell support instead of implementing a full client & server model which would be the much higher hanging fruit and harder to maintain.

[jkellerer]

As a quick wrap-up this is what I think would be required when we choose to support remote shells:

General

• resticprofile allows running commands in local or remote shells (likely SSH but not limited to).
• Profiles are enhanced with a list of run-targets selecting the shell (and target specific options).
• Commands that perform backup or restore must run per run-target.
• Commands that manage the repository can run locally (but may need information of a run-target as input, e.g. retention needs hostname)

Security

• If the remote shell offers port forwarding, it should be used for supported repositories (unless explicitly disabled).
• A remote shell should be restricted in what it can perform, e.g. SSH supports limiting the commands to execute and other aspects of the connection including restricting the IP address of the client (aka backup-server in a pull based system).

Usability

• To allow using restricted remote shells, resticprofile should be the only command that a remote shell needs to run directly. Technically this means resticprofile communicates with resticprofile on the other side (should to be secured, e.g. via pairing).
• Pairing between client and server should be supported and managed to make it simple but secure. E.g. for SSH this could mean resticprofile can print the entry for authorized_keys that the clients needs to use including all restrictions that should apply and it could manage known_hosts locally containing all the clients.
• We could consider whether resticprofile has some convenience features, e.g. auto-deployment of restic on supported platforms.

Under the assumption that the best match for a remote shell is SSH to begin with, the only disadvantages I see is supporting Windows and Desktops as in both cases SSH is uncommon.

[creativeprojects]

I think I like this version better than the original idea:

With this, no need to create a separate service, install it on a client and wait for requests. All we need is allowing SSH to the machine. There's openSSH included in Windows 10 now (maybe starting with pro I don't know, but I had it on my Windows laptop at my previous job)

We could also implement ephemeral keys: the server creates a new repository key before sending it to the client and destroys it after the backup is finished. Although if the attacker already has access to the client, it could potentially intercept the backup request and steal the key. Still, that's another level of protection to go through.

[creativeprojects]

Quick ideas: https://github.com/creativeprojects/resticprofile/wiki/Design

Are you able to update the wiki? (I never used that feature before)

[jkellerer]

I've not used it either. Looks like I can clone it but not push. Also have no buttons for editing.