pointers for working with rclone and environment variables

[GuitarBilly]

I have an existing backup to OneDrive via rclone which runs daily via a window batch file triggered from the scheduler. All is fine.

Now I try to put it in npbackup but am struggling to get it to work and would appreciate some hints. Bear with me as I write it down from the end user perspective.

Here is how I constructed the npbackup Configuration.

Destination tab

repo URI = rclone:onedrive:MyPreciousData\

Backup tab

Additional backup only parameters =
-o rclone.program=%RCLONEPROG% -o rclone.args=%RCLONEARGS% --files-from C:\path\to\my\restic.includes..txt

Environment variables tab

RCLONEPROG = C:\path\to\rclone.exe
RCLONEARGS = "serve restic --stdio --b2-hard-delete --user-agent ISV|rclone.org|rclone/v1.65.2"
RESTICPROG=C:\LocalData\tools\Restic\restic.exe

Pre/Post exec tab

%RESTICPROG% cache --cleanup

This gets accepted without any errors.
However when I change to this repo in the main window it gives an error.

Listing snapshots of repo OneDrive_via_rclone
Repository is not initialized or accessible
Runner took 0.091304 seconds for snapshots

Backend is not ready to perform operation snapshots. Repo maybe inaccessible or not initialized. You can try to run a backup to initialize the repository:
Fatal: unable to open repository at rclone:_(o_O)_hidden_by_npbackup: cmd.Start: exec: "rclone": executable file not found in %PATH%

So the error is clear, and rclone is not in my%PATH%. I added it and removed the -o rclone.program option.

Here are the (multiple) errors of the run with rclone:

Running backup of ['<backup_paths_redacted>'] to repo OneDrive_via_rclone
Using VSS snapshot to backup
Runner took 16.516281 seconds for backup


Pre-execution of command %RESTICPROG% cache --cleanup failed with:
'__ooOO' is not recognized as an internal or external command,
operable program or batch file.

Pre-execution of command %RESTICPROG% cache --cleanup failed with:
'__ooOO' is not recognized as an internal or external command,
operable program or batch file.

Pre-execution of command %RESTICPROG% cache --cleanup failed with:
'__ooOO' is not recognized as an internal or external command,
operable program or batch file.

Pre-execution of command %RESTICPROG% cache --cleanup failed with:
'__ooOO' is not recognized as an internal or external command,
operable program or batch file.

Pre-execution of command %RESTICPROG% cache --cleanup failed with:
'__ooOO' is not recognized as an internal or external command,
operable program or batch file.

Pre-execution of command %RESTICPROG% cache --cleanup failed with:
'__ooOO' is not recognized as an internal or external command,
operable program or batch file.

Pre-execution of command %RESTICPROG% cache --cleanup failed with:
'__ooOO' is not recognized as an internal or external command,
operable program or batch file.

Pre-execution of command %RESTICPROG% cache --cleanup failed with:
'__ooOO' is not recognized as an internal or external command,
operable program or batch file.

Pre-execution of command %RESTICPROG% cache --cleanup failed with:
'__ooOO' is not recognized as an internal or external command,
operable program or batch file.

Pre-execution of command %RESTICPROG% cache --cleanup failed with:
'__ooOO' is not recognized as an internal or external command,
operable program or batch file.

Pre-execution of command %RESTICPROG% cache --cleanup failed with:
'__ooOO' is not recognized as an internal or external command,
operable program or batch file.

Pre-execution of command %RESTICPROG% cache --cleanup failed with:
'__ooOO' is not recognized as an internal or external command,
operable program or batch file.

Pre-execution of command %RESTICPROG% cache --cleanup failed with:
'__ooOO' is not recognized as an internal or external command,
operable program or batch file.

Pre-execution of command %RESTICPROG% cache --cleanup failed with:
'__ooOO' is not recognized as an internal or external command,
operable program or batch file.

Pre-execution of command %RESTICPROG% cache --cleanup failed with:
'__ooOO' is not recognized as an internal or external command,
operable program or batch file.

Pre-execution of command %RESTICPROG% cache --cleanup failed with:
'__ooOO' is not recognized as an internal or external command,
operable program or batch file.
rclone: Error: unknown command "%RCLONEARGS%" for "rclone"

Pre-execution of command %RESTICPROG% cache --cleanup failed with:
'__ooOO' is not recognized as an internal or external command,
operable program or batch file.
rclone: Error: unknown command "%RCLONEARGS%" for "rclone"
rclone: Run 'rclone --help' for usage.
rclone: You could use 'rclone selfupdate' to get latest features.
rclone: 
rclone: 2025/02/11 23:28:55 Fatal error: unknown command "%RCLONEARGS%" for "rclone"
Fatal: unable to open repository at rclone:onedrive:MyPreciousData: error talking HTTP to rclone: exit status 1
Backend failed operation:
rclone: Error: unknown command "%RCLONEARGS%" for "rclone"
rclone: Run 'rclone --help' for usage.
rclone: You could use 'rclone selfupdate' to get latest features.
rclone: 
rclone: 2025/02/11 23:28:55 Fatal error: unknown command "%RCLONEARGS%" for "rclone"
Fatal: unable to open repository at rclone:onedrive:MyPreciousData: error talking HTTP to rclone: exit status 1
Backup is smaller than configured minmium backup size
Operation finished with failure

I quickly fixed the many pre-exec repetitions by enabling this:

and fixed the smaller than minimum backup size by configuring that to zero.

remaining questions:

1. how to pass the rclone arguments, is that possible?
2. how to work with the environmental variables, the substitution does not work?

EDIT1: i got bare basics to work by:
- clearing "Additional backup only parameters" ( no rclone.args, no --from-files)
- manually adding the include files and folders via "Add manually"
- clearing Pre/Post exec command

EDIT2: removed the --from-files question, answered/fixed in https://github.com/netinvent/npbackup/issues/121

[deajan]

I've explicitly disabled shell execution in order to avoid some security issues.
So far, I've expanded users and variables for envrionment variables. As of your comment I added additional parameters and additional backup parameters expand users and vars here

There's one question that bugs me: You're using RESTICPROG variable to point to a local restic executable, whereas npbackup ships with an internal restic executable (which can be overrided btw if needed).
I've made NPBACKUP_BACKEND_BINARY environment variable available here so you don't have to rely on an externa binary anymore. This env variable is available to all additional commands, env variables, pre and post scripts.

[GuitarBilly]

@deajan let me reply to your feedback.

why did I use RESTICPROG? Simple, since for this machine (Windows10) i already have a .BAT file which does all restic backup work, launched via a "cmd.exe start ..." launched off the Task Scheduler.
The .BAT uses these set variables to do all calls to restic for the backup and cache cleanup etc. Here is a stripped down version of it:

set RESTICPROG=C:\LocalData\tools\Restic\restic.exe
set RCLONEPROG=C:/LocalData/tools/rclone/rclone.exe
set ResticRclone=%RESTICPROG% -o rclone.program=%RCLONEPROG% -o rclone.args="serve restic --stdio --b2-hard-delete --user-agent ISV|rclone.org|rclone/v1.65.2"
set REPO=rclone:onedrive:MyPreciousData
::   prevent "found old cache directory" messages
ECHO ...1/4  cleanup old cache if any
%RESTICPROG% cache --cleanup
::   if H: mapped back it up as well
ECHO ...2/4  backup H:\ data
IF EXIST "H:\nulI" (
    %ResticRclone% --verbose --repo %REPO% backup --tag "H_drive" --files-from c:\_RESTIC\scripts_windows\W10.restic.include.H_drive.txt
) ELSE (
    ECHO  ^>^>^>  H:\ drive not mapped. skipping.  ^<^<^<
)
ECHO ...3/4  backup TC settings
%ResticRclone% --verbose --repo %REPO% backup --tag "TC" --files-from c:\_RESTIC\scripts_windows\W10.restic.include.TC.txt
ECHO.
ECHO ...4/4  check random repo data subset 1%%
ECHO run countdown for 10 seconds to safely release any prior repository lock.
timeout /T 10
%ResticRclone% --repo %REPO% --verbose check --read-data-subset 1%%

Looking for a place in npbackup to do "step 1/4 cache cleanup" before the actual backup i came to the Pre/Post exec tab. I filled in there as if it would be executed at a CMD prompt, which is apparantly not how it works.
Can you give an example on how to run pre/post actions?

SET UP RCLONE

I am only a basic user with rclone and OneDrive, I googled my way into this working setup.

1. prepare rclone for one's OneDrive account via these steps:

https://rclone.org/onedrive/

2. optimize rclone to prevent throttling and hang-ups:

https://forum.restic.net/t/possible-fix-for-onedrive-sharepoint-throttling/7025
and
https://learn.microsoft.com/en-us/sharepoint/dev/general-development/how-to-avoid-getting-throttled-or-blocked-in-sharepoint-online#how-to-decorate-your-http-traffic-to-avoid-throttling
which leads to the execution in the above .BAT example with -o rclone.program and -o rclone.args

RCLONE IN NPBACKUP

minimum setup that can complete a backup without error but also without above optimizations.

1. make sure rclone.exe can be found in the PATH enironment variable.

2. prepare rclone for one's OneDrive account via these steps:

https://rclone.org/onedrive/

3. define rclone OneDrive target in the backup repo URI/path as rclone:onedrive:MyPreciousData

p.s. I have now noticed a few times (two or three times) that the "Current permissions" in the configuration are reset to Full permissions and no Manager password. Will try to see if I can get that better reproduced. v3.0.0-rc13.

update: added npbackup rclone step1 PATH env var.

[deajan]

Thanks for the insight about rclone.

As for your initial restic cleaning step, npbackup is designed to not divulgate any sensible data (restic password/repository/encrypted envrionment variables) to any scripts, so your scripts will know about environment variables, but not the ones above.

The best I can think of is running npbackup with --raw command which allows basically to run any command for restic, without compromising security.
So you'd call npbackup twice:

npbackup-cli -c /my/configfile.conf --raw "cache --cleanup"
npbackup-cli -c /my/configfile.conf --backup

For the GUI, you'd run the same except you add --run-as-cli in order to use the same GUI binary:

npbackup-gui -c /my/configfile.conf --run-as-cli --raw "cache --cleanup"
npbackup-gui -c /my/configfile.conf --run-as-cli --backup

Does this solve step 1 ?

[deajan]

p.s. I have now noticed a few times (two or three times) that the "Current permissions" in the configuration are reset to Full permissions and no Manager password. Will try to see if I can get that better reproduced. v3.0.0-rc13.

Reproduced and fixed. REALLY thanks for that one.

[GuitarBilly]

running restic cache --cleanup is not a dealbreaker if I would not run it, I only do it on one windows laptop.
This command would not require password or repo information nor encrypted variables. Regardless, your suggestion for
npbackup-cli -c /my/configfile.conf --raw "cache --cleanup" works for me:

2025-02-14 19:20:58,358 :: INFO :: Running raw command: cache --cleanup
no old cache dirs found
2025-02-14 19:21:24,463 :: INFO :: Successfully run raw command:
no old cache dirs found

NPBACKUP CONFIGURATION

Can you give some example on how to use these two configuration fields?
tab Backup -

tab Environment variables -

RCLONE OPTIONS

for rclone options i was looking at using environmental variables:
https://rclone.org/docs/#environment-variables
https://rclone.org/docs/#precedence

I also found back how i came to the other rclone options, proudly found elsewhere:
https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html#other-services-via-rclone
Now i have defined RCLONE_VERBOSE with value 2 to print verbose information. This should help verify if the user agent is set correctly to prevent throttling.

my takeaways:
defining a rclone backup already makes restic do some magic under the hood i.e.
rclone: 2025/02/14 21:37:46 DEBUG : rclone: Version "v1.65.2" starting with parameters ["rclone" "serve" "restic" "--stdio" "--b2-hard-delete" "onedrive:MyPreciousData"]
but user agent seems not active even though set separately via environmental variable:

rclone: 2025/02/14 21:39:07 DEBUG : Too many requests. Trying again in 268 seconds.
rclone: 2025/02/14 21:39:07 DEBUG : pacer: low level retry 1/10 (error activityLimitReached: throttledRequest: The request has been throttled)
rclone: 2025/02/14 21:39:07 DEBUG : pacer: Rate limited, increasing sleep to 4m28s
rclone: 2025/02/14 21:39:07 DEBUG : pacer: Reducing sleep to 3m21s
rclone: 2025/02/14 21:39:07 DEBUG : Too many requests. Trying again in 267 seconds.

This then repeats many times. I will compare this to my .BAT, also run with RCLONE_VERBOSE=2.
There the user agent string is passed but I also see throttling happening.
Interesting!, and now enough experimenting for today.

[GuitarBilly]

quick reply on the throttling: in my .BAT script it is mostly during check --read-data-subset 1%.
GOOD NEWS: this seems to be handled gracefully by rclone.

I did not succeed to have 'npbackup with rclone' accept or pass on the rclone --user-agent argument. Then the next step would be to have npbackup replicate my .BAT backup without rclone --user-agent set, and confirm rclone still handles it gracefully. Currently npbackup does not work with RCLONE_VERBOSE set to 2, I logged #132 for that

[deajan]

This command would not require password or repo information nor encrypted variables

You can run %NPBACKUP_BACKEND_BINARY% cache --cleanup in the prescript section, that should do the job.
For additional parameters, you can just add whatever restic takes:
Example:

Enough npbackup for me today :)
Have a look at RC14 release which contains all the improvements/fixes you suggested.
Happy to get your feedback.