Unable to write file: Function not implemented on ReFS file system since version 2.48.1

[0xfeeddeadbeef]

After I've updated my Git to v2.48.1.windows.1, the commands git init, git gc and git add (probably many more, I haven't tried) started failing in repositories that reside on ReFS formatted disks.

This problem was not present in previous release 2.47.1(2).

Example:

# edit global.json and try to git add:
> git add ./global.json

error: unable to write file .git/objects/b1/6c1afa0351d335de649b03647d10bdc8a4e023: Function not implemented
error: global.json: failed to insert into database
error: unable to index file 'global.json'
fatal: updating files failed

The same commands still work without problems on NTFS formatted disks. The same commands also work in previous 2.47.1(2) Git version on the same ReFS disks.

And no, disks are not damaged, I tried same commands on two different SSDs, and two different HDDs.

Additional info

OS version: Windows Server 2022 10.0.20348.3207
Some relevant .gitconfig entries:

[core]
    longPaths = true
    symlinks = true
    fscache = true
    commitGraph = true
[rerere]
    enabled = true
    autoUpdate = true
[fetch]
    prune = true
    writeCommitGraph = true
[pack]
    threads = 1
[gc]
    auto = 1
    writeCommitGraph = true
[maintenance]
    auto = true

[dscho]

I use a DevDrive (which is ReFS, I believe) on 10.0.26100.3194 and I do not get this error with git version 2.48.1.windows.1...

Is there maybe any way you can build Git and see what exact function call reports ENOSYS?

[jensver]

I have the same issue with version 2.48.1 on Windows Server 2022 (10.0.20348.3091) when using an ReFS disk.
I tested it on an NTFS disk on the same server, and it works fine. So, the issue occurs only with repositories on an ReFS disk.
For now, the solution was to roll back to version 2.47.1.

[dscho]

I have the same issue with version 2.48.1 on Windows Server 2022 (10.0.20348.3091) when using an ReFS disk.

@jensver so maybe you can build Git and single-step to find out which function call reports ENOSYS? That would probably help resolve the issue much quicker.

For now, the solution was to roll back to version 2.47.1.

If you want to be stuck with that version, that's a fine approach.

[jensver]

@jensver so maybe you can build Git and single-step to find out which function call reports ENOSYS? That would probably help resolve the issue much quicker.

I'm sorry, that's not really my area of expertise.

[dscho]

that's not really my area of expertise.

@jensver sorry to hear that.

[dhirschfeld]

I have a ReFS D drive on a Windows Server 2022 box and I'm observing the same:

Administrator in D:/refs-bug
❯ git --version
git version 2.48.1.windows.1

Administrator in D:/refs-bug
❯ git init
error: could not write config file D:/refs-bug/.git/config: Function not implemented
fatal: could not set 'core.repositoryformatversion' to '0'

If I downgrade git it works again:

Administrator in D:/refs-bug
❯ git --version
git version 2.47.1.windows.1

Administrator in D:/refs-bug
❯ git init
Initialized empty Git repository in D:/refs-bug/.git/

I happen to be using git provided by conda-forge but if others are also seeing it then it's likely a bug in the source, introduced after 2.47.1.

---

Edit:
Unfortunately, upgrading to the latest-and-greatest 2.49.0 didn't fix the problem:

Administrator in D:/refs-bug
❯ git --version
git version 2.49.0.windows.1

Administrator in D:/refs-bug
❯ git init
error: could not write config file D:/refs-bug/.git/config: Function not implemented
fatal: could not set 'core.repositoryformatversion' to '0'

[dscho]

@dhirschfeld Is there maybe any way you can build Git and see what exact function call reports ENOSYS?

[dhirschfeld]

Is there maybe any way you can build Git and see what exact function call reports ENOSYS?

I might be able to build it, but I'm not sure how I'd then go about seeing what function call reports ENOSYS. I'm mostly a Python dev and not really set up for any serious debugging in C 😰

[dscho]

@dhirschfeld unfortunately I cannot reproduce and am therefore unable to debug it.

It's not that big of a deal to debug C, really. I tried my best to describe it in https://gitforwindows.org/debugging-git.html. The gist of it is:

1. install the Git for Windows SDK
2. Create /usr/src/git/config.mak with the following contents to disable compiler optimization and ASLR:

DEVELOPER=1
ifndef NDEBUG
CFLAGS := $(filter-out -O2,$(CFLAGS))
ASLR_OPTION := -Wl,--dynamicbase
BASIC_LDFLAGS := $(filter-out $(ASLR_OPTION),$(BASIC_LDFLAGS))
endif

1. run make in /usr/src/git/.

After that, you can run Git's executables in GDB like so:

gdb --args ./git.exe -C /path/to/refs/drive init test-git-repo

Now, it is useful to ask the debugger to pause the program when the code path is entered that outputs an error message. The functions in Git that output error messages are error_builtin() and die_builtin(). So you can set the breakpoints

b error_builtin
b die_builtin

before calling run in gdb to stop execution at the appropriate time. Doing this will stop execution when the error_builtin() function is called, i.e. when the error message you reported would be shown. Once that is the case, you can obtain a back trace with bt. Example:

(gdb) bt
#0  setup_git_directory_gently_1 (nongit_ok=0x0) at setup.c:846
#1  0x000000000057dfd8 in setup_git_directory_gently (nongit_ok=0x0)
    at setup.c:937
#2  0x000000000057e211 in setup_git_directory () at setup.c:1014
#3  0x0000000000487bfc in cmd_rev_parse (argc=2, argv=0x1951a88, prefix=0x0)
    at builtin/rev-parse.c:589
#4  0x000000000040296a in run_builtin (p=0x5fb8e8 <commands+2184>, argc=2,
    argv=0x1951a88) at git.c:373
#5  0x0000000000402c8d in handle_builtin (argc=2, argv=0x1951a88) at git.c:579
#6  0x0000000000402e2a in run_argv (argcp=0x108fe00, argv=0x108fdb8)
    at git.c:637
#7  0x0000000000402fb9 in cmd_main (argc=2, argv=0x1951a88) at git.c:709
#8  0x000000000049f699 in mingw_main (argc=3, argv=0x1951a80)
    at common-main.c:40
#9  0x000000000049f64d in main (argc=3, argv=0x3230498) at common-main.c:25

This back trace shows which functions were called by which other functions. The inner-most code location is #0, which was called from #1, which was called from #2, etc. In the example above, the main() function called the mingw_main() function which in turn called the cmd_main() function, and so on, until eventually the setup_git_directory_gently() function was called, which then called the setup_git_directory_gently_1() function, where the execution was stopped to let you, the user, inspect the current state in the debugger.

You can inspect the source code at the current code location using the l command (or specify a line number or even a file and a line number like so: l git.c:709, or a function name, like so: l run_builtin).

You can navigate between the different stack frames using up and down (which is confusingly the opposite direction as in the list above).

Importantly in this instance, you can go up to the function that issues the error message, then inspect the surrounding code with l. This should already be very helpful in the investigation to figure what is going wrong, and if all you do to help this ticket is run with the debugging so far and then paste the output of gdb in this ticket, it should go a long way to fix this bug.

Please let us know how it goes!

[fusumayuki]

@dhirschfeld unfortunately I cannot reproduce and am therefore unable to debug it.

It's not that big of a deal to debug C, really. I tried my best to describe it in https://gitforwindows.org/debugging-git.html. The gist of it is:

1. install the Git for Windows SDK
2. Create /usr/src/git/config.mak with the following contents to disable compiler optimization and ASLR:

DEVELOPER=1
ifndef NDEBUG
CFLAGS := $(filter-out -O2,$(CFLAGS))
ASLR_OPTION := -Wl,--dynamicbase
BASIC_LDFLAGS := $(filter-out $(ASLR_OPTION),$(BASIC_LDFLAGS))
endif

1. run make in /usr/src/git/.

After that, you can run Git's executables in GDB like so:

gdb --args ./git.exe -C /path/to/refs/drive init test-git-repo

Now, it is useful to ask the debugger to pause the program when the code path is entered that outputs an error message. The functions in Git that output error messages are error_builtin() and die_builtin(). So you can set the breakpoints

b error_builtin
b die_builtin

before calling run in gdb to stop execution at the appropriate time. Doing this will stop execution when the error_builtin() function is called, i.e. when the error message you reported would be shown. Once that is the case, you can obtain a back trace with bt. Example:

(gdb) bt
#0  setup_git_directory_gently_1 (nongit_ok=0x0) at setup.c:846
#1  0x000000000057dfd8 in setup_git_directory_gently (nongit_ok=0x0)
    at setup.c:937
#2  0x000000000057e211 in setup_git_directory () at setup.c:1014
#3  0x0000000000487bfc in cmd_rev_parse (argc=2, argv=0x1951a88, prefix=0x0)
    at builtin/rev-parse.c:589
#4  0x000000000040296a in run_builtin (p=0x5fb8e8 <commands+2184>, argc=2,
    argv=0x1951a88) at git.c:373
#5  0x0000000000402c8d in handle_builtin (argc=2, argv=0x1951a88) at git.c:579
#6  0x0000000000402e2a in run_argv (argcp=0x108fe00, argv=0x108fdb8)
    at git.c:637
#7  0x0000000000402fb9 in cmd_main (argc=2, argv=0x1951a88) at git.c:709
#8  0x000000000049f699 in mingw_main (argc=3, argv=0x1951a80)
    at common-main.c:40
#9  0x000000000049f64d in main (argc=3, argv=0x3230498) at common-main.c:25

This back trace shows which functions were called by which other functions. The inner-most code location is #0, which was called from #1, which was called from #2, etc. In the example above, the main() function called the mingw_main() function which in turn called the cmd_main() function, and so on, until eventually the setup_git_directory_gently() function was called, which then called the setup_git_directory_gently_1() function, where the execution was stopped to let you, the user, inspect the current state in the debugger.

You can inspect the source code at the current code location using the l command (or specify a line number or even a file and a line number like so: l git.c:709, or a function name, like so: l run_builtin).

You can navigate between the different stack frames using up and down (which is confusingly the opposite direction as in the list above).

Importantly in this instance, you can go up to the function that issues the error message, then inspect the surrounding code with l. This should already be very helpful in the investigation to figure what is going wrong, and if all you do to help this ticket is run with the debugging so far and then paste the output of gdb in this ticket, it should go a long way to fix this bug.

Please let us know how it goes!

I've tried these steps and here's the output:

#0  error_builtin (err=0x5feb60 "could not write config file %s: Function not implemented", params=0x5fef98 "\320\f\aL\032\004") at usage.c:89
#1  0x00007ff66014ae59 in error_errno (fmt=0x7ff66023f778 <fsync_component_names+3160> "could not write config file %s") at usage.c:281
#2  0x00007ff65ff89f1b in repo_config_set_multivar_in_file_gently (r=0x7ff6603002a0 <the_repo>, config_filename=0x41a4c070cd0 "D:/Dev/Source/test-git-repo/.git/config",
    key=0x7ff66026c432 <__ac_HASH_UPPER+1154> "core.repositoryformatversion", value=0x41a4c070c80 "0", value_pattern=0x0, comment=0x0, flags=0) at config.c:3400
#3  0x00007ff65ff8a02e in repo_config_set_multivar_in_file (r=0x7ff6603002a0 <the_repo>, config_filename=0x41a4c070cd0 "D:/Dev/Source/test-git-repo/.git/config",
    key=0x7ff66026c432 <__ac_HASH_UPPER+1154> "core.repositoryformatversion", value=0x41a4c070c80 "0", value_pattern=0x0, flags=0) at config.c:3430
#4  0x00007ff65ff8a152 in repo_config_set_multivar (r=0x7ff6603002a0 <the_repo>, key=0x7ff66026c432 <__ac_HASH_UPPER+1154> "core.repositoryformatversion",
    value=0x41a4c070c80 "0", value_pattern=0x0, flags=0) at config.c:3457
#5  0x00007ff65ff893f5 in repo_config_set (r=0x7ff6603002a0 <the_repo>, key=0x7ff66026c432 <__ac_HASH_UPPER+1154> "core.repositoryformatversion", value=0x41a4c070c80 "0")
    at config.c:3092
#6  0x00007ff6600f9898 in git_config_set (key=0x7ff66026c432 <__ac_HASH_UPPER+1154> "core.repositoryformatversion", value=0x41a4c070c80 "0")
    at config.h:799
#7  0x00007ff6600fecd0 in initialize_repository_version (hash_algo=1, ref_storage_format=REF_STORAGE_FORMAT_FILES, reinit=0) at setup.c:2265
#8  0x00007ff6600ff0c8 in create_default_files (template_path=0x0, original_git_dir=0x41a4c010460 "D:/Dev/Source/test-git-repo/.git", fmt=0x5ff560, init_shared_repository=-1)
    at setup.c:2367
#9  0x00007ff6600ffbc6 in init_db (git_dir=0x41a4c030420 "D:/Dev/Source/test-git-repo/.git", real_git_dir=0x0, template_dir=0x0, hash=0,
    ref_storage_format=REF_STORAGE_FORMAT_UNKNOWN, initial_branch=0x0, init_shared_repository=-1, flags=2) at setup.c:2606
#10 0x00007ff65fe9352c in cmd_init_db (argc=1, argv=0x41a4c020380, prefix=0x0, repo=0x0) at builtin/init-db.c:251
#11 0x00007ff65fe13219 in run_builtin (p=0x7ff6601d6590 <commands+1392>, argc=2, argv=0x41a4c020380, repo=0x7ff6603002a0 <the_repo>) at git.c:480
#12 0x00007ff65fe13789 in handle_builtin (args=0x5ffdb0) at git.c:744
#13 0x00007ff65fe13a31 in run_argv (args=0x5ffdb0) at git.c:811
#14 0x00007ff65fe13fbd in cmd_main (argc=2, argv=0x41a4c0300a8) at git.c:951
#15 0x00007ff65ff2dd0e in main (argc=5, argv=0x41a4c030090) at common-main.c:9

[dscho]

@fusumayuki thank you for rising to the challenge!

[...]
#1  0x00007ff66014ae59 in error_errno (fmt=0x7ff66023f778 <fsync_component_names+3160> "could not write config file %s") at usage.c:281
#2  0x00007ff65ff89f1b in repo_config_set_multivar_in_file_gently (r=0x7ff6603002a0 <the_repo>, config_filename=0x41a4c070cd0 "D:/Dev/Source/test-git-repo/.git/config",
    key=0x7ff66026c432 <__ac_HASH_UPPER+1154> "core.repositoryformatversion", value=0x41a4c070c80 "0", value_pattern=0x0, comment=0x0, flags=0) at config.c:3400
[...]

That is interesting. config.c:3400 is in this block:

	if (commit_lock_file(&lock) < 0) {
		error_errno(_("could not write config file %s"), config_filename);
		ret = CONFIG_NO_WRITE;
		goto out_free;
	}

Contrary to what I had expected, the problem does not arise when the file is opened for writing, but when the temporary config.lock file is supposed to be renamed to config atomically, i.e. overwriting the existing config file in a way that there is no time window during which an incomplete config file would exist (or no config file at all).

Let's follow that code. Here is the commit_lock_file() function. It calls get_locked_file_path(), which is unlikely to produce the error you observed, and it calls commit_lock_file_to(), which is the most plausible candidate. That function is defined here and it is merely a shim for rename_tempfile(). That rename_tempfile() function has quite a few candidate calls that could produce an ENOSYS, but I deem the most likely to be the rename() call because in Git for Windows, that call actually calls mingw_rename(), and the mingw_rename() implementation saw a rather fundamental change in v2.48.0 (and since there was no Git for Windows v2.48.0, v2.48.1 would have been the first chance to observe any breakage introduced by said fundamental change).

My guess is that the return value of GetLastError() is different from the ERROR_INVALID_PARAMETER expected by that new code.

@fusumayuki could I ask you to launch the command in gdb again (after deleting the now-stale .git/config.lock file), set a breakpoint like b mingw.c:2894 before r (to run the command), and when it stops at that breakpoint, inspect the value of gle via both p gle (which will output it in decimal) and printf "%x\n", gle (which will print it in hex)? If my theory is correct, the shown value would differ from 87 (ERROR_INVALID_PARAMETER).

If my theory is incorrect, that breakpoint would not be hit. In this case, I would like to ask you to set a breakpoint on the mingw_rename() function like so: b mingw_rename. Once that breakpoint is hit, I'd like to ask you to single-step using the u command (which will "step over" function calls) or the s command (which will "step into" function calls). The first time 'round, supports_file_rename_info_ex should be non-zero and therefore the code should flow into this block. Once that fails to rename the file, it should fall back to this conditional block and I am fairly certain that this is currently not the case, and the bugfix will be to ensure that it does fall back that way.

Another potential vector to figure this out would be to set a breakpoint on mingw.c:163, i.e. the line where a Windows error value is translated into an errno value; If the value of error is 40 (the ENOSYS value), a bt will reveal which function call resulted in this error.

@fusumayuki thank you so much for assisting me in fixing this bug!

[fusumayuki]

@dscho thank you for your reply! The breakpoint at mingw.c:2894 was hit and the value of gle is 50 (0x32)

Here's the backtrace:

#0  mingw_rename (pold=0x5a564030cc0 "D:/Dev/Source/test-git-repo/.git/config.lock", pnew=0x5a564030cf0 "D:/Dev/Source/test-git-repo/.git/config") at compat/mingw.c:2894
#1  0x00007ff65232af8c in rename_tempfile (tempfile_p=0x5ff018, path=0x5a564030cf0 "D:/Dev/Source/test-git-repo/.git/config") at tempfile.c:348
#2  0x00007ff65220ff28 in commit_lock_file_to (lk=0x5ff018, path=0x5a564030cf0 "D:/Dev/Source/test-git-repo/.git/config") at lockfile.h:317
#3  0x00007ff6522105cc in commit_lock_file (lk=0x5ff018) at lockfile.c:210
#4  0x00007ff652199ef9 in repo_config_set_multivar_in_file_gently (r=0x7ff6525102a0 <the_repo>, config_filename=0x5a564070c30 "D:/Dev/Source/test-git-repo/.git/config",
    key=0x7ff65247c432 <__ac_HASH_UPPER+1154> "core.repositoryformatversion", value=0x5a564070be0 "0", value_pattern=0x0, comment=0x0, flags=0) at config.c:3399
#5  0x00007ff65219a02e in repo_config_set_multivar_in_file (r=0x7ff6525102a0 <the_repo>, config_filename=0x5a564070c30 "D:/Dev/Source/test-git-repo/.git/config",
    key=0x7ff65247c432 <__ac_HASH_UPPER+1154> "core.repositoryformatversion", value=0x5a564070be0 "0", value_pattern=0x0, flags=0) at config.c:3430
#6  0x00007ff65219a152 in repo_config_set_multivar (r=0x7ff6525102a0 <the_repo>, key=0x7ff65247c432 <__ac_HASH_UPPER+1154> "core.repositoryformatversion",
    value=0x5a564070be0 "0", value_pattern=0x0, flags=0) at config.c:3457
#7  0x00007ff6521993f5 in repo_config_set (r=0x7ff6525102a0 <the_repo>, key=0x7ff65247c432 <__ac_HASH_UPPER+1154> "core.repositoryformatversion", value=0x5a564070be0 "0")
    at config.c:3092
#8  0x00007ff652309898 in git_config_set (key=0x7ff65247c432 <__ac_HASH_UPPER+1154> "core.repositoryformatversion", value=0x5a564070be0 "0")
    at config.h:799
#9  0x00007ff65230ecd0 in initialize_repository_version (hash_algo=1, ref_storage_format=REF_STORAGE_FORMAT_FILES, reinit=0) at setup.c:2265
#10 0x00007ff65230f0c8 in create_default_files (template_path=0x0, original_git_dir=0x5a564010460 "D:/Dev/Source/test-git-repo/.git", fmt=0x5ff560, init_shared_repository=-1)
    at setup.c:2367
#11 0x00007ff65230fbc6 in init_db (git_dir=0x5a5640305a0 "D:/Dev/Source/test-git-repo/.git", real_git_dir=0x0, template_dir=0x0, hash=0,
    ref_storage_format=REF_STORAGE_FORMAT_UNKNOWN, initial_branch=0x0, init_shared_repository=-1, flags=2) at setup.c:2606
#12 0x00007ff6520a352c in cmd_init_db (argc=1, argv=0x5a564020380, prefix=0x0, repo=0x0) at builtin/init-db.c:251
#13 0x00007ff652023219 in run_builtin (p=0x7ff6523e6590 <commands+1392>, argc=2, argv=0x5a564020380, repo=0x7ff6525102a0 <the_repo>) at git.c:480
#14 0x00007ff652023789 in handle_builtin (args=0x5ffdb0) at git.c:744
#15 0x00007ff652023a31 in run_argv (args=0x5ffdb0) at git.c:811
#16 0x00007ff652023fbd in cmd_main (argc=2, argv=0x5a5640300a8) at git.c:951
#17 0x00007ff65213dd0e in main (argc=5, argv=0x5a564030090) at common-main.c:9

[dscho]

The breakpoint at mingw.c:2894 was hit and the value of gle is 50 (0x32)

Excellent! @fusumayuki could I ask you to try out the fix in #5515?

[fusumayuki]

The breakpoint at mingw.c:2894 was hit and the value of gle is 50 (0x32)

Excellent! @fusumayuki could I ask you to try out the fix in #5515?

It works now! Thank you so much for your help!

[dhirschfeld]

Thanks @dscho & @fusumayuki for getting to the bottom of this! ❤️

[dscho]

@dhirschfeld could you test the latest snapshot, please?