Boost filesystem directory_iterator problem on iOS?

[mackworth]

Hi, I'm having a weird problem, and before I raise it to the boost folks, I was hoping I can get your "Boost on iOS" opinion, as I'm not a boost expert (barely c++ capable).

When I use the boost filesystem::directory_iterator in my iOS program, it just plain doesn't work.

I started by building the current boost library using your script:

./boost.sh -ios -macos --boost-version 1.73.0 --ios-sdk 13.4 --ios-archs "armv7 arm64" --min-ios-version 10.2 --hidden-visibility --universal --boost-libs "date_time filesystem program_options regex system"

directory_iterator wasn't working and sometimes crashed in the project I'm working on, so I created a new Xcode project (attached below) to narrow it down. All I did was:

• Build a new single-view project in Xcode
• Rename AppDelegate.m to AppDelegate.mm
• Add the code (bottom of this post) to didFinishLaunchingWithOptions
• Add boost headers to Header Search Path
• Add library directory to Library Search paths
• Add libboost_filesystem.a to "Link Binary with Libraries" phase
• Then run the program.

I'm using current Xcode Version 11.4.1 (11E503a) on current MacOS: 10.15.4 (19E287) and current iOS: 13.4.1 (17E262).

Results: the itr->path exists; once for each file, but is always empty, giving error msg: "" does not exist. I point out that the preceding Cocoa access to the same directory works fine. so the file(s) are there, and the app has rights to get to them. As discussed below, sometimes I also get a malloc crash in directory.cpp > dir_itr_close when the for-loop finishes.

I did the same steps on a macOS version (both a command line version in main.mm and a sandboxed app version in applicationDidFinishLaunching; also attached for your interest). No changes to code at all, and both worked fine, either giving me a listing of my real Documents directory, or just showing the dummy file in the sandboxed Documents.

What triggered my investigation, was that I got two warning messages when compiling in my real project (these are the ones from the test project; I compressed the long names to $DerivedData). As discussed below, these only appear when I compile on the simulator with Inline Methods Hidden: NO

Direct access in function 'boost::filesystem::detail::directory_iterator_construct(boost::filesystem::directory_iterator&, boost::filesystem::path const&, unsigned int, boost::system::error_code*)' from file '$DerivedData/boost-ios/build/iOS/libboost_filesystem.a(directory.o)' to global weak symbol 'boost::system::detail::is_generic_value(int)::gen' from file '$DerivedData/Debug-iphonesimulator/testBoost.build/Objects-normal/x86_64/AppDelegate.o' means the weak symbol cannot be overridden at runtime. This was likely caused by different translation units being compiled with different visibility settings.>

Direct access in function 'boost::system::detail::system_error_category::default_error_condition(int) const' from file '$DerivedData/boost-ios/build/iOS//libboost_filesystem.a(directory.o)' to global weak symbol 'boost::system::detail::is_generic_value(int)::gen' from file '$DerivedData//Debug-iphonesimulator/testBoost.build/Objects-normal/x86_64/AppDelegate.o' means the weak symbol cannot be overridden at runtime. This was likely caused by different translation units being compiled with different visibility settings.

So, with that clue... I tried changing Xcode's Hidden options. In the test project, if I set Inline Methods Hidden (matching the compilation), then I get no warnings, empty file names reported and then a malloc crash. If I turn it off, I get the warnings, empty file names, but no crash. The Symbols Hidden by Default option seems to have no effect in either case. I note that the warnings only appear when I'm compiling on the simulator, but the other results are the same.

I then rebuilt the boost library removing the --hidden-visibility option. Absolutely no effect: Inline Methods hidden still gives no warning, no names and malloc crash; Inline Methods hidden off still gives warning, no names, and no malloc crash.

So, any obvious things I've missed, or suggestions for other things to try before escalating to the boost group?

thx
/Hugh

 using namespace std;
namespace fs = boost::filesystem;

@implementation AppDelegate
- (BOOL)application:(UIApplication *)application didFinishLaunchingWithOptions:(NSDictionary *)launchOptions {
 //Create a temp file to list
  NSString * objCPath = [NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES) objectAtIndex:0];
  fs::path doc_path(objCPath.fileSystemRepresentation);
  fs::ofstream( doc_path / "/dummyFile.txt" );
 // fs::ofstream( doc_path / "/dummyFile2.txt" );
  
  NSLog(@"Directory Contents: %@", [[NSFileManager defaultManager] contentsOfDirectoryAtPath:objCPath error:nil]);

  fs::directory_iterator end_itr;  // default construction yields past-the-end
  for (fs::directory_iterator itr(doc_path); itr != end_itr; ++itr) {
   fs::path path = itr->path();
   //
    if (exists(path))
      if (is_regular_file(path))
        std::cout << path << "; file size is " << file_size(path) << '\n';
      else if (is_directory(path))
        std::cout << path << " is a directory\n";
      else
        std::cout << path << "exists, but is neither a file nor a directory\n";
    else
      cout << path << " does not exist\n";
  }
  cout << "All done";
  return YES;

TestBoost.zip

[faithfracture]

I'm not sure off the top of my head, but I'll take a look when I get a chance.

[mackworth]

Any thoughts on this, or should i submit to the main boost group?

[faithfracture]

Well, unfortunately I don't have an answer for you. I tried several different ways of accessing the directory with boost::filesystem and nothing worked. I'm seeing the same thing you are regarding inlines: I get warnings with the default settings, and a EXC_BAD_ACCESS when freeing the iterator if I build boost with --hidden-visibility and set inline methods hidden in the Xcode project settings.

What is interesting to me is that the filesystem iterator loop does execute once, printing "" does not exist, which suggests that it is finding the file. If you delete the app from the simulator and re-run it with the line fs::ofstream( doc_path / "/dummyFile.txt" ); removed, the loop doesn't execute at all.

If I run this:

fs::path dummyFilePath( objCPath.fileSystemRepresentation + std::string("/dummyFile.txt"));
if (exists(dummyFilePath))
    if (is_regular_file(dummyFilePath))
        std::cout << dummyFilePath << "; file size is " << file_size(dummyFilePath) << '\n';
    else if (is_directory(dummyFilePath))
        std::cout << dummyFilePath << " is a directory\n";
    else    
        std::cout << dummyFilePath << "exists, but is neither a file nor a directory\n";
    else
        cout << dummyFilePath << " does not exist\n";

I get :

"/Users/jordanbondo/Library/Developer/CoreSimulator/Devices/41690CE5-E025-4DFA-A842-8C1E11F1A4E3/data/Containers/Data/Application/F45DA358-CAF3-45D9-9CD2-59FD98DD8856/Documents/dummyFile.txt"; file size is 0

I tried running this exact same code in a macOS CLI app and it works just fine.

So boost::filesystem obviously works - there just appears to be a problem with directory_iterator on iOS.

At this point, if you really need to use boost::filesystem:directory_iterator, you're probably going to have to try to talk with the Boost guy.

Out of curiosity, why can't you just read files from the users documents directory using normal iOS functions?

[mackworth]

Thank you very much for fully checking it out; always nice to know I’m not crazy.

Any suggestion where the best place to submit to the larger project would be?

We’re using this code in a pure c++ model that is shared between Android and iOS. I was cleaning up a bunch of compiler warnings when I came across this issue. It’s not a particularly critical piece of code, so I might just restructure. For now, I just ignore the loader warnings, and it seems to work great.

Thanks again, and thanks for the installer. It’s been a huge timesaver in using boost.

[faithfracture]

It's been quite a while since I communicated with the Boost guys, so I don't recall exactly what the best line of communication is. They have this community page with links to their mailing list & IRC channels. I'm pretty sure I used IRC last time, but my question was more easily answered than I think this will be. Still, I'd try there first.

We do have some bits in our shared library that I've had to explicitly re-write in mobile platform compatible ways (various filesystem stuff, cryptography, etc). We typically abstract the interface away & then I'll just supply my platform specific implementation. If it's just a small piece of code, I might add an #ifdef, but anymore than that and we prefer to split things up.

[faithfracture]

Of note:

I switched from using boost::filesystem to std::filesystem and this code works fine. I had to switch the C++ Dialect build setting to C++17. If that's an option for you it might save you the headache of communicating with Boost devs.

[mackworth]

I’ll give that a try. Thanks again!

[mackworth]

Just for your interest, I did some further debugging. Had to switch optimization off to single-step through the boost code (after recompiling, I admit I was somewhat surprised that it still fails).

Setting a breakpoint when it enters the loop.

for (fs::directory_iterator itr(doc_path); itr != end_itr; ++itr) {
==>   fs::path path = itr->path();

p itr

(boost::filesystem::directory_iterator) $1 = {
  m_imp = {
    px = 0x0000600002045300
  }
}

p itr.m_imp.px->dir_entry.m_path

(boost::filesystem::path) $2 = (m_pathname = “[[long iOS path here]]/Documents/dummyFile.txt”)

but then a call to path (which should simply returns m_path) fails, both from lldb and the program...
p itr->path()
(const boost::filesystem::path) $3 = (m_pathname = "")
When I single-step through the itr->path() call, it first takes me to
iterator_facade arrow operator ->, which calls apply(*this->derived()). (derived seems to be just a cast)
* calls iterator_core_access(aka Facade)::dereference which calls f.dereference()

the source of f.dereference is just

  boost::iterator_facade<    directory_iterator,  directory_entry,   boost::single_pass_traversal_tag >::reference
 dereference() const  {
    BOOST_ASSERT_MSG(!is_end(), "attempt to dereference end directory iterator");
    return m_imp->dir_entry;
  }

BUT if you print f.dereference() you get the bad path, and if you print what f.dereference() is supposed to return: f.m_imp.px->dir_entry, you get the right path.

p f

(const boost::filesystem::directory_iterator) $3 = {
  m_imp = {
    px = 0x0000600001739400
  }
}

p &(f.dereference()) bad one
p f.dereference()

(boost::filesystem::directory_entry *) $5 = 0x0000600001739408
(boost::filesystem::directory_entry) $2 = {
  m_path = (m_pathname = "")
  m_status = (m_value = status_error, m_perms = no_perms)
  m_symlink_status = (m_value = fifo_file | socket_file | 0xfffffff0, m_perms = -1)
}

p &(f.m_imp.px->dir_entry) correct one
p f.m_imp.px->dir_entry

(boost::filesystem::directory_entry *) $4 = 0x0000600001739448
(boost::filesystem::directory_entry) $1 = {
  m_path = (m_pathname = "[[long iOS path]]/Documents/dummyFile.txt")
  m_status = (m_value = status_error, m_perms = perms_not_known)
  m_symlink_status = (m_value = status_error, m_perms = perms_not_known)
}

I note that the addresses of both of these directory entries are very similar, but I have no idea where the bad path one comes from; it looks like general garbage. In particular, that's not the "end of directory" entry that we had just created as that has a null px pointer.

And that's about as far as I can take it. It looks like a function dereference is returning a correct value, but the calling function gets a different one.

[faithfracture]

Wow. I did a little digging too, but I didn't get quite this far. That's very interesting. I'm sure it's probably something like "you need to pass this flag when building for iOS (or maybe ARM?)". The Boost guys are a different kind of C++ magician. I've looked through the source many times and it's still very mysterious to me. Sometimes I can see what's going on, but most of the time I just wind up with "I have no idea why or how this works, but it does... Let's leave it at that. Maybe I'll understand this some day."

If you do wind up opening a discussion with them about this & get it figured out, please let me know! If it's some build setting we can change in the script it would be awesome to fix it.

[mackworth]

FYI: boostorg/filesystem#147
(No new info, just submitting the issue).

[mackworth]

So, you can see in the filesystem issue, they gave me a clue that enabled me track it down to a mutex pointer problem due to the use of pthreads. You can see the discussion at the bottom there, but one theory is that one should use BOOST_SP_USE_SPINLOCK instead. mutex v atomic counter v spinlocks in ARM multi-processor environments is now officially above my paygrade, but I thought I should let you know that maybe EXTRA_ARM_FLAGS should change (or maybe not).

[mackworth]

Just a bump here that the pThreads used in EXTRA_ARM_FLAGS is incompatible with iOS; one symptom as documented above is that the filesystem directory_iterator fails. EXTRA_ARM_FLAGS should be:
EXTRA_ARM_FLAGS="-DBOOST_SP_USE_SPINLOCK -g -DNDEBUG"

[kambala-decapitator]

I confirm that this solution works. Thanks @mackworth !

[mackworth]

. @kambala-decapitator: Ah. I was just coming back to let you know my sample project worked fine with your code (Unfortunately, Github sends the original comment as an email and not the edits.).

Glad to see it worked for you after all!

[kambala-decapitator]

sorry for the noise :) I simply forgot to change path to the built libs while testing the solution.

[mbendiksen]

Just wanted to comment that this crash also happens when building macOS silicon (not just iOS) since it includes $EXTRA_ARM_FLAGS as well.

[mackworth]

@mbendiksen Were you able to make it work with this updated one: https://github.com/mackworth/Apple-Boost-BuildScript ?

[mbendiksen]

Yes, your new proposed $EXTRA_ARM_FLAGS seem to work well.

[parcool]

I'm a beginner of c/c++ and ios. @mbendiksen where add the $EXTRA_ARM_FLAGS flag? @mackworth where add the EXTRA_ARM_FLAGS="-DBOOST_SP_USE_SPINLOCK -g -DNDEBUG" ?

[kambala-decapitator]

@parcool check how it's done in respective PR #65 or just use the PR's branch

[parcool]

@kambala-decapitator Thanks so much. I think that commits too complicated for me, so I use the pr's branch to build, my real device works well but there is no a ios simulator floder. what can i do? I wondering why the cool pr not be merged?

[kambala-decapitator]

@parcool I can't say why simulator wasn't built without seeing the command you used. Please also tell which Xcode version you use.

PRs aren't merged because the repo seems abandoned.

btw you can also use my branch where I have some other fixes applied, if it fits you: https://github.com/kambala-decapitator/Apple-Boost-BuildScript/tree/all-improvements-from-PRs

[parcool]

@kambala-decapitator hello good man. I have tried your link post there and It get some error in the log file. I just use the command ./boost.sh --boost-libs 'filesystem'. Maybe my xCode(14.0.1) or mac(12.6) is not compatibility? Never mind,I use that pr's branch is okay even if no simulator .

[kambala-decapitator]

@parcool sorry, I haven't tried with Xcode 14 yet (only with 13), maybe there're new issues with it.

by default Boost is built for all available platforms, maybe that's where the error is coming from. You can pass e.g. -ios to build only for iOS.