Discussion: What to do in event of no file_layout for an ArtifactSource

[graza-io]

Context

If no default file_layout is provided by the table and no file_layout is provided by the configuration, currently all items will be obtained.

I.e: This will pull all objects from the root onwards for an aws_s3_bucket source.

Whilst this is the behaviour we would expect; is it correct?

Additional Notes:

• What should we do if patterns are provided but no file_layout exists?
• Should we warn/fail if no file_layout and no default file_layout?
• If no file_layout is provided, we get no date information thus collection_state becomes large as it stores all items.
• This could be problematic if numerous file types are within any of the paths walked, as we'd expect them to match the designated mapper, etc.

[graza-io]

Notes:

• Build a distinct list of tp_source_locations (for files which data was removed) and passing these to the collect request
• ^ Only do above when no time information is available
  • Does the collection need to be more aware of granularity etc