functional? multiple errors trying to run (aws cloudtrail)

[kapilt]

Was just trying out tailpipe to see if it could be useful to a question / problem i had..
first attempt to run

tailpipe collect  --from T-90d
Error: collection error: failed to delete partition: failed to query parquet folder names: Invalid Input Error: File '/home/ubuntu/.tailpipe/data/default/tp_table=aws_cloudtrail_log/tp_partition=prod/tp_index=xxxxxxx/tp_date=2025-02-14/data_20250220191824_037313_0.parquet' too small to be a Parquet file

tweaked my config to ignore read events / manually deleted data directories, reran, two hours later get another error.

 tailpipe collect  --from T-90d

Collecting logs for aws_cloudtrail_log.prod from 2024-11-22

Artifacts:
  Discovered: 1,449,698
  Downloaded:    62,452 5.9GB
  Extracted:     62,413

Rows:
  Received: 32,650,740
  Enriched: 32,650,740
  Saved:     1,266,762
  Errors:    1,387,273

Completed: 2h0m0s
panic: send on closed channel

goroutine 66 [running]:
github.com/turbot/tailpipe/internal/collector.(*Collector).Notify(0x40004d8f40?, 0x4013119cc0)
        /go/src/tailpipe/internal/collector/collector.go:176 +0x40
github.com/turbot/tailpipe/internal/plugin_manager.(*PluginManager).readCollectionEvents(0x4000a22f00, {0x38bbfd8, 0x400058af50}, {0x38c5270, 0x4000131130})
        /go/src/tailpipe/internal/plugin_manager/plugin_manager.go:346 +0x15c
created by github.com/turbot/tailpipe/internal/plugin_manager.(*PluginManager).Collect in goroutine 1
        /go/src/tailpipe/internal/plugin_manager/plugin_manager.go:152 +0x908

also noting lack of clarity on the error count

A clear and concise description of what the bug is.

Tailpipe version (tailpipe -v)
Tailpipe v0.1.1

Plugin version (tailpipe plugin list)

tailpipe plugin list
INSTALLED VERSION PARTITIONS
hub.tailpipe.io/plugins/turbot/aws@latest 0.5.0-rc.2 aws_cloudtrail_log.prod

To reproduce
umm. just trying to collect

Expected behavior
something useful on error message, possibly working.

[cbruno10]

Hi @kapilt , sorry you're running into issues!

Can you please share your ~/.tailpipe/config/aws.tpc file (with sensitive info removed)?

Also, if you try to collect for a smaller range, e.g., tailpipe collect aws_cloudtrail_log --from T-7d, does collection and then querying work?

[kaidaguerre]

@kapilt there is a known issue where under some circumstances malformed parquet files are created. We will be addressing this issues to:

• try and prevent the creation of the bad file in the first place (if possible)
• if a bad file is found, handle the error

The second issue is a bit unpleasant. The stack trace should help in tracking it down, I'll have a dig.

[kaidaguerre]

(@kapilt transferring this issue to the tailpipe repo)

[graza-io]

@kapilt - we're looking at ways to make error reporting better.

In the meantime, it is possible to acquire the logs by setting the log level and then (recommended) piping stderr to a file:

TAILPIPE_LOG_LEVEL=WARN tailpipe collect aws_cloudtrail_log --from T-7d 2> /path/to/error.log

Note: If you enable log display into terminal whilst running collection, we know the interactive UI will cause alignment issues.