How to handle invalid config?

[jeffmendoza]

Allstar config is expected in yaml format and to unmarshal into the expected structs without error. Currently, when an error is found, Allstar just silently reverts to the default config. We would like users to be able to tell when they have an error. Some options:

• Allstar could create an issue, similar to the issue action from the policies. Some considerations:
  • Should this be done by default? If not, how to enable it? If by default, will it be spammy?
  • Which repo? If running policies on repo "foo" but the error is in repo ".allstar" or ".github"?
  • When to create a new issue, vs (re)open an existing? What if the issue is already open, but the previous error message is different?
  • Should issues close automatically? How to show the issue is fixed?
• Validator tool, A simple cli tool could be built that check for allstar yaml files and unmarshal them and print errors. this could be added to the .allstar repo as a workflow/action.

[azeemshaikh38]

Ideally, it should have been possible for AllStar to notify GitHub App Managers of an organization regarding any validation/setup related issues. I couldn't find any documentation to do this though. cc @josepalafox - does GitHub provide a mechanism for GitHub Apps to notify App Managers? If not, is this a feature we could request from GitHub?

In the short-term, a CLI might be a good workaround.

[azeemshaikh38]

Allstar could create an issue, similar to the issue action from the policies.

Creating GitHub issue is a pretty good idea. 👍

• Should this be done by default? If not, how to enable it? If by default, will it be spammy?

+1 to be done by default. It'll be equally spammy as AllStar policy violations, so not too bad?

• Which repo? If running policies on repo "foo" but the error is in repo ".allstar" or ".github"?

Would be nice if we can file issues in .allstar repo. I'm not sure who would get notified about it though, @jeffmendoza any idea?
Also, how do you mean error in .github?

• When to create a new issue, vs (re)open an existing? What if the issue is already open, but the previous error message is different?

Maybe we have one issue related to InvalidConfig and all errors go there?

• Should issues close automatically? How to show the issue is fixed?

Nice to have, but maybe not a requirement for a v0?

@brianrussell2 fyi.

[agonzalez-plume]

Agree, if config & also run errors could be raised in a summarized issue in the .allstar repo or I guess in the .github if someone is using that location. Allow someone to turn on error logging.

[ArisBee]

Is there any update on this issue?

I see from this PR that I should be able to opt out Archived repository. It doesn't seem to work so I wonder if there isn't an error with my allstar.yaml config file:

optConfig:
  optOutStrategy: true
  optOutPrivateRepos: false
  optOutPublicRepos: false
  optOutArchivedRepos: true
  optOutForkedRepos: false
  disableRepoOverride: true
schedule:
  timezone: "Europe/Amsterdam"
  days: ["saturday", "sunday"]

The schedule option to tell Allstar not to execute during weekends isn't effective either. I wonder if there isn't an issue with this file.

[jeffmendoza]

For the method of notification, the questions in the top post are still open questions. Any input on how you would like to be notified?

Yes, optOutArchivedRepos should work, and yes if there is an issue with the file not parsing it would affect that setting. Is the repo public, can you point me to the full config, so I can take a look?

[ArisBee]

Unfortunately this .allstar repo is private. The structure is as below and all the other config yaml files are executed as expected:

Regarding parsing issue notification, an error in the application logs would be more than enough to debug 😅

[pburkholder]

It would be really helpful if AllStar emitted its merged configuration at the INFO level at the start of each run, and anytime there's a local override.

I can't figure out if there's a bug in OptOutForkedRepos or whethe I'm just configuring things wrong.