refactor: simplify journal and restore streamer cancellation

[BorysTheDev]

fixes: #4284
refactor ExecutionState: now we can have 3 independent states RUN, CANCELLED, ERROR

[kostasrim]

@BorysTheDev plz do not force push once a PR has comments. It's impossible to keep track of the logical changes between different commits.

1. If nobody reviewed -> squash and force push as much as you want
2. If there are reviews -> squash unpushed local commits but leave the previous ones unchanged

[kostasrim]

Change of the semantics here.

Before: we sleep either if operation::canceled or if we reported via error ReportError.

Now: we only sleep if there was an error. In other words, we no longer treat cancellation as error which is a mistake.

Cancellation is error and that's why errno exists with: ECANCELED == errc::operation_canceled.

I would like us to keep those semantics for clarity

[BorysTheDev]

we have cancel_error as well. I don't see any problems if you want create cancel error you call reportCancelError() if you want just cancel the process you cancel the process

[kostasrim]

This function is still being used (in replication and in other places). The issue here is that after this function is called IsError() is true. This is a bug and the correct values should be false and IsCancelled true.

You do not update the state == Cancelled here but this is also not needed. See my top comment regarding the 3 states you introduced.

[BorysTheDev]

if you report an error even if it CancelError the state will be error

[kostasrim]

We do not need these states. You want to figure out if:

1. Operation cancelled
2. Operation cancelled with an error

Cancelled operation is an error. That's why we have ReportCancelcedError which cancels the context with the error ECANCELED. Therefore you don't need any of the 3 states here. All you have to do is:

  bool IsError() const {
    return ec && !IsCancelled();
  }

  bool IsCancelled() const {
    return ec && err_ == ECANCELED;
  }

[kostasrim]

In your state machine:

RUN == (ec_ == std::error_code{})
CANCELLED == (ec_ == ECANCELED)
ERROR == (ec_ && !CANCELLED) 

[BorysTheDev]

if you use ec_ you need mutex and it's slow. Let's have a call.