fix(other): gmail messages not shown

[christer77]

see issue

ref: RFC

[indridieinarsson]

May I add a comment?
The issue ( #1402 ) turns out to be really two different things. One is the inability of gmail to process BODY.PEEK[0.1] (but I think BODY[0.1] is fine, so might consider changing that one back). And I think this PR fixes that (but I'm not quite sure what info we might be missing by changing from [0.1] to [1]? )

The other thing is that in the combined view, the automatic check and renewal of the xoauth2 bearer token (which is valid for 1 hour after renewal) does not take place, and this PR doesn't fix that.

I made some code that seems to fix this part of it : indridieinarsson@3c74c2a
For the dedicated mailbox pages, the old code checks which mail server it is serving, and checks that server for expiry. But the request for the combined view yields and empty list of servers so no check is performed. In this case, I just check expiry for all available servers. A bit course, but maybe harmless.

[christer77]

May I add a comment? The issue ( #1402 ) turns out to be really two different things. One is the inability of gmail to process BODY.PEEK[0.1] (but I think BODY[0.1] is fine, so might consider changing that one back). And I think this PR fixes that (but I'm not quite sure what info we might be missing by changing from [0.1] to [1]? )

The other thing is that in the combined view, the automatic check and renewal of the xoauth2 bearer token (which is valid for 1 hour after renewal) does not take place, and this PR doesn't fix that.

I made some code that seems to fix this part of it : indridieinarsson@3c74c2a For the dedicated mailbox pages, the old code checks which mail server it is serving, and checks that server for expiry. But the request for the combined view yields and empty list of servers so no check is performed. In this case, I just check expiry for all available servers. A bit course, but maybe harmless.

hello @indridieinarsson,

Can you push your suggestion in current PR please?

[indridieinarsson]

May I add a comment? The issue ( #1402 ) turns out to be really two different things. One is the inability of gmail to process BODY.PEEK[0.1] (but I think BODY[0.1] is fine, so might consider changing that one back). And I think this PR fixes that (but I'm not quite sure what info we might be missing by changing from [0.1] to [1]? )
The other thing is that in the combined view, the automatic check and renewal of the xoauth2 bearer token (which is valid for 1 hour after renewal) does not take place, and this PR doesn't fix that.
I made some code that seems to fix this part of it : indridieinarsson@3c74c2a For the dedicated mailbox pages, the old code checks which mail server it is serving, and checks that server for expiry. But the request for the combined view yields and empty list of servers so no check is performed. In this case, I just check expiry for all available servers. A bit course, but maybe harmless.

hello @indridieinarsson,

Can you push your suggestion in current PR please?

Sorry for the late answer - I'm quite busy these days.

I honestly don't know how I can push my branch into an existing PR. My code is available in it's own branch here : https://github.com/indridieinarsson/cypht/tree/issue_1402. Feel free to merge or cherry pick what you want.

My code is not ready. It works, in the way that one we navigate to the combined mailbox, the oauth token is renewed if needed. However, if the site is left open for a while, all gmail messages will disappear after about an hour (unless the page is reloaded). The reason is that after an hour, the token has expired, and the code that checks and renews it is only run on page load.
I was thinking this should rather be done closer to the imap module. I was thinking to catch the reply from the imap server (where it tells us that the token has expired), and trigger the renewal from there. But that's a bit more work, and I haven't found the time for it yet.

edit: forgot to mention @christer77 so he actually sees my message.

[christer77]

May I add a comment? The issue ( #1402 ) turns out to be really two different things. One is the inability of gmail to process (but I think is fine, so might consider changing that one back). And I think this PR fixes that (but I'm not quite sure what info we might be missing by changing from [0.1] to [1]? )
The other thing is that in the combined view, the automatic check and renewal of the xoauth2 bearer token (which is valid for 1 hour after renewal) does not take place, and this PR doesn't fix that.
I made some code that seems to fix this part of it : indridieinarsson@3c74c2a For the dedicated mailbox pages, the old code checks which mail server it is serving, and checks that server for expiry. But the request for the combined view yields and empty list of servers so no check is performed. In this case, I just check expiry for all available servers. A bit course, but maybe harmless.BODY.PEEK[0.1]``BODY[0.1]

hello @indridieinarsson,
Can you push your suggestion in current PR please?

Sorry for the late answer - I'm quite busy these days.

I honestly don't know how I can push my branch into an existing PR. My code is available in it's own branch here : https://github.com/indridieinarsson/cypht/tree/issue_1402. Feel free to merge or cherry pick what you want.

My code is not ready. It works, in the way that one we navigate to the combined mailbox, the oauth token is renewed if needed. However, if the site is left open for a while, all gmail messages will disappear after about an hour (unless the page is reloaded). The reason is that after an hour, the token has expired, and the code that checks and renews it is only run on page load. I was thinking this should rather be done closer to the imap module. I was thinking to catch the reply from the imap server (where it tells us that the token has expired), and trigger the renewal from there. But that's a bit more work, and I haven't found the time for it yet.

edit: forgot to mention @christer77 so he actually sees my message.
Normally, we have a request that allows us to do this verification.Are you sure that it wasn't your session that had expressed and that you had to reconnect?
add_handler('ajax_hm_folders', 'imap_oauth2_token_check', true, 'imap', 'load_imap_servers_from_config', 'after');

[christer77]

May I add a comment? The issue ( #1402 ) turns out to be really two different things. One is the inability of gmail to process (but I think is fine, so might consider changing that one back). And I think this PR fixes that (but I'm not quite sure what info we might be missing by changing from [0.1] to [1]? )
The other thing is that in the combined view, the automatic check and renewal of the xoauth2 bearer token (which is valid for 1 hour after renewal) does not take place, and this PR doesn't fix that.
I made some code that seems to fix this part of it : indridieinarsson@3c74c2a For the dedicated mailbox pages, the old code checks which mail server it is serving, and checks that server for expiry. But the request for the combined view yields and empty list of servers so no check is performed. In this case, I just check expiry for all available servers. A bit course, but maybe harmless. BODY.PEEK[0.1]BODY[0.1] ``

hello @indridieinarsson,
Can you push your suggestion in current PR please?

Sorry for the late answer - I'm quite busy these days.
I honestly don't know how I can push my branch into an existing PR. My code is available in it's own branch here : https://github.com/indridieinarsson/cypht/tree/issue_1402. Feel free to merge or cherry pick what you want.
My code is not ready. It works, in the way that one we navigate to the combined mailbox, the oauth token is renewed if needed. However, if the site is left open for a while, all gmail messages will disappear after about an hour (unless the page is reloaded). The reason is that after an hour, the token has expired, and the code that checks and renews it is only run on page load. I was thinking this should rather be done closer to the imap module. I was thinking to catch the reply from the imap server (where it tells us that the token has expired), and trigger the renewal from there. But that's a bit more work, and I haven't found the time for it yet.
edit: forgot to mention @christer77 so he actually sees my message.
Normally, we have a request that allows us to do this verification.Are you sure that it wasn't your session that had expressed and that you had to reconnect?
add_handler('ajax_hm_folders', 'imap_oauth2_token_check', true, 'imap', 'load_imap_servers_from_config', 'after');

@indridieinarsson

[indridieinarsson]

Hello - I'm trying to get this to work.
This issue arises (e.g.) in the combined view. The combined view pages don't have a particular mail server attached to them, which leads to them not renewing the oauth bearer token. I made a partial fix, which re-authenticates brute-force for all imap servers within the Hm_Handler_imap_oauth2_token_check handler

But it's only a partial fix, since the combined pages refresh the message list regularly (via ajax, so without reloading the page). After 1hr, the gmail oauth bearer token becomes invalid, so we get a statuscode 400 when retreiving the gmail message list.

At this point, I'm unsure what to do, as the handler modules seem to only run on page load, which isn't happening here.

Could someone explain how the "handler modules" work and how I could trigger the process function on it from within the authenticate function on the HM_Imap class, without reloading the page?

[indridieinarsson]

@christer77 @kroky
Ok, much ado about nothing. Seems the issue is fixed with the current PR. I was testing on a version since december, where apparently snoozing messages wasn't being called when logged in through an IMAP account. I nuked my config in the meantime, so can't bisect it properly.

(the snooze worker goes on in the background and runs the oauth refresh. Snooze is part of the imap module, just as the oauth stuff, so should not be accidentally disabled)

So, with the current master, this PR fixes it, for me at least.

[kroky]

Sorry to be nit-picky but proper past tense of the verb find is found...

[kroky]

This function will deal with fraction of the cases as there are numerous transfer encodings, mime details and other stuff that might break here. Can you use zbateson/mail-mime-parser that is already included in Cypht - it has methods to get the first plaintext part of html part or whatever you need. If it is html part, then, maybe you need to strip tags first before trimming?