/bounty 250

💎 $250 bounty • Onyx (YC W24)

Steps to solve:

1. Start working: Comment /attempt #2282 with your implementation plan
2. Submit work: Create a pull request including /claim #2282 in the PR body to claim the bounty
3. Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

Thank you for contributing to onyx-dot-app/onyx!

Add a bounty • Share on socials

Note on the above: please check the Before opening PR section in the Connector README for additional requirements (testing + style)!

/attempt #2282

/attempt #2282

/attempt #2282

@Weves
After some research.
found this => https://github.com/orgs/community/discussions/59659

But if the user can provide the github pages repo.
we can get the repo content (i.e page contents)via github api.

@Weves ok how do we start ,do you need to be assigned to work on this or we can just submit our PR?

/attempt #2282

status ?

/attempt #2282

/attempt #2282

/attempt #2282

This pull request looks promising #3411 any plans to release this in the near future? My org could really use this.

@Weves, regarding PR #3411, is it still needed? If so, I'll resolve conflicts, otherwise, I can close it.

I want to work on this issue can you please assign to me @yuhongsun96 ? https://github.com/onyx-dot-app/onyx/blob/main/backend/danswer/connectors/README.md also the link is broken

assigned

@Weves, regarding PR #3411, is it still needed? If so, I'll resolve conflicts, otherwise, I can close it.

@yuhongsun96, I had already submitted a PR for this feature back in December (#3411) and followed up last week asking if it was still needed, but didn’t receive a response. Since my PR is already implemented (though it needs conflict resolution), I would appreciate some clarity on whether it is being considered or if I should close it. Thanks!

I see, let me take a look tomorrow or over the weekend then. @AayushSaini101 maybe hold off until I can verify

I'm going to pass this one to @AayushSaini101. Some feedback in the proposed PR:
Github Pages Connector Feedback:

• The load_from_state method isn’t implemented properly.
• The poll source method is missing logic to fetch pages based on start and end times.
• The yield statement isn’t used correctly.
• Right now, the site’s HTML is being fetched twice—once for the URL and again for the page content. This can be combined.
• _crawl_github_pages won’t capture all links if it hits the batch size limit, causing some links to be skipped.
• There’s no check for the updated_at time or date, which means the same documents might get indexed repeatedly.
• The retry mechanism could be handled better.
• _get_all_crawled_urls may run infinitely.
• In github pages connector we can also think about reusing the github credentials

@yuhongsun96, thank you for reviewing my pr and providing feedback. I understand these areas you mentioned need improvement and I'm ready to make those changes. As I've already put in the effort to create a PR, I'd like to ensure I have the opportunity to implement your suggestions and see this through.

@yuhongsun96, thank you for reviewing my pr and providing feedback. I understand these areas you mentioned need improvement and I'm ready to make those changes. As I've already put in the effort to create a PR, I'd like to ensure I have the opportunity to implement your suggestions and see this through.

@akhilender-bongirwar i have already started working on this, please check slack channel to know more about the current status of direction of your PR thanks cc: @yuhongsun96

@akhilender-bongirwar i have already started working on this, please check slack channel to know more about the current status of direction of your PR thanks cc: @yuhongsun96

@AayushSaini101, I understand you've started working on this yesterday as @yuhongsun96 suggested. However, I've already invested significant time and effort into PR's #3009, #3411, and #4233, including waiting for approvals and feedback. Just yesterday, I received first suggestions or first review for improvement on #4233, only to see the issue reassigned.

You could use my existing implementation in #4233 as a foundation for your work, building upon it with the suggested modifications. This would acknowledge the work I've already done.

@yuhongsun96, At the very least, I'd appreciate some form of recognition or compensation for the effort I've put into this issue, especially considering the bounty associated with it.