Bug 1922986 - Create script that exports BMO data as JSON suitable for import into a BigQuery instance in GCP #2370
Conversation
…r import into a BigQuery instance in GCP
…g ./extensions/BMO/bin/export_bmo_etl.pl
* upstream/master: Bumped version to 20241211.1
extensions/BMO/bin/export_bmo_etl.pl
Outdated
| # to make sure other entries with this date are not already present. | ||
| check_for_duplicates(); | ||
|
|
||
| ### Bugs |
There was a problem hiding this comment.
The code related to each individual table should be moved to a separate function, like extract_bugs, instead of being one large script separated by comments. This goes for attachments, flags, keywords etc.
Nit: also, looking at each of these sections, it seems there is some repeated structure amongst them. Each one preps a statement, runs a while loop over the total number of rows, fetches a row, checks the cache, then extracts the data and submits. You could eliminate that repetition.
extensions/BMO/template/en/default/hook/admin/params/editparams-current_panel.html.tmpl
Outdated
Show resolved
Hide resolved
|
I need to make some other non-related changes anyway as I realized i left out two additional requirements. 1) I need to skip sending bug data related to the 'Legal' product and 2) I need to filter the summaries, etc. for bugs that are private. |
|
I am going to make one last schema change to the BQ data so I will need to create the needed PRs in webservices-infra for this :( Once that is done, I will be able to submitted the updated code here for last review. Context: The bugs.group field needs to be a string as I originally had it as 1) most bugs do not have more than one group anyway and 2) if there is more than one group, we just want to put the group name with the least amount of users. I missed that little detail from the schema outlined here: https://docs.google.com/spreadsheets/d/1LDrYkGxZGnktI3-Ei3fpPK2TAXjxTj7fBUjHaJ8tpBE/edit Sorry for the extra delay. |
…tions as suggested.
- Added test to make sure we also add flag entries for deletions - Fixed some typos in export script
…table into memory ** This is better for memory efficiency and container crashes * Created a single function for all two column tables
|
Ready for hopefully final review. Optimized as much as I could. Refactored to create functions where possible. Switched to using LIMIT/OFFSET to allow for breaking down the tables that do not have a primary key into chunks instead of loading the whole table into memory (i.e. dependencies, etc.). Let me know if anything needs to be elaborated on. |
- Proper boolean value for is_new instead of 0 or 1
Sorry this so late. Lots of moving pieces had to be coordinated with webservices-infra, etc. just so I could test and then small changes one after another to get to this point.
Let me know if you have any questions as it is a lot.