Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GPT OCR followup #148

Merged
merged 16 commits into from
Oct 27, 2024
+32,898 −63,970
Merged

GPT OCR followup #148

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
8aaa661
clean GPT OCR in ingest.py
danvk Oct 27, 2024
ff3d3d5
split interior whitespace
danvk Oct 27, 2024
7eeac9a
normalize more interior whitespace
danvk Oct 27, 2024
a3df2de
pipeline-style
danvk Oct 27, 2024
6eba892
try being much more lax about is_negative
danvk Oct 27, 2024
dddda1a
2/426 false positives
danvk Oct 27, 2024
77ea822
this is pretty good
danvk Oct 27, 2024
596de3f
images.ndjson diff script
danvk Oct 27, 2024
9857cd8
split up (1), (2), (3) etc on single one; 180 matches, 2 false positives
danvk Oct 27, 2024
2919b9c
remove F. S. LINCOLN stamps
danvk Oct 27, 2024
4bb8815
strip NEW YORK PUBLIC LIBRARY stamp (108)
danvk Oct 27, 2024
16ed45c
Drop "President Borough of Manhattan" (862!)
danvk Oct 27, 2024
2d48b06
update keep-ids with new cleaner
danvk Oct 27, 2024
53feee9
add another manual list
danvk Oct 27, 2024
318a8bf
run cleaner on old site OCR, too
danvk Oct 27, 2024
bc211db
squash excessive newlines
danvk Oct 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
88,827 changes: 29,609 additions & 59,218 deletions data/gpt-ocr.json
View file
Open in desktop

Large diffs are not rendered by default.

6,182 changes: 3,091 additions & 3,091 deletions data/images.ndjson
View file
Open in desktop

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions data/originals/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,3 +17,4 @@ Files related to the 2024 change from Ocropus → GPT-based OCR:
- `ocr-heavy-editor.ids.txt`: backing IDs of items that were reviewed by heavy users of the OldNYC OCR Corrector (heavy = 100+ edits).
- `ocr-big-movers.txt`: Dan's manual review of ~500 items with a large edit distance between the existing on-site OCR and GPT.
- `ocr-spelld25.txt`: Dan's manual review of ~80 items where GPT has more misspelled words and the length of the transcriptions varied by 25+ characters.
- `ocr-followup.txt`: More manual choices from follow-up review.
Loading