thai_website_crawlers

Scripts for crawling the 500 most visited websites in Thailand according to Alexa for th and en parallel texts.

Scripts

get_alexa_rankings.py - get top 500 most visited domains in Thailand according to Alexa
scrape_robots.py - scrape all sitemaps from https://domain/robots.txt
scrape_urls.py - scrape all URLs from https://domain/sitemap.xml
clean_urls.py - send head requests to verify that URLs return 200
scrape_requests.py - scrape URL contents and align by tags
check_health.py - simple dataset health checkup script for scraped datasets

Name		Name	Last commit message	Last commit date
Latest commit History 8 Commits
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
check_health.py		check_health.py
clean_urls.py		clean_urls.py
cleaning_notes.py		cleaning_notes.py
get_alexa_rankings.py		get_alexa_rankings.py
scrape_requests.py		scrape_requests.py
scrape_robots.py		scrape_robots.py
scrape_urls.py		scrape_urls.py