Markk116/huizenbot
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

ever onwards

Browse Source
This commit is contained in:
Mark Kalsbeek
2026-04-03 16:58:57 +02:00
parent 918042d27e
commit b35025b9cb
4 changed files with 132 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
add_scraper_context.md
View File

@@ -219,6 +219,8 @@ If the CMS is unknown, the tool prints structural diagnostics (card selectors, f
## Important Notes
Don't treat detail pages as optional, we always want all the info!
### Status Mapping
Brokers use different status strings. Always map to one of:
- `"beschikbaar"` — Available for sale
@@ -270,6 +272,7 @@ The database stores this as JSON in the `extra` column.
- Nominatim (geocoding) has a 1 req/s limiter built into `huizenbot.py`
- Never spawn parallel requests without the human's approval
- Always use the `USER_AGENT` header (includes contact info for respectful scraping)
- Don't keep curling the same endpoint, pipe it to a <name makelaar>.dump and then rg through it to find what you need. Can also pipe it through the bsprettify.py and then rg that.
---