29 Commits

Author SHA1 Message Date
fc6f3ff809 add another makelaar 2026-04-13 23:19:05 +02:00
1841412c93 Update schielandborsboom scraper for new URL structure
- Change listings URL to /wonen/zoeken/heel-nederland/prijs=200000-300000/schiedam/
- Update status extraction to read from card-house__status badge text
- Update detail parser to handle house-features__block HTML structure
- Test verified: successfully fetches Aleidastraat 130 A listing
2026-04-13 22:58:30 +02:00
c6328cee46 fix: extract postcode for vanoord and vanherk scrapers
Van Oord: postcode is in the first .elementor-heading-title on detail pages.
Van Herk: postcode extracted via regex from <title> tag; also pick up kamers
and energielabel from the features list which were previously ignored.
Test output now includes woonoppervlak and energielabel fields.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-11 23:53:17 +02:00
f74e9bcfb0 refactor: split ssr.py into package, enrich OG Online detail pages, fix travel upsert
- Split src/adapters/ssr.py (2160 LOC) into ssr/ package grouped by CMS:
  realworks.py, sure.py, schiedam.py, denhaag.py, overige.py
- Add _og_detail() to api.py; all OG Online scrapers now fall back to
  detail page fetch when energielabel/bouwjaar are missing from the API
- Fix run() to recalculate travel times for existing listings where
  fiets_mark IS NULL; upsert() now writes travel cols on existing rows too
- Update tests/cache.py to patch fetch_soup in every ssr submodule
- Update docs to reflect new package structure and mark API enrichment TODO done

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-11 23:39:35 +02:00
1011d9cf87 feat: add scrapers for vandriel, vanherk, vanoord
- vandriel (Schiedam): OG Online API, filtered by city=schiedam
- vanherk (Schiedam): SURE WordPress plugin (card-house), detail page kenmerken
- vanoord (Delft + Schiedam): Elementor WordPress, two filtered listing URLs, rw-object-features-list detail parsing
- makelaars.md: mark all three as done, add TODO for API scraper detail page enrichment

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-11 19:31:08 +02:00
9149d11a06 infra: get claude code from unstable 2026-04-05 19:06:05 +02:00
6bb2538143 feat: energielabel badge, disable-filters toggle, freetext search
- Render energielabel as a coloured EU-style letter badge (A+++ to G)
  instead of plain icon text
- Add "Filters uit" toggle button that bypasses all numeric filters
  while keeping sort and freetext search active; turns orange when on
- Add freetext search bar that filters across adres, stad, postcode,
  source_makelaar and woningtype in real time
- Reset button also clears search and deactivates the disable toggle

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-05 19:05:22 +02:00
77f8e91f07 feat: show energielabel on main card 2026-04-05 18:58:35 +02:00
7096220203 docs: update scraper prompt with OG Online, SURE, Realworks patterns and postcode tip
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-04 23:32:37 +02:00
75c5b6f26d add scrapers: 88makelaars, Borgdorff (SSR) + Elzenaar, DOEN (OG Online API) for Den Haag
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-04 23:30:06 +02:00
6beae1133b add scrapers: Olsthoorn (SURE), Post Makelaardij, Morris (Realworks) for Delft
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-04 23:07:27 +02:00
bfd69e3542 add scrapers: V&W, ZO Makelaars (Realworks), Roepman (JSON-LD) for Delft
- fetch_vwmakelaars, fetch_zomakelaars: one-liner Realworks wrappers
- fetch_roepman: custom JSON-LD scraper (Realworks CMS uses div.aanbodEntry
  instead of li.aanbodEntry; price from potentialAction priceSpecification)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-04 21:43:43 +02:00
d310a7a560 add scrapers: Van Daal (API), Van Silfhout (SSR) for Delft
- fetch_vandaal: OG Online API, covers Delft/Rijswijk/Den Haag area,
  includes is_bought→verkocht status mapping
- fetch_vansilfhout: HTML scraper, all listings on single page,
  extracts postcode from embedded JS variable (objectZipcode)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-04 21:39:02 +02:00
c92ddb5812 add scrapers: Moerman & De Jong (API) and Schieland Borsboom (SSR)
- fetch_moerman: OG Online realtime-listings API (same platform as bjornd),
  includes bouwjaar from dateOfConstruction, energielabel, strips postcode space
- fetch_schielandborsboom: paginated HTML scraper filtered to Schiedam,
  fetches #kenmerken detail page for full specs (bouwjaar, kamers, etc.)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-04 21:34:58 +02:00
edd2580919 fix: config naming error 2026-04-04 16:05:36 +02:00
942170ef7f change web port 2026-04-04 15:51:01 +02:00
84e5656ca0 add prod web deployment 2026-04-04 15:49:22 +02:00
e1745841b1 restyle page, add area filters 2026-04-04 15:27:12 +02:00
fbe50790da tweaks and first real run 2026-04-04 15:23:09 +02:00
423a429f56 add dev override and web layer 2026-04-04 14:47:01 +02:00
f1748214ce drop email support 2026-04-04 14:11:07 +02:00
8450c33887 HA webhook works, also more makelaars 2026-04-04 01:35:29 +02:00
b35025b9cb ever onwards 2026-04-03 16:58:57 +02:00
918042d27e Add D&S Makelaars scraper (Schiedam)
Fetches 51+ listings from D&S with full details:
- Paginates through /aanbod/koopwoningen
- Extracts property postcode from Google Maps iframe URL
- Parses all kenmerken (features) from detail pages
- Includes price, address, rooms, area, build year, energy label

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
2026-04-03 16:42:52 +02:00
18c01139c2 give in to the vibe 2026-04-03 16:32:00 +02:00
4f37a1dd37 improve logging 2026-04-03 16:15:29 +02:00
efd31686be update autoscraper, add another makellaar 2026-04-03 16:09:38 +02:00
17b35d1997 add some more makelaars, and some more infra 2026-04-03 15:49:42 +02:00
26d9d936f4 first setup, travel works, bjornd api works 2026-04-03 13:53:39 +02:00