CzechAI Scraping Infrastructure — Kompletní Audit 2026-04-06
Vygenerovano: 2026-04-05 23:59 | Typ: scraping-audit | CzechAI
1. Tok dat: od portalu do databaze
🌐 194 portalu (all_portals_list.json na 134)
│
▼ mega_all_portals_scraper.py → DISABLED (#DISABLED_BY_PIPELINE)
│ Mel iterovat 143 portalu, rate limit 0.5s, dedup URL hash
│ Nahrazen: reality-pipeline daemon (22-step orchestrator)
│
▼ reality-pipeline daemon → CRASHED (ConnectionResetError)
│ PM2 id 37, status STOPPED, 28 restarts, nikdo nemonitoruje
│
▼ Jedine co bezi:
│ bezrealitky-scraper (PM2, ONLINE 14D) → idnes.cz + bezrealitky
│ drazby-mega (ONLINE 20D) → drazby/aukce
│ primy-majitel-pipeline (ONLINE 7D) → hledani primych majitelu
│ scraping-data-plane v2.1 (ONLINE 39h) → infrastruktura, nikdo nevola
│
▼ Parsovani HTML:
│ Extrahuje: title, price, area, url, property_type, transaction_type
│ NEEXTRAHUJE: description, condition, floor, photos, city_part, street
│ Duvod: scraper cte jen LISTING page (seznam), ne DETAIL page
│
▼ MASTER_LIVE.db na serveru 134 (240 MB, ~227K zaznamu)
│ 80 sloupcu v schematu, ~75 z nich PRAZDNYCH
│
▼ sync_db_from_134.sh (cron kazdych 30 min)
│ Prosty SCP celeho souboru: 134 → /tmp → mv → 179
│ ⚠️ PREPISE DB zatimco lokalni writery do ni zapisuji!
│ ⚠️ WAL/SHM se nesynchronizuji → SQLite korupce
│
▼ MASTER_LIVE.db na serveru 179 (kopie)
│
▼ Enrichment pipeline (179, cronove joby):
│ batch_valuate.py → V15f (port 5310) → price_se2 (jen PRODEJE!)
│ batch_images.py → OG:image z meta tagu (= placeholder, NE fotka!)
│ opportunity_alert.py → is_opportunity flag
│ export_signal_json.py → JSON pro dashboardy
│
▼ Konzumenti: Hunter Agent, Lovec, Dashboard, XLSX export
2. PM2 sluzby — co bezi a co ne
Server 134 (Scraper)
| Sluzba | Status | Co dela | Problem |
|---|
| scraping-data-plane | ONLINE 39h | Data Plane v2.1 — Crawl4AI + Playwright + httpx | Infrastruktura OK, ale nikdo ji pravidelne nevola |
| bezrealitky-scraper | ONLINE 14D | bezrealitky_connector.py | Bezi, ale Sreality data maji spatny transaction_type/price mapping |
| scout-agent | ONLINE | Web investigator | Reaktivni (na pozadani), neni pravidelny scraper |
| primy-majitel-pipeline | ONLINE 7D | Hleda prime majitele (ne RK) | OK |
| drazby-mega | ONLINE 20D | Drazby/aukce | Jen aukce |
| reality-pipeline | ⛔ STOPPED | Hlavni 22-step orchestrator (mel nahradit vsechny disabled crony) | CRASHED — ConnectionResetError + KeyboardInterrupt. 28 restartu. NIKDO NEMONITORUJE. |
| enrichment-batch | ⛔ STOPPED | Batch enrichment detail pages | Nikdy nespusten |
| feature-enrichment | ⛔ STOPPED | Feature extraction | Nikdy nespusten |
| mega-price-v7 | ⛔ STOPPED | Cenova mapa v7 | Legacy, nahrazeno V15f |
Server 179 (Enrichment + API)
| Sluzba | Status | Co dela |
|---|
| v15f-api (5310) | ONLINE | ML valuator, MAPE 3.6% |
| cenova-mapa-v15 (5150) | ONLINE | V15f uvicorn server |
| super-ensemble-v2 (5111) | ONLINE | 7-engine ensemble |
| znalecky-panel (5112) | ONLINE | 3 AI znalci + RAG (~4min/req) |
| ocenovac-api (5300) | ONLINE | GPS/KNN, 145K ref records |
| hunter-agent (5800) | ONLINE | Deal hunter |
| data-watchdog | ONLINE | DB monitor |
| signal-api | ONLINE | Signal export |
3. 194 portalu → kolik se SKUTECNE scrapuje?
⚠️ V souboru all_portals_list.json na serveru 134 je 194 portalu. V cronu je 8 scrapers s #DISABLED_BY_PIPELINE. reality-pipeline daemon (nahradu) CRASHNUL. Vysledek: scrapuji se efektivne 2-3 portaly.
| Portal | Zaznamu v DB | Status scraperu |
|---|
| reality.idnes.cz | ~99K | ONLINE (bezrealitky-scraper connector) |
| sreality.cz | ~54K | DISABLED (cron #DISABLED_BY_PIPELINE) |
| eurobydleni | 14K | STARA data (mega_all_portals DISABLED) |
| bazos.cz | 14K | DISABLED (bazos_hyper_avizo DISABLED) |
| reality.cz | 5K | Legacy, nescrapuje se |
| realitymix.cz | ~7K | Legacy |
| bezrealitky.cz | ~4K | ONLINE ale data kvalita spatna |
| Zbylych ~187 portalu: 0 zaznamu / stara data / nescrapuji se |
4. Proc nejsou fotky a udaje
Data v DB maji 80 sloupcu, ale:
| Pole | Vyplnenost | Proc |
|---|
| title, price, area, url | 99% | Z listing page (jednoduche) |
| description | ~0% | Vyzaduje navstivit DETAIL page → scraper to nedela |
| condition (stav bytu) | ~0% | Dtto — detail page, CSS selector |
| floor/patro | ~0% | Dtto |
| image_url (realna fotka) | ~0% | batch_images.py bere OG:image = sharing placeholder (facebook.png). Realne fotky jsou v Fancybox galerii, nikdo je neextrahuje. |
| gps_lat/lon | ~40% | geocode_addresses.py cron bezi, ale jen pro zaznamy s parsovanou adresou |
| price_se2 (odhad ceny) | ~31% | batch_valuate.py bezi, ale V15f ocenuje jen PRODEJE |
Hlavni problem: Scraper ma 2 faze:
1. LISTING page (seznam) → URL + title + price + area (BEZI)
2. DETAIL page (inzerat) → popis + fotky + stav + patro + vybaveni (NEBEZI — enrichment-batch je STOPPED)
Dokud faze 2 nebezi, 75 ze 80 sloupcu v DB zustane prazdnych.
5. DB sync a korupce
⚠️ ROOT CAUSE opakujici se DB korupce (2× za den 2026-04-05):
sync_db_from_134.sh kazdych 30 min:
1. SCP cely MASTER_LIVE.db (240 MB) z 134 do /tmp/
2. mv /tmp/MASTER_LIVE_sync.db → MASTER_LIVE.db
ZATIMCO na 179 bezi data-watchdog a cenova-mapa-v15 ktere do DB PISUJI.
SQLite nezvlada nahrazeni souboru pod aktivnimi writery → B-tree damage.
WAL a SHM se nesynchronizuji (SCP kopiruje jen .db soubor).
Fix: Nahradit SCP za sqlite3 .backup pres SSH pipe, NEBO zastavit lokalni writery pred sync, NEBO pouzit SQLite replikaci (Litestream).
6. Valuacni retezec
| Engine | Port | Status | Presnost | Problem |
|---|
| V15f API | 5310 | ONLINE | MAPE 3.6%, 90% <5% | Jen PRODEJE. Pronajmy nemaji zadny valuator! |
| Super-Ensemble v2.1 | 5111 | ONLINE | 7 engines | Potrebuje GPS → chybi pro ~60% zaznamu |
| Ocenovac | 5300 | ONLINE | GPS/KNN 145K ref | OK ale jen pro prodeje |
| Znalecky Panel | 5112 | ONLINE | 3 AI znalci | Pomaly (4min/req), jen top deals |
| cenova-mapa-ai V8 (5093) | | DEAD | ML ensemble | Port nereaguje |
| cenova-mapa-portaly (5094) | | DEAD | - | Legacy mrtvy |
7. Kompletni seznam ROZBITYCH veci
- reality-pipeline daemon STOPPED na 134 — hlavni 22-step orchestrator co mel nahradit 8 disabled cron jobu. Bez nej se scrapuje jen idnes. 190 ze 194 portalu stoji.
- sync_db_from_134.sh zpusobuje DB korupci — SCP prepise DB zatimco lokalni writery (data-watchdog, cenova-mapa-v15) do ni zapisuji. Kazdy 30min cyklus = riskujeme corrupt.
- Detail page enrichment NEBEZI — enrichment-batch a feature-enrichment STOPPED na 134. Proto chybi fotky, popisy, stav, patro, vybaveni.
- Sreality Connector DISABLED v cronu — 54K zaznamu jsou stare, nove nepribyuaji.
- Bazos/Hyper/Avizo DISABLED v cronu — dalsi portaly stoji.
- mega_all_portals_scraper DISABLED — 143 portalu z JSON listu se nescrapuje.
- batch_images.py bere jen OG:image = sharing placeholder, ne realna fotka bytu.
- V15f ocenuje jen prodeje — pronajmy nemaji zadny valuation engine.
- Crawl4AI endpoint /v1/crawl4ai na routeru vraci 404 — chybi nginx location. Data Plane na 134:5100 funguje lokalne ale neni exponovan.
8. Prioritizovany plan oprav
| P | Co opravit | Dopad | Cas |
|---|
| P0 | Restartovat reality-pipeline daemon na 134 + pridat monitoring (pipeline_watchdog uz existuje, ale nemonitoruje daemon) | Ozivi scraping 194 portalu | 5 min restart + 30 min monitoring |
| P0 | Opravit sync_db: nahradit SCP za sqlite3 .backup pres SSH, nebo stopnout lokalni writery pred sync | Zastavi opakovanou DB korupci | 30 min |
| P1 | Spustit enrichment-batch: pro kazdy zaznam navstivit detail page, extrahovat description + condition + photos (Fancybox a[data-fancybox]) + floor + amenities | Vyplnenost dat z 5% na 80%+ | 2-4h implementace |
| P1 | Odkomentovat Sreality Connector v cronu na 134 | +22K cerstvych Sreality zaznamu/den | 5 min |
| P2 | Pridat pronajem valuator (median Kc/m2 per ctvrt z DB dat, ne ML model) | Pronajmy dostanou trzni odhad | 1-2h |
| P2 | Exponovat Crawl4AI z Data Plane pres nginx na router.czechai.io/v1/crawl4ai | Kazdy agent muze scrapovat detail page pres API | 15 min |
| P3 | Odkomentovat bazos, hyper, avizo, mega_all_portals v cronu | +17K zaznamu z dalsich portalu | 5 min + monitoring |