Three Mysteries in the Swarm: Volume Locks, Ghost Columns, and Classic ASP Fallbacks

Tonight began with a routine multi-dimensional codebase audit of boun-scrape — the high-velocity ingestion engine responsible for crawling, indexing, and serving academic course schedules and live registration quotas for Boğaziçi University.

Everything looked pristine in continuous integration: 300+ pytest nodes running green, deterministic SQLite WAL configurations, zero-copy slot tokenization, and sub-second delta engines.

And yet, when we switched our perspective to production telemetry and real-world scraping on the live Docker Swarm cluster, the system began presenting three distinct, perplexing anomalies.

Here is the autopsy of three subtle engineering mysteries, how we proved the failure mechanisms, and the zero-overhead invariants we put in place.

---

Mystery 1: The “Invalid Credentials” 502 Ghost

When logging into the production dashboard on scraper.bountools.com, entering valid credentials (admin:<password>) failed with a generic alert.

Looking at the network tab, the frontend received HTTP 502 Bad Gateway from Traefik / Docker Swarm ingress. But running the same backend image locally authenticated cleanly in 4ms with HTTP 200 OK.

Why would an authentication endpoint return a 502 Bad Gateway on an existing, running container?

The Forensic Trace

We inspected the container logs on worker node WorkHorse (10.34.0.2):

sqlite3.OperationalError: attempt to write a readonly database
  File "/app/src/boun_scrape/storage/database.py", line 125, in execute_write
    cursor.execute(query, params)

In modern containerized deployments, security best practices dictate running processes as an unprivileged user (USER 10001:10001 in the Dockerfile).

When the Dokploy stack mounted the persistent named volume boununi-scraper-pw6pfa_schedules_data into /data, Docker created the host directory with root:root ownership (0:0, mode 0755):

# On worker node WorkHorse (10.34.0.2)
ls -ld /var/lib/docker/volumes/boununi-scraper-pw6pfa_schedules_data/_data
drwxr-xr-x 2 root root 4096 Aug 31 16:45 ...

The application could read existing data. But SQLite requires write access not only to schedules.db, but also to the parent directory to create atomic WAL journal locks (schedules.db-wal and schedules.db-shm).

When the user logged in, the auth route attempted to update last_login_at in the SQLite database. SQLite raised OperationalError: attempt to write a readonly database. Because the exception happened during the critical path of an unhandled async handler, the Uvicorn worker process panicked, the reverse proxy lost upstream connection, and Traefik surfaced a 502 Bad Gateway.

The frontend, seeing a network failure, fell back to displaying: “Invalid credentials or server error.”

The Invariant

We resolved the host volume ownership immediately (chown -R 10001:10001), and then established two architectural boundaries:

1. Pre-flight Database Diagnostics: On startup and on connection initialization, the DatabaseManager verifies write permissions to both the directory and the DB file, logging a structured, actionable CRITICAL alert if UID permissions are misconfigured.
2. Defensive Frontend Network Interceptors: Updated the frontend API client to explicitly intercept 502/503/504 Gateway Errors and 429 Rate Limits, presenting distinct diagnostic toasts instead of confusing credential error messages.

---

Mystery 2: The Portal Column-Offset Shift

In term 2026/2027-1, course queries began returning strange corrupted fields:

• instructor was set to "Info" across all courses.
• delivery_method contained class slot hours like "345" or "234".
• exam_date contained classroom strings like "M 1171 | M 1171".
• The instructor’s full name ("FATİH F. YILMAZ") was chopped character-by-character into single-letter dummy day slots (day: "F", day: "A", day: "T").

The Root Cause: Static Index Assumptions

Looking into the legacy HTML parser, schedule table parsing had historically relied on hardcoded table cell indices:

# The fragile legacy approach:
code_sec = tds[0].get_text(strip=True)
slot_title = tds[2].get_text(strip=True)
credits_raw = tds[3].get_text(strip=True)
ects_raw = tds[4].get_text(strip=True)
instructor = tds[5].get_text(strip=True)  # <-- Expecting instructor at index 5!
days_str = tds[6].get_text(strip=True)
hours_str = tds[7].get_text(strip=True)

For older semesters, this worked because the portal had 15 columns. But for 2026/2027-1, the university registration portal introduced an extra "Info" / Syllabus link column at index 2:

Header: [Code.Sec, Abbr., Info, Name, Cr., Ects, Instr., Days, Hours, Delivery, Exam Loc., Rooms, Exam Date, ...]

Because of that single extra column, every downstream index shifted by +1. Index 5 became "Info", Index 6 became the Instructor, and Index 7 became the Days!

The Fix: Dynamic Semantic Header Mapping

We replaced static integer offsets with dynamic table header inspection:

def _build_column_index_map(soup: BeautifulSoup) -> dict[str, int]:
    """Map semantic field names to column indices dynamically from table header."""
    title_tr = soup.find("tr", class_="schtitle") or soup.find(
        "tr", class_=lambda c: bool(c and "title" in str(c).lower())
    )
    if not title_tr:
        return DEFAULT_COLUMN_MAP

    headers = [td.get_text(strip=True).lower() for td in title_tr.find_all(["td", "th"])]
    col_map: dict[str, int] = {}

    for idx, h in enumerate(headers):
        clean_h = re.sub(r"[^a-z0-9\.]", "", h)
        if clean_h in ("code.sec", "codesec", "code", "derskodu"):
            col_map["code_sec"] = idx
        elif clean_h in ("name", "dersadi", "coursename", "title"):
            col_map["name"] = idx
        elif ("instr" in clean_h or "ogretim" in clean_h):
            col_map["instructor"] = idx
        elif clean_h in ("days", "gun", "gunler"):
            col_map["days"] = idx
        elif clean_h in ("hours", "saat", "saatler"):
            col_map["hours"] = idx
        # ... matches delivery, exam locations, rooms, dates dynamically
    return col_map

Whether the university portal serves 14, 15, 16, or 18 columns, the parser binds each semantic field to its exact recognized header, making the scraper immune to future column insertions.

---

Mystery 3: The Classic ASP 500 on Renamed Departments

During a full crawl of term 2024/2025-3, scraping failed with:

boun_scrape.scraper.client.BounHttpError: Server error 500 requesting /scripts/sch.asp
[ERROR] All-terms cycle: term 2024/2025-3 failed: SCED: Server error 500

Yet, curl requests to ATA and CMPE in the same term returned HTTP 200 OK.

Why would the university’s IIS server return 200 for 31 departments, but crash with an unhandled 500 for SCED?

Reverse-Engineering the ASP Query

We ran direct curl requests against Boğaziçi’s registration server (registration.bogazici.edu.tr) to isolate the variables:

# 1. Querying SCED with current department title:
curl "https://registration.bogazici.edu.tr/scripts/sch.asp?donem=2024/2025-3&kisaadi=SCED&bolum=SECONDARY+SCHOOL+SCIENCE+AND+MATHEMATICS+EDUCATION"
# -> HTTP 200 OK

# 2. Querying SCED with empty bolum parameter:
curl "https://registration.bogazici.edu.tr/scripts/sch.asp?donem=2024/2025-3&kisaadi=SCED&bolum="
# -> HTTP 200 OK

# 3. Querying SCED with historic cached title ("MATHEMATICS AND SCIENCE EDUCATION"):
curl "https://registration.bogazici.edu.tr/scripts/sch.asp?donem=2024/2025-3&kisaadi=SCED&bolum=MATHEMATICS+AND+SCIENCE+EDUCATION"
# -> HTTP 500 Internal Server Error (Microsoft IIS / SQL Server Recordset Exception)

The university’s Classic ASP backend runs an unparameterized SQL recordset filter matching both kisaadi and bolum. If a department has been renamed across academic years, querying with a historic name triggers a backend crash in Microsoft IIS.

Crucially, leaving bolum="" empty never crashes.

The Surgical Fallback

In src/boun_scrape/scraper/flow.py, we implemented an automatic fallback retry:

try:
    response = await client.get(SCHEDULE_DEPT_URL, params=params)
except BounHttpError as exc:
    # If the ASP portal failed with 500 and bolum was non-empty, retry with empty bolum
    if exc.status_code == 500 and bolum:
        logger.info(
            "Portal returned 500 for department %s with bolum='%s'; retrying with empty bolum fallback",
            dept_code,
            bolum,
        )
        fallback_params = {"donem": term, "kisaadi": dept_code, "bolum": ""}
        response = await client.get(SCHEDULE_DEPT_URL, params=fallback_params)
    else:
        raise

If the university alters a department title tomorrow, the scraper will seamlessly retry with empty bolum="" on the first 500, self-healing the crawl pipeline without manual intervention.

---

The Lessons of Live Web Ingestion

Distributed scraping against legacy institutional systems is never just about writing HTML selectors. It is about understanding the boundaries where systems meet:

1. Host-Container Boundaries: Container security models (USER 10001) must align with volume filesystem permissions (chown 10001:10001), or internal database errors will masquerade as networking failures.
2. Contract Boundaries: Never assume HTML tables have fixed column counts. Parse headers semantically at the boundary.
3. Legacy Server Quirks: When legacy backends behave unpredictably, isolate the exact query permutations. A single empty parameter fallback can turn a brittle crawler into an unbreakable one.

With all 305 tests passing in 31 seconds and zero debt markers remaining, the scraper runs calm and steady in the swarm.