ui.base_algorithm: run long-running mining outside the UI script thread #54

Open
opened 2026-10-06 13:53:51 +00:00 by ffrauenberger · 0 comments
Owner

Severity: improvement (high: a parameter change during mining runs two minings at once on one model; reproduced as a crash)

Where

  • app/ui/base_algorithm_ui/base_algorithm_view.py:188-200
  • app/ui/base_algorithm_ui/base_algorithm_controller.py:239-260
  • app/mining_algorithms/genetic_mining.py:28-29, :110-166
  • app/ui/genetic_miner_ui/genetic_miner_controller.py:152-205 — the genetic miner's own run()
  • app/ui/base_algorithm_ui/base_algorithm_controller.py:216-219 — model construction outside the spinner

Problem

perform_mining runs inside st.spinner in the Streamlit script thread:

  • No progress reporting and no cancellation. The only "cancellation" is the genetic miner's process-global run ID, which is broken (#3).
  • User interactions during mining start new reruns (see the corrections below). Streamlit only interrupts the old script at Streamlit API calls, so the CPU-bound loop keeps running.
  • CPU-bound Python work holds the GIL and slows down all other sessions in the same process (#31).
  • Results are not cached across sessions, so identical requests are recomputed.

Corrections from the review

  • Reruns run concurrently; they are not queued. runner.fastReruns defaults to True. A widget change during mining starts a new script run at once, which calls perform_mining on the same model while the old thread is still computing. For the genetic miner this was reproduced as RuntimeError: Set changed size during iteration and a lost graph. The genetic controller then reports a population-size problem (genetic_miner_controller.py:190-194). For the other miners the RuntimeError is not caught (base_algorithm_controller.py:242-261).
  • Within a session, unchanged parameters already reuse the graph.
  • The model is built in process_session_state, outside any spinner (base_algorithm_controller.py:216-219). For AlphaMining this is the full XL/YL computation (alpha_mining.py:17).
  • A process pool cannot run the models while they read and write st.session_state (#46).

Related: #3, #31, #39.

Fix

In this order:

  1. Decouple the models from Streamlit (#46): move the threshold syncing (base_mining.py:121-124, :755-783) into the controller, and pass all parameters to generate_graph.
  2. Make a mining job a pure function (log, algorithm, params) → result. Run it in a bounded executor kept in st.cache_resource, and keep the job handle in st.session_state. A new request cancels the previous job of the same session.
  3. Replace the class-level run ID (#3) with a per-job cancellation flag. Check it inside the long loops, not only once per generation.
  4. Poll progress and results with st.fragment(run_every=…). The view shows progress and a Cancel button.
  5. Cache only deterministic miners, in a bounded LRU keyed by (log_fingerprint, algorithm, params).

Interim fix, small and independent: set runner.fastReruns = false in app/.streamlit/config.toml. This serializes reruns per session and removes the concurrent-mining crash. It does not fix the cross-session cancellation or the GIL contention.

Check

  • tests/mining_algorithms/genetic_mining_test.py: two GeneticMining instances run in parallel threads, and both produce a graph (fails today).
  • Controller test: a second perform_mining while one is running cancels the first and does not raise.

from issues/ARCH-009-long-running-mining-on-ui-thread.md (code analysis of commit 6bcc646, 2026-10-03), corrected by the review in verdicts-claude/ARCH-009-long-running-mining-on-ui-thread.md

**Severity:** improvement (high: a parameter change during mining runs two minings at once on one model; reproduced as a crash) ## Where - `app/ui/base_algorithm_ui/base_algorithm_view.py:188-200` - `app/ui/base_algorithm_ui/base_algorithm_controller.py:239-260` - `app/mining_algorithms/genetic_mining.py:28-29`, `:110-166` - `app/ui/genetic_miner_ui/genetic_miner_controller.py:152-205` — the genetic miner's own `run()` - `app/ui/base_algorithm_ui/base_algorithm_controller.py:216-219` — model construction outside the spinner ## Problem `perform_mining` runs inside `st.spinner` in the Streamlit script thread: - No progress reporting and no cancellation. The only "cancellation" is the genetic miner's process-global run ID, which is broken (#3). - User interactions during mining start new reruns (see the corrections below). Streamlit only interrupts the old script at Streamlit API calls, so the CPU-bound loop keeps running. - CPU-bound Python work holds the GIL and slows down all other sessions in the same process (#31). - Results are not cached across sessions, so identical requests are recomputed. ### Corrections from the review - **Reruns run concurrently; they are not queued.** `runner.fastReruns` defaults to `True`. A widget change during mining starts a new script run at once, which calls `perform_mining` on the **same** model while the old thread is still computing. For the genetic miner this was reproduced as `RuntimeError: Set changed size during iteration` and a lost graph. The genetic controller then reports a population-size problem (`genetic_miner_controller.py:190-194`). For the other miners the `RuntimeError` is not caught (`base_algorithm_controller.py:242-261`). - Within a session, unchanged parameters already reuse the graph. - The model is built in `process_session_state`, outside any spinner (`base_algorithm_controller.py:216-219`). For `AlphaMining` this is the full XL/YL computation (`alpha_mining.py:17`). - A process pool cannot run the models while they read and write `st.session_state` (#46). Related: #3, #31, #39. ## Fix In this order: 1. Decouple the models from Streamlit (#46): move the threshold syncing (`base_mining.py:121-124`, `:755-783`) into the controller, and pass all parameters to `generate_graph`. 2. Make a mining job a pure function `(log, algorithm, params) → result`. Run it in a bounded executor kept in `st.cache_resource`, and keep the job handle in `st.session_state`. A new request cancels the previous job of the same session. 3. Replace the class-level run ID (#3) with a per-job cancellation flag. Check it inside the long loops, not only once per generation. 4. Poll progress and results with `st.fragment(run_every=…)`. The view shows progress and a Cancel button. 5. Cache only deterministic miners, in a bounded LRU keyed by `(log_fingerprint, algorithm, params)`. Interim fix, small and independent: set `runner.fastReruns = false` in `app/.streamlit/config.toml`. This serializes reruns per session and removes the concurrent-mining crash. It does not fix the cross-session cancellation or the GIL contention. ## Check - `tests/mining_algorithms/genetic_mining_test.py`: two `GeneticMining` instances run in parallel threads, and both produce a graph (fails today). - Controller test: a second `perform_mining` while one is running cancels the first and does not raise. --- from `issues/ARCH-009-long-running-mining-on-ui-thread.md` (code analysis of commit 6bcc646, 2026-10-03), corrected by the review in `verdicts-claude/ARCH-009-long-running-mining-on-ui-thread.md`
Sign in to join this conversation.
No milestone
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
ffrauenberger/ProcessIntel#54
No description provided.