io_operations: define a versioned, data-only model export format #53

Open
opened 2026-10-06 13:53:35 +00:00 by ffrauenberger · 1 comment
Owner

Severity: improvement (medium)

Where

  • app/ui/export_ui/export_controller.py:158-166
  • app/io_operations/export_operations.py:77-106
  • app/io_operations/import_operations.py:51-69

Problem

The "native model format" is pickle.dumps(mining_model). Besides the security problem (#32):

  • It is tied to the exact module, class and attribute layout. Any refactoring (e.g. #46, #49) makes all existing exports unreadable.
  • It embeds a graphviz.Digraph, numpy arrays, logger objects and the full event log with filtered copies, so files are large and opaque.
  • No other tool can read it.

Related: #32.
Likely a duplicate of upstream GitHub issue SWISDATA/processintel#6 Model Format and Import.

Fix

A versioned, documented export consisting of:

  • the event log (XES or CSV)
  • the algorithm and its parameter set
  • optionally the mined model in a standard format (PNML for Petri nets, BPMN 2.0 XML, DOT/SVG for graphs)

Re-mining from log + parameters is deterministic for all algorithms except Genetic; add a random seed parameter for that one.


from issues/ARCH-008-model-persistence-via-pickle-of-runtime-objects.md (code analysis of commit 6bcc646, 2026-10-03)

**Severity:** improvement (medium) ## Where - `app/ui/export_ui/export_controller.py:158-166` - `app/io_operations/export_operations.py:77-106` - `app/io_operations/import_operations.py:51-69` ## Problem The "native model format" is `pickle.dumps(mining_model)`. Besides the security problem (#32): - It is tied to the exact module, class and attribute layout. Any refactoring (e.g. #46, #49) makes all existing exports unreadable. - It embeds a `graphviz.Digraph`, numpy arrays, logger objects and the full event log with filtered copies, so files are large and opaque. - No other tool can read it. Related: #32. Likely a duplicate of upstream GitHub issue [SWISDATA/processintel#6](https://github.com/SWISDATA/processintel/issues/6) *Model Format and Import*. ## Fix A versioned, documented export consisting of: - the event log (XES or CSV) - the algorithm and its parameter set - optionally the mined model in a standard format (PNML for Petri nets, BPMN 2.0 XML, DOT/SVG for graphs) Re-mining from log + parameters is deterministic for all algorithms except Genetic; add a random seed parameter for that one. --- from `issues/ARCH-008-model-persistence-via-pickle-of-runtime-objects.md` (code analysis of commit 6bcc646, 2026-10-03)
Author
Owner

Reviewed by Codex (GPT-6) against origin/main 6bcc646eb464.

Review: partially correct.

The persistence/portability concern is supported, but “any refactoring makes all existing exports unreadable” is incorrect. Pickle can survive compatible attribute changes or a refactor that preserves import paths/compatibility aliases; module/class moves and incompatible state changes are the relevant risks.

Also, a Genetic seed parameter must control both Python random and NumPy's RNG, and normalize set iteration order (_create_partition turns a set into a list before shuffling). A seed alone does not guarantee identical exports across processes/hash seeds or dependency versions. If exact model preservation is required, store the mined result as data as well as log/parameters; distinguish equivalent mined behavior from byte-identical graph output.

Code references: app/io_operations/export_operations.py:106, app/mining_algorithms/genetic_mining.py:562, app/mining_algorithms/genetic_mining.py:612.

Reviewed by **Codex (GPT-6)** against origin/main [`6bcc646eb464`](https://code.swisdata.eu/ffrauenberger/ProcessIntel/commit/6bcc646eb464e0dd897228b5e49867f65d7255c4). **Review: partially correct.** The persistence/portability concern is supported, but “any refactoring makes all existing exports unreadable” is incorrect. Pickle can survive compatible attribute changes or a refactor that preserves import paths/compatibility aliases; module/class moves and incompatible state changes are the relevant risks. Also, a Genetic seed parameter must control **both** Python `random` and NumPy's RNG, and normalize set iteration order (`_create_partition` turns a set into a list before shuffling). A seed alone does not guarantee identical exports across processes/hash seeds or dependency versions. If exact model preservation is required, store the mined result as data as well as log/parameters; distinguish equivalent mined behavior from byte-identical graph output. Code references: [app/io_operations/export_operations.py:106](https://code.swisdata.eu/ffrauenberger/ProcessIntel/src/commit/6bcc646eb464e0dd897228b5e49867f65d7255c4/app/io_operations/export_operations.py#L106), [app/mining_algorithms/genetic_mining.py:562](https://code.swisdata.eu/ffrauenberger/ProcessIntel/src/commit/6bcc646eb464e0dd897228b5e49867f65d7255c4/app/mining_algorithms/genetic_mining.py#L562), [app/mining_algorithms/genetic_mining.py:612](https://code.swisdata.eu/ffrauenberger/ProcessIntel/src/commit/6bcc646eb464e0dd897228b5e49867f65d7255c4/app/mining_algorithms/genetic_mining.py#L612).
Sign in to join this conversation.
No milestone
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
ffrauenberger/ProcessIntel#53
No description provided.