ui.home: non-UTF-8 or malformed CSV files crash the import #14

Open
opened 2026-10-06 13:46:41 +00:00 by ffrauenberger · 1 comment
Owner

Severity: medium (common Excel CSV exports show an exception page)

Where

  • app/ui/home_ui/home_controller.py:122-140, :272-296
  • app/io_operations/import_operations.py:10-28, :106-126
  • app/analysis/detection_model.py:68-90

Problem

  • In HomeController.set_df, the CSV branch calls self.import_model.read_csv(...) without a try/except (only the XES branch is protected). A UnicodeDecodeError (e.g. a cp1252/Latin-1 export from Excel, very common in the DACH region) or a pandas.errors.ParserError (wrong delimiter, ragged rows) is raised inside a button on_click callback and shows a Streamlit exception page.
  • read_line decodes the first line as UTF-8. If that fails, delimiter detection silently falls back to ",".
  • A UTF-8 BOM is not stripped (encoding="utf-8-sig" is not used), so the first column name starts with  and column prediction does not recognize e.g. case.
  • The delimiter is sniffed from the header line only; csv.Sniffer on a single line can pick odd characters, such as a letter that occurs in every field name.

Steps to reproduce

  1. Open the application
  2. Upload a CSV saved in Windows-1252 / Latin-1 that contains umlauts (e.g. exported from Excel)

Alternatively: upload a CSV with ragged rows, or a UTF-8 CSV with BOM.

Expected behavior

The file is imported, or the user gets a clear message explaining the problem (encoding, delimiter, malformed rows).

Actual behavior

A Streamlit exception page (UnicodeDecodeError / ParserError). For files with BOM, the first column is not recognized.

Fix

Wrap CSV reading in error handling with a user-facing message. Try utf-8-sig first and fall back to a detected or user-selected encoding. Sniff the delimiter on a sample of several lines.


from issues/BUG-014-csv-import-encoding-and-parse-errors-crash.md (code analysis of commit 6bcc646, 2026-10-03)

**Severity:** medium (common Excel CSV exports show an exception page) ## Where - `app/ui/home_ui/home_controller.py:122-140`, `:272-296` - `app/io_operations/import_operations.py:10-28`, `:106-126` - `app/analysis/detection_model.py:68-90` ## Problem - In `HomeController.set_df`, the CSV branch calls `self.import_model.read_csv(...)` without a try/except (only the XES branch is protected). A `UnicodeDecodeError` (e.g. a cp1252/Latin-1 export from Excel, very common in the DACH region) or a `pandas.errors.ParserError` (wrong delimiter, ragged rows) is raised inside a button `on_click` callback and shows a Streamlit exception page. - `read_line` decodes the first line as UTF-8. If that fails, delimiter detection silently falls back to `","`. - A UTF-8 BOM is not stripped (`encoding="utf-8-sig"` is not used), so the first column name starts with `` and column prediction does not recognize e.g. `case`. - The delimiter is sniffed from the header line only; `csv.Sniffer` on a single line can pick odd characters, such as a letter that occurs in every field name. ### Steps to reproduce 1. Open the application 2. Upload a CSV saved in Windows-1252 / Latin-1 that contains umlauts (e.g. exported from Excel) Alternatively: upload a CSV with ragged rows, or a UTF-8 CSV with BOM. ### Expected behavior The file is imported, or the user gets a clear message explaining the problem (encoding, delimiter, malformed rows). ### Actual behavior A Streamlit exception page (`UnicodeDecodeError` / `ParserError`). For files with BOM, the first column is not recognized. ## Fix Wrap CSV reading in error handling with a user-facing message. Try `utf-8-sig` first and fall back to a detected or user-selected encoding. Sniff the delimiter on a sample of several lines. --- from `issues/BUG-014-csv-import-encoding-and-parse-errors-crash.md` (code analysis of commit 6bcc646, 2026-10-03)
Author
Owner

Reviewed by Codex (GPT-6) against origin/main 6bcc646eb464.

Review: partially correct.

The encoding/parser error-handling findings are supported, but the BOM claim is incorrect for the reviewed environment. Reproduced ImportOperations.read_csv() on UTF-8 BOM bytes with pandas 2.3.3: the columns are ['case', 'event', 'time'], with no leading BOM. pandas handles the initial UTF-8 BOM even though this call does not specify encoding='utf-8-sig'. Remove “the first column name starts with BOM” and the BOM-only failure reproduction. Encoding choices, error handling and multi-line delimiter sampling remain valid improvements.

Code references: app/io_operations/import_operations.py:27, app/ui/home_ui/home_controller.py:138.

Reviewed by **Codex (GPT-6)** against origin/main [`6bcc646eb464`](https://code.swisdata.eu/ffrauenberger/ProcessIntel/commit/6bcc646eb464e0dd897228b5e49867f65d7255c4). **Review: partially correct.** The encoding/parser error-handling findings are supported, but the BOM claim is incorrect for the reviewed environment. Reproduced `ImportOperations.read_csv()` on UTF-8 BOM bytes with pandas 2.3.3: the columns are `['case', 'event', 'time']`, with no leading BOM. pandas handles the initial UTF-8 BOM even though this call does not specify `encoding='utf-8-sig'`. Remove “the first column name starts with BOM” and the BOM-only failure reproduction. Encoding choices, error handling and multi-line delimiter sampling remain valid improvements. Code references: [app/io_operations/import_operations.py:27](https://code.swisdata.eu/ffrauenberger/ProcessIntel/src/commit/6bcc646eb464e0dd897228b5e49867f65d7255c4/app/io_operations/import_operations.py#L27), [app/ui/home_ui/home_controller.py:138](https://code.swisdata.eu/ffrauenberger/ProcessIntel/src/commit/6bcc646eb464e0dd897228b5e49867f65d7255c4/app/ui/home_ui/home_controller.py#L138).
Sign in to join this conversation.
No milestone
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
ffrauenberger/ProcessIntel#14
No description provided.