forked from SWISDATA/processintel
transformations: CSV timestamps are sorted as strings, so events in a trace are in the wrong order #1
Labels
No milestone
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
ffrauenberger/ProcessIntel#1
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Severity: critical (every CSV log with non-ISO dates silently gets wrong traces, including the bundled example)
Where
app/io_operations/import_operations.py:27app/transformations/dataframe_transformations.py:62-69Problem
When a CSV event log is imported, the timestamp column is never converted to datetimes.
ImportOperations.read_csvcallspd.read_csv(...)withoutparse_dates, so the column stays a string (object) column unless it is purely numeric.DataframeTransformations.dataframe_to_cases_listthen sorts by it:The timestamps are therefore sorted lexicographically, and every miner (Alpha, Heuristic, Inductive, IMf, Fuzzy, Genetic) receives traces in the wrong order. The user gets no warning. The XES import path is not affected, because it converts timestamps in
xes_to_csv_converter.py:148.Two typical failure patterns:
"11.3.10"sorts before"9.3.10"."9.3.10 10:15"sorts before"9.3.10 8:05".In addition,
sort_valuesusesquicksortby default, which is not stable. Events with the same timestamp (common with day-level dates) lose their original file order, and the result can change between runs or pandas versions.Steps to reproduce
tests/testcsv/CallcenterExample.csvExpected behavior
Events within each case are ordered chronologically. Events with identical timestamps keep their order from the file.
Actual behavior
Events are ordered as strings. In Case 1 the app gives
['Handle Case', 'Call Outbound', 'Inbound Call']; the chronological order is['Inbound Call', 'Handle Case', 'Call Outbound']. The mined model is wrong.Corrections from the review
"11.3.10"sorts before"9.3.10"). Unpadded hours also break the order.CallcenterExample.csv, 341 of 3,885 cases get the wrong event order, and the log grows from 354 to 416 variants.na_position="last").Fix
DataframeTransformations.dataframe_to_cases_listor a helper it calls, so all six miners are fixed at once.objectcolumns with an explicitformat, or withdayfirstchosen by the user.format="mixed". Withoutdayfirst,pd.to_datetimesilently misreads 2,901 of 7,548 fixture dates. It never raises and never gives NaT, so a count of unparseable rows would say 0.app/ui/column_selection_ui/), not in the model. If only one parses everything, use it.errors="coerce"only after the format is fixed. Count NaT and missing values, show the count, and stop or drop those rows explicitly.sort_values(by=[time_label], kind="stable").CsvToXesConverter._parse_timestamps: it has the same day-first bug (#27).Check
New
tests/transformations/dataframe_transformations_test.py(the existing file is never run, see #56):CallcenterExample.csvCase 1 is['Inbound Call', 'Handle Case', 'Call Outbound']from
issues/BUG-001-csv-timestamps-sorted-as-strings.md(code analysis of commit6bcc646, 2026-10-03), corrected by the review inverdicts-claude/BUG-001-csv-timestamps-sorted-as-strings.mdReviewed by Codex (GPT-6) against origin/main
6bcc646eb464.Review: partially correct.
The main bug and fixture numbers are confirmed: 341/3,885 cases differ from stable chronological ordering; variants change from 354 to 416; 2,901/7,548 timestamps are misread by the converter's mixed parser. However, “every CSV log with non-ISO dates” is too broad: string sorting only changes a trace when its lexical and chronological order differ. Likewise, ISO-8601 is a safe lexical workaround only with a consistent representation, precision and timezone/offset. Mixed ISO offsets need datetime parsing too. Keep the stable-sort/parsing fix, but qualify the severity and workaround.
Code references: app/transformations/dataframe_transformations.py:62, app/io_operations/import_operations.py:27.
🤖 work-issue
Fixed on branch
fix/csv-timestamp-parsing(uncommitted in the working tree).Change
app/transformations/timestamp_parsing.py(new):parse_timestamps. Numeric and datetime columns (XES) stay as they are. Text columns are parsed with the user'sstrftimeformat (orISO8601). With no format, ISO 8601 and common day-first, month-first and year-first formats (separators.,/,-; with or without time) are tried. A format is chosen only if it is unambiguous. If several formats fit with different results, or none fits, or a value is missing, it raisesTimestampParseException.app/transformations/dataframe_transformations.py:69: parses before sorting and sorts withkind="stable". New optionaltimestamp_format/timestampFormatargument.app/exceptions/io_exceptions.py:TimestampParseException.app/ui/column_selection_ui/standard_column_selection_view.py: new "Timestamp format (optional)" input.column_selection_controller.pyon_mine_clickstores it asselected_timestamp_format.app/ui/base_algorithm_ui/base_algorithm_controller.py: passes the format on. OnTimestampParseExceptionthe conversion stops, no model is created, and the user returns to the column selection page with the error message and their selections restored.tests/mining_algorithms/genetic_mining_test.py:genetic_test.csvuses09-3-2004:15.01, which is not a known format, so the test now passes%d-%m-%Y:%H.%M.Verification
tests/transformations/dataframe_transformations_test.py: unpadded days and hours, US dates, ISO with mixed offsets, ties in file order, numeric and datetime columns, ambiguous dates, user format, non-matching values, missing values, andCallcenterExample.csvCase 1 =['Inbound Call', 'Handle Case', 'Call Outbound'].python -m unittest discover -s tests -p '*_test.py': 266 tests OK.blackapplied.