Progress Manager System

Module: xefm/progress_manager.py

ProgressManager tracks the state of one long-running file operation and formats it for display. It is deliberately small: it holds the current operation’s counts and current item, drives a ProgressAnimator frame, and renders a status line or rich text segments. It does not own threads, cancellation, priorities, time estimates, or persistence — those belong to the task layer (xefm/task.py) and the file-operations worker (xefm/file_operations.py).

Scope

What ProgressManager does:

• Track a single active operation (OperationType, total, processed, current item, byte progress, error count, and a “still counting” flag).
• Advance a spinner frame via an internal ProgressAnimator.
• Format the operation as a plain status line or as layout segments.
• Optionally push updates to a progress_callback (throttled).

What it does not do (handled elsewhere):

• Threading and the worker loop → TaskManager / the file-ops worker.
• Cancellation → Task.checkpoint() / Task.request_cancel() raising Cancelled.
• Rendering the modal dialog and prompts → ProgressDialog (in xefm/task.py).

There is no operation priority system, no ETA calculation, and no resume/replay — earlier drafts of this document described those, but they are not in the code.

OperationType

class OperationType(Enum):
    COPY = "copy"
    MOVE = "move"
    DELETE = "delete"
    ARCHIVE_CREATE = "archive_create"
    ARCHIVE_EXTRACT = "archive_extract"

API

ProgressManager(config=None)          # builds its own spinner ProgressAnimator

start_operation(operation_type, total_items, description="", progress_callback=None)
update_operation_total(total_items, description="", total_bytes=0)  # ends "counting"
update_progress(current_item, processed_items=None)   # None → auto-increment
update_file_byte_progress(bytes_copied, bytes_total)  # single-current-file byte bar

# Per-file transfer slots — the copy engine's path (parallel-safe):
file_begin(item, total_bytes=0) -> slot   # claim a row; does NOT count the item
file_bytes(slot, copied, total=None)      # cumulative; delta feeds processed_bytes
file_end(slot)                            # count the item; credit unstreamed bytes
get_transfers() -> [(slot, state), ...]   # locked snapshot for the dialog rows

refresh_animation()                                   # force a callback (no data change)
increment_errors()                                    # bump the error counter
finish_operation()                                    # clears state; callback(None)

is_operation_active() -> bool
get_current_operation() -> dict | None
get_progress_percentage() -> int
get_progress_text(max_width=80) -> str
get_progress_segments() -> list

Notes on the real shapes (these differ from older drafts):

• finish_operation() takes no arguments; it clears state and, if a callback is set, calls it once with None to clear the display.
• update_progress(current_item, processed_items=None) — the first argument is the item name; the count auto-increments unless overridden.
• Errors are a simple integer counter (increment_errors()), not a structured error list.
• The current operation is a plain dict with keys: type, total_items, processed_items, current_item, description, errors, file_bytes_copied, file_bytes_total, counting, total_bytes, processed_bytes, transfers.

Two-phase progress: counting then determinate

An operation starts with counting=True and (typically) total_items=0, so the UI shows an indeterminate “Preparing…” state. Once the worker has recursively counted the work, it calls update_operation_total(total), which flips counting off and makes the primary bar determinate. update_progress also clears counting as a safety net.

Byte-level progress

update_file_byte_progress(copied, total) feeds a secondary bar for the current file. It is only surfaced for files larger than 1 MiB (file_bytes_total > 1024*1024), rendered compactly (e.g. [15M/32.0G]); small files show no byte bar.

copied is cumulative for the current file, and the growth since the last report also accumulates into the operation-wide processed_bytes — the same thing file_bytes does for a slot. That is what lets an operation which works one file at a time (archives) have a byte-weighted primary bar too. update_progress zeroes the per-file counter as it names the next item, which is what keeps each file’s first report from being credited with the previous one’s bytes again.

Transfer slots (the copy engine’s path, issue #268)

The copy engine — sequential or parallel — reports each file through a slot instead of the single current-file fields:

• file_begin(item) claims the lowest free slot and names it. It does not advance processed_items; four workers starting four files no longer reads as 4/5 done.
• file_bytes(slot, copied, total) takes cumulative bytes for that file; the growth since the last report accumulates into the operation-wide processed_bytes. Each slot owns its own counts, so concurrent workers never fight over one bar. While the slot’s file is still the most recently begun one, the legacy file_bytes_* fields mirror it for the flat-text renderers.
• file_end(slot) counts the item as processed and credits any bytes the copy path never streamed (a one-shot small file, an instant clone, a skip), so processed_bytes reaches total_bytes however the file traveled.
• file_closing(slot) marks the file as written but not yet closed. Where the destination holds a file in a local cache until close — WebDAV, NFS’s close-to-open flush — that close is where the bytes actually travel: one measured 64 MiB spent 0.04s in the write loop and 5.25s in the close. Without this the row reached its total and then sat unchanged for the whole of it, which is the operation’s real work rendered as a number that had stopped moving. transfer_bytes_text (in task.py) renders such a row as finishing… 12s in place of the counts. Nothing about it is specific to a network volume — a destination whose close is instant passes through in one frame — and it is optional: a caller that never calls it behaves as before.

Archive operations report through the same slots, via archive_progress.ByteProgress — begin / advance / finish around each member — even though most of their paths work through one member at a time and so fill one row. A dialog that changed shape depending on which format was being read was the alternative: 7z extraction really does have several members in flight, and everything else would have kept the older item-name-plus-second-bar layout. Creation has one more state to show, since after the last member the archive itself is still being written out — one transfer of the whole file on a volume that holds it until close, and nothing left for the bars to say. The task title carries that.

A finished file stays in its slot (done: True) until the worker’s next file_begin reuses it — ProgressDialog keys its per-transfer rows by slot, so rows update in place instead of blinking. get_transfers() returns a snapshot copied under the manager’s lock for exactly that consumer.

The progress percentage is weighted by bytes

get_progress_percentage() computes (processed_bytes + W·processed_items) / (total_bytes + W·total_items) with W = _ITEM_WEIGHT (8 KiB per item). Operations that never report byte totals (delete) reduce to the old pure item ratio; for copies and archive extraction, one 4 GiB file among a dozen small ones holds the bar back for the time it will actually take. update_operation_total(..., total_bytes=...) supplies the denominator — the counting pass already measures it.

The fixed per-item weight is not a rounding detail: on a destination where every file costs a round trip whatever its size (a WebDAV or SMB mount), it is the only part of the estimate that represents that cost, and a pure byte ratio would collapse on an archive of thousands of tiny members.

Rendering

• get_progress_text(max_width) — a single flat line ("⠋ Copying... 42/120 - report.pdf [15M/32.0G]"), handy for logs and tests.
• get_progress_segments() — rich segments for the layout engine (AsIsSegment + a middle-abbreviating FilepathSegment for the filename + an AllOrNothingSegment for byte progress), so the line degrades gracefully as width shrinks.
• get_progress_percentage() is available for a bar fraction but is not part of the text line.

The spinner frame comes from an internal ProgressAnimator (pattern_override='spinner', speed_override=0.08); see Progress Animation System.

Update throttling

Callbacks are throttled to at most one per callback_throttle_ms (50 ms), with force=True and the final update bypassing the throttle. This is only relevant when a progress_callback is used (push mode). The task UI renders in pull mode (below), reading state each frame instead.

Copy progress (worked example)

File copy is the richest consumer of ProgressManager and shows how the pieces fit. The threading and UI belong to the task layer; ProgressManager is just the shared state both sides read and write.

Threading model

A copy (like move/delete/duplicate) runs as a Task:

1. FileOperations builds a Task, calls task.progress.start_operation(...), and hands a run(task) closure to TaskManager.submit.
2. TaskManager.submit shows a ProgressDialog for the task and spawns a daemon worker thread running run(task). On the main thread it registers an animation tick that, each frame, pumps the task’s UI bridge and repaints the panel; when the worker signals completion it tears down the dialog and reports the result.
3. The worker mutates task.progress as it goes; the ProgressDialog only reads that state during draw. This pull-based rendering is what keeps the spinner and bars moving smoothly — the per-frame repaint re-reads the animator (which advances by elapsed time), independent of how often the worker updates counts. No separate animation-refresh thread is needed.

Because the worker only ever touches its own Task/ProgressManager and the main thread only reads during draw, there is no shared mutable UI state across the boundary.

The worker (linear, top-to-bottom)

FileOperations._run executes prepare → resolve → execute:

• Resolve conflicts first (cheap existence checks). Destination collisions are resolved one-by-one through the task’s UI bridge (task.ask(...)), which posts a request the main-thread pump turns into a conflict dialog and blocks on the answer — so prompts appear sequentially. Choices: skip, overwrite, keep-both (a fresh ` (N)` name), or cancel.
• Count the surviving plan recursively (_count / _count_node) to get total_items and total_bytes, then prog.update_operation_total(total_items, total_bytes=...) to leave the “Preparing…” phase (delete passes 0 bytes — it never reports any).
• Execute each top-level target. Before each unit, task.checkpoint() raises Cancelled if cancellation was requested. Directories and deletes use prog.update_progress(name); each file runs inside a file_begin … file_bytes … file_end slot, counting toward the item total only when it finishes.

Byte-level copy

_copy_file streams a file through _copy_bytes when it is large (size >= 1 MiB) or crosses storage backends; otherwise it uses a plain copy_to. A move to another filesystem reaches this path too — it is not a rename, so _is_atomic_move sends it down the copy tree and each large file drives its byte row (see File Operations System). For local large files it copies in 1 MiB chunks, calling prog.file_bytes(slot, copied, size) per chunk and checkpointing between chunks (a cancel deletes the partial file so no stub is left). Cross-storage copies delegate to Path.copy_to(...) with a progress callback that forwards cumulative counts into the same slot (and doubles as the cancel checkpoint — _remote_progress). Files that move in one call — a small copy_to, an APFS clone, a skip over an existing file — report nothing along the way; file_end credits their full size so the aggregate bar stays honest.

Errors and cancellation

• Per-entry failures are collected as (path, reason) and reported to the caller; one bad file never aborts the rest of the target. prog.increment_errors() bumps the visible error count.
• Cancellation flows through Cancelled raised from a checkpoint (or a “Cancel” conflict choice), unwinding into a clean partial summary. finish_operation() runs in a finally so progress state is always cleared.

Integration

# One TaskManager per app; each Task owns a ProgressManager as task.progress.
task = Task("Copy…", config=self.config, kind="copy")
task.progress.start_operation(OperationType.COPY, 0, description="")
task_manager.submit(task, panel, run=run, on_done=on_complete)

Synchronous mode (used by tests) runs the worker inline with no dialog, and task.ask resolves to its headless default.

Testing

• Progress calculations: start/update/finish, percentage, counting→determinate transition, byte-progress thresholds.
• Text and segment rendering at varying widths.
• Copy/move/delete workers: conflict resolution, per-entry error collection, cancellation via checkpoints, and partial-file cleanup.

Related

• Progress Animation System — the frame engine.
• xefm/task.py — Task / TaskManager / ProgressDialog (threading, UI bridge, rendering).
• xefm/file_operations.py — the copy/move/delete worker.
• doc/FILE_OPERATIONS_FEATURE.md — end-user documentation.

</content>