MultiBackendJobManager: avoid mixing openEO job status and internal status in same column

MultiBackendJobManager involves a status column which is used for

official openEO status, e.g. "created", "queued", "running", "finished", ...
internal housekeeping e.g. "not_started", "queued_for_start"

Mixing both has popped up quite often as a source of confusion during MultiBackendJobManager feature requests, bug fixes, and attempts to improve test coverage (especially when we started to do things in threads and the order of execution stopped being deterministic)

the official job status and internal status should be separated

example of last incident: #839 (comment)

in 706fafe I naively wanted to stop counting "created" jobs as "running", which sounded reasonable, but then a bunch of MultiBackendJobManager unit tests became flaky

Revert 706faf and 3d3ca1 for #839/#840

@HansVRP

cc @HansVRP

Indeed, let's put it higher on the agenda.

However I do believe that this should be internal cuisine and hidden from the user.
These days they build color-map tools based on the dataframe which look at the single status column.

So I would propose to tackle this in the database only if possible

Revert 706faf and 3d3ca1 for #839/#840

 _column_requirements = _ColumnRequirements({
        # Official openEO backend status (from API calls)
        "backend_status": _ColumnProperties(dtype="str", default=None), -->"created", "queued", "running", "finished", "error", "canceled"
        
        # keep status with all non official states
        "status": _ColumnProperties(dtype="str", default="not_started"), --> "not_started", "created", "queued_for_start", "queued" "start_failed", "skipped", "running", finished, error, "downloading"

Then in the code internally we can work and act upon the backend status where needed?

These days they build color-map tools based on the dataframe which look at the single status column.

but what kind of status values do they expect there? Official openEO job statuses or do they also support these ad-hoc internal statuses?

also their adhoc statuses (e.g. a downloading status)

@VictorVerhaert

Another case of wrestling with the mixed up status field:

small fixes to status management in JobManager #872 Changes to job manager queue limit and queued_for_start #873

cc @VictorVerhaert

Sketched out a first solution where the status column remains unchanged, but we do introduce a column with only pure back-end statuses.

openeo-python-client/openeo/extra/job_management/_manager.py

Line 201 in 8bc4ac8

"backend_status": _ColumnProperties(dtype="str", default=None),

By allowing get_by_status to select either column

openeo-python-client/openeo/extra/job_management/_job_db.py

Line 73 in 8bc4ac8

    
           def get_by_status(self, statuses, max=None, column: str = "status") -> pd.DataFrame:

we can selectively pick jobs with a certain back-end status and still act upon their job_manager status

openeo-python-client/openeo/extra/job_management/_manager.py

Line 845 in 8bc4ac8

    
           active = job_db.get_by_status(statuses=["created", "queued", "running"], column="backend_status").copy()

Currently too early for a PR; but maybe good to spur the discussion? While it does resolve some of this issues; I do sense it adds a layer of complexity to have multiple statuses per job in various columns....

I think that this issue tries to reduce the complexity for developers for future changes to the job manager. I think however that introducing a new column creates new complexity for the users instead.
I see some users creating maps/summaries of their job_db's, who would now have to look at two different columns, which might become confusing.

What I think we need is clear documentation of which statuses exist, what the possible flows are and some checks on these.
One example is the recent bug where queued_for_start was being overwritten by the created status in the backend. This could also be the cause of the tests becoming flaky when stopping counting created as running.

A concrete proposal from my end is to consider the job manager statusses as "sub-statusses", where each job manager status corresponds to one single possible backend status.

not_started -> no backend status as job is not created yet (once again an ambiguous name)
created -> created
queued_for_start -> created
queued -> queued
start_failed -> created
skipped -> no status
running -> running
finished -> finished
downloading -> finished
error -> error

These should be checked in the _track_statusses

The mapping as a hidden contract, is probably more fragile than an explicit column. Furthermore the second column gets its status explicitly from the backend; independent of any assumption done in the mapping

Important to note; the status column would not change. So we would remain backwards compatible.

nevertheless; I agree that more columns to the database does benefit the users. We already have an issue here with start time and running start time. The information is really helpful if you know what you are doing, but when you are new; it complicates the matter.

I believe the discussion is moving into another direction: #877

I might also be wrong here, but while testing the MultiBackendJobManager, I noticed that sometimes a job gets "stuck" in the created stage. I guess this could happen whenever the request between creating the job and actually starting it might go wrong? However, looking at the code, I get the impression that the created status is considered to be a "running" status, which means that the batch job manager is not retrying if it got stuck there.

if starting the job, fails, it should go to the start_failed status. Curious to see where this does not occur

Navigation Menu

Repository navigation

MultiBackendJobManager: avoid mixing openEO job status and internal status in same column #840

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

MultiBackendJobManager: avoid mixing openEO job status and internal status in same column #840

Description

Activity

soxofaan commented on Dec 17, 2025

soxofaan commented on Dec 17, 2025

HansVRP commented on Dec 19, 2025

HansVRP commented on Dec 22, 2025

soxofaan commented on Jan 5, 2026

HansVRP commented on Jan 6, 2026

soxofaan commented on Mar 23, 2026

HansVRP commented on Mar 24, 2026

VictorVerhaert commented on Mar 25, 2026

HansVRP commented on Mar 27, 2026

HansVRP commented on Mar 27, 2026

JanssenBrm commented on Jul 12, 2026

HansVRP commented on Jul 13, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions