Truncate exc_info in JSON log output to avoid records being dropped by Elasticsearch #542

Very large exc_info values in the JSON log output can cause the whole log record to be lost in Elasticsearch.

Observed on batch job j-2610120840454bf0a8959acab10e8ee7 (cdse-prod, 2026-10-12), which failed in load_stac with "No features found for collection ...". The OpenEO batch job failed: ... record logged with logger.exception never showed up in Elastic. Only the raw traceback that Python prints on re-raise was there, as separate unparsed lines without job_id or name.

The exc_info of that record is about 33.5 KB: a chained Py4JJavaError with the full Java stack, an OpenEOApiException, and around 60 recursive convert_node/apply_process frames. In the openeo-cdse-batchjobs-* index exc_info is a plain keyword field without ignore_above, and Lucene rejects terms over 32766 bytes, so the whole document is most likely dropped. I did not confirm the rejection in the Vector or Elasticsearch logs. Other OpenEO batch job failed records I checked have an exc_info of 0.7 to 11 KB.

Proposal: truncate exc_info in the JSON formatter in openeo_driver/util/logging.py to roughly 30 KB (for example keep the head and tail with a "... truncated ..." marker), or reconsider how much is printed there. Changing the Elasticsearch index template is out of scope for this issue.