Skip to content

[BUG] Missing columnLineage facet when using SaveIntoDataSourceCommand and reading from Relation Nodes #3922

Open
#3940
Open
[BUG] Missing columnLineage facet when using SaveIntoDataSourceCommand and reading from Relation Nodes#3922
#3940
Labels
kind:bugSomething isn't workingstate:needs-triageRequires triage to determine kind, area and priority

Description

@luke-hoffman1
luke-hoffman1
opened on Jul 22, 2025

OpenLineage integration/client

Spark integration

OpenLineage version

io.openlineage:openlineage-spark_2.12:1.34.0

Technology and package versions

  • spark 3.5 (scala 2.12)
  • oracle driver: com.oracle.database.jdbc:ojdbc11:23.3.0.23.09
  • spark BQ Connector: com.google.cloud.spark:spark-3.5-bigquery:0.42.3

Environment configuration

No response

Deployment details

No response

Problem details

When I read from Oracle (JDBC) or from a CSV file and write to BigQuery, then the columnLineage facet is missing in the OL events.

What you think should happen instead

No response

How to reproduce

Oracle to BQ:

df = (
    spark.read.format("jdbc")
        .option("url", db_url)
        .option("query", "SELECT employee_id FROM SYSTEM.employees")
        .option("driver", "oracle.jdbc.OracleDriver")
        .option("user", properties["user"])
        .option("password", properties["password"])        
        .load()
)

df.write \
    .format("bigquery") \
    .option("project",google_project_id) \
    .option("writeMethod", "DIRECT") \
    .option("createDisposition","CREATE_IF_NEEDED") \
    .option('parentProject', google_project_id) \
    .option('temporaryGcsBucket', gcs_bucket) \
    .option('materializationDataset', google_bq_dataset) \
    .option('viewsEnabled', 'true') \
    .option("table", f"{google_bq_dataset}.oracle_source1") \
    .mode("overwrite") \
    .save()

CSV to BigQuery

df = spark.read.csv(f"{directory}/joined_csv_data", header=True, mode="overwrite")

df.write \
    .format("bigquery") \
    .option("project",google_project_id) \
    .option("writeMethod", "DIRECT") \
    .option("createDisposition","CREATE_IF_NEEDED") \
    .option('parentProject', google_project_id) \
    .option('temporaryGcsBucket', gcs_bucket) \
    .option('materializationDataset', google_bq_dataset) \
    .option('viewsEnabled', 'true') \
    .option("table", f"{google_bq_dataset}.csv_source") \
    .mode("overwrite") \
    .save()

Additional details

Logical Plans, respectively:

SaveIntoDataSourceCommand com.google.cloud.spark.bigquery.v2.Spark35BigQueryTableProvider@3b139c2e, Map(createdisposition -> CREATE_IF_NEEDED, project -> projectid materializationdataset -> dataset_name, viewsenabled -> true, writemethod -> DIRECT, temporarygcsbucket -> bucket, parentproject -> projectid, table -> dataset_name.oracle_source), Overwrite
   +- Relation [EMPLOYEE_ID#0] JDBCRelation(SYSTEM.employees) [numPartitions=1]

SaveIntoDataSourceCommand com.google.cloud.spark.bigquery.v2.Spark35BigQueryTableProvider@7e6d068, Map(createdisposition -> CREATE_IF_NEEDED, project -> projectid, materializationdataset -> dataset_name, viewsenabled -> true, writemethod -> DIRECT, temporarygcsbucket -> bucket, parentproject -> projectid, table -> dataset_name.csv_source), Overwrite
   +- Relation [EMPLOYEE_ID#17] csv

Do you plan to make this contribution yourself?

  • I am interested in doing this work

Activity

  1. luke-hoffman1
    added
    kind:bugSomething isn't working
    state:needs-triageRequires triage to determine kind, area and priority
    on Jul 22, 2025

  2. ddebowczyk92 commented on Jul 25, 2025

    @ddebowczyk92
    ddebowczyk92
    on Jul 25, 2025
    Contributor

    Hi @luke-hoffman1 , can you update your openlineage-spark version to a newer one? with io.openlineage:openlineage-spark_2.12:1.34.0 I couldn't recreate the issue

    {
  "eventTime": "2025-07-25T13:41:59.836Z",
  "producer": "https://github.com/OpenLineage/OpenLineage/tree/1.35.0/integration/spark",
  "schemaURL": "https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunEvent",
  "eventType": "COMPLETE",
  "run": {
    "runId": "019841d1-6562-766f-979c-8543f369bccf",
    "facets": {
      "parent": {
        "_producer": "https://github.com/OpenLineage/OpenLineage/tree/1.35.0/integration/spark",
        "_schemaURL": "https://openlineage.io/spec/facets/1-1-0/ParentRunFacet.json#/$defs/ParentRunFacet",
        "run": {
          "runId": "019841d1-5f60-7967-96be-fa23c6b33bf4"
        },
        "job": {
          "namespace": "default",
          "name": "openlineage_test_bigquery_connector"
        },
        "root": {
          "run": {
            "runId": "019841d1-5f60-7967-96be-fa23c6b33bf4"
          },
          "job": {
            "namespace": "default",
            "name": "openlineage_test_bigquery_connector"
          }
        }
      },
      "spark_properties": {
        "_producer": "https://github.com/OpenLineage/OpenLineage/tree/1.35.0/integration/spark",
        "_schemaURL": "https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunFacet",
        "properties": {
          "spark.master": "local",
          "spark.app.name": "openlineage_test_bigquery_connector"
        }
      },
      "processing_engine": {
        "_producer": "https://github.com/OpenLineage/OpenLineage/tree/1.35.0/integration/spark",
        "_schemaURL": "https://openlineage.io/spec/facets/1-1-1/ProcessingEngineRunFacet.json#/$defs/ProcessingEngineRunFacet",
        "version": "3.5.0",
        "name": "spark",
        "openlineageAdapterVersion": "1.35.0"
      },
      "environment-properties": {
        "_producer": "https://github.com/OpenLineage/OpenLineage/tree/1.35.0/integration/spark",
        "_schemaURL": "https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunFacet",
        "environment-properties": {}
      }
    }
  },
  "job": {
    "namespace": "default",
    "name": "openlineage_test_bigquery_connector.execute_save_into_data_source_command.test_dataset_table666",
    "facets": {
      "jobType": {
        "_producer": "https://github.com/OpenLineage/OpenLineage/tree/1.35.0/integration/spark",
        "_schemaURL": "https://openlineage.io/spec/facets/2-0-3/JobTypeJobFacet.json#/$defs/JobTypeJobFacet",
        "processingType": "BATCH",
        "integration": "SPARK",
        "jobType": "SQL_JOB"
      }
    }
  },
  "inputs": [
    {
      "namespace": "file",
      "name": "test_data.csv",
      "facets": {
        "dataSource": {
          "_producer": "https://github.com/OpenLineage/OpenLineage/tree/1.35.0/integration/spark",
          "_schemaURL": "https://openlineage.io/spec/facets/1-0-1/DatasourceDatasetFacet.json#/$defs/DatasourceDatasetFacet",
          "name": "file",
          "uri": "file"
        },
        "schema": {
          "_producer": "https://github.com/OpenLineage/OpenLineage/tree/1.35.0/integration/spark",
          "_schemaURL": "https://openlineage.io/spec/facets/1-1-1/SchemaDatasetFacet.json#/$defs/SchemaDatasetFacet",
          "fields": [
            {
              "name": "id",
              "type": "integer"
            },
            {
              "name": "age",
              "type": "integer"
            },
            {
              "name": "location",
              "type": "string"
            }
          ]
        }
      },
      "inputFacets": {
        "inputStatistics": {
          "_producer": "https://github.com/OpenLineage/OpenLineage/tree/1.35.0/integration/spark",
          "_schemaURL": "https://openlineage.io/spec/facets/1-0-0/InputStatisticsInputDatasetFacet.json#/$defs/InputStatisticsInputDatasetFacet",
          "size": 84,
          "fileCount": 1
        }
      }
    }
  ],
  "outputs": [
    {
      "namespace": "bigquery",
      "name": "xxx.test_dataset.table666",
      "facets": {
        "dataSource": {
          "_producer": "https://github.com/OpenLineage/OpenLineage/tree/1.35.0/integration/spark",
          "_schemaURL": "https://openlineage.io/spec/facets/1-0-1/DatasourceDatasetFacet.json#/$defs/DatasourceDatasetFacet",
          "name": "bigquery",
          "uri": "bigquery"
        },
        "columnLineage": {
          "_producer": "https://github.com/OpenLineage/OpenLineage/tree/1.35.0/integration/spark",
          "_schemaURL": "https://openlineage.io/spec/facets/1-2-0/ColumnLineageDatasetFacet.json#/$defs/ColumnLineageDatasetFacet",
          "fields": {
            "id": {
              "inputFields": [
                {
                  "namespace": "file",
                  "name": "test_data.csv",
                  "field": "id",
                  "transformations": [
                    {
                      "type": "DIRECT",
                      "subtype": "IDENTITY",
                      "description": "",
                      "masking": false
                    }
                  ]
                }
              ]
            },
            "age": {
              "inputFields": [
                {
                  "namespace": "file",
                  "name": "test_data.csv",
                  "field": "age",
                  "transformations": [
                    {
                      "type": "DIRECT",
                      "subtype": "IDENTITY",
                      "description": "",
                      "masking": false
                    }
                  ]
                }
              ]
            },
            "location": {
              "inputFields": [
                {
                  "namespace": "file",
                  "name": "test_data.csv",
                  "field": "location",
                  "transformations": [
                    {
                      "type": "DIRECT",
                      "subtype": "IDENTITY",
                      "description": "",
                      "masking": false
                    }
                  ]
                }
              ]
            }
          }
        },
        "schema": {
          "_producer": "https://github.com/OpenLineage/OpenLineage/tree/1.35.0/integration/spark",
          "_schemaURL": "https://openlineage.io/spec/facets/1-1-1/SchemaDatasetFacet.json#/$defs/SchemaDatasetFacet",
          "fields": [
            {
              "name": "id",
              "type": "integer"
            },
            {
              "name": "age",
              "type": "integer"
            },
            {
              "name": "location",
              "type": "string"
            }
          ]
        }
      },
      "outputFacets": {}
    }
  ]
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind:bugSomething isn't workingstate:needs-triageRequires triage to determine kind, area and priority

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions