Repository navigation
BigQuery: create_dataset, create_table and create_routine with exists_ok=True ignore timeout on the lookup after a 409 #18766
Description
Determine this is the right repository
- I determined this is the correct repository in which to report this bug.
Summary of the issue
Context
I call create_dataset, create_table or create_routine with exists_ok=True and a timeout, so the call stays bounded when the resource already exists.
Expected Behavior:
When the insert returns 409 and the client looks up the existing resource, that lookup uses the same timeout.
Actual Behavior:
The lookup runs with no timeout at all. The fallback in client.py calls get_dataset, get_table or get_routine with retry but without timeout, and BigQuery's DEFAULT_TIMEOUT is None, so a slow or stuck GET can block the caller far past the timeout it asked for.
API client name and version
google-cloud-bigquery 3.46.2 (main at 57366ee)
Reproduction steps: code
file: main.py
import time
from google.auth.credentials import AnonymousCredentials
from google.cloud import bigquery
# A local stand in for the REST API: every insert answers 409, every GET waits 10 seconds.
client = bigquery.Client(
project="p",
credentials=AnonymousCredentials(),
client_options={"api_endpoint": "http://127.0.0.1:8794"},
)
start = time.time()
client.create_table("p.d.t", exists_ok=True, timeout=2, retry=None)
print(f"returned after {time.time() - start:.1f}s")Reproduction steps: supporting files
file: fake_bq.py
import json, time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
class H(BaseHTTPRequestHandler):
protocol_version = "HTTP/1.1"
def _send(self, code, body):
data = json.dumps(body).encode()
self.send_response(code); self.send_header("content-type", "application/json")
self.send_header("content-length", str(len(data))); self.end_headers(); self.wfile.write(data)
def do_POST(self):
self.rfile.read(int(self.headers.get("content-length") or 0))
self._send(409, {"error": {"code": 409, "message": "Already Exists", "errors": [{"reason": "duplicate"}]}})
def do_GET(self):
time.sleep(10)
parts = self.path.split("?")[0].strip("/").split("/")
self._send(200, {"tableReference": {"projectId": parts[3], "datasetId": parts[5], "tableId": parts[7]}})
def log_message(self, *a): pass
ThreadingHTTPServer(("127.0.0.1", 8794), H).serve_forever()Reproduction steps: actual results
returned after 10.0s
The same happens for create_dataset (10.1s) and create_routine (10.0s).
Reproduction steps: expected results
The lookup gives up after 2 seconds, like the insert does:
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='127.0.0.1', port=8794): Read timed out. (read timeout=2)
OS & version + platform
Linux 7.0 x86_64
Python environment
Python 3.12.3
Python dependencies
google-api-core 2.42.0
google-auth 2.62.0
google-cloud-bigquery 3.46.2
google-cloud-core 2.8.0
requests 2.34.2
Additional context
The fix is to pass timeout=timeout to the three fallback lookups. I have a PR with tests ready and will link it here.
Activity