This repository was archived by the owner on Feb 26, 2026. It is now read-only.
This repository was archived by the owner on Feb 26, 2026. It is now read-only.
If VM metadata fetch fails in google.cloud.logging.Client.setup_logging(), all logging starts to fail #710
Description
Environment details
- OS type and version: Google Cloud Container Optimized OS (gke-12314-gke401-cos-93-16623-295-14-v221129-c-pre)
- Python version: 3.8.10
- pip version: 22.3.1
google-cloud-loggingversion: 3.4.0
Steps to reproduce
- Hack retrieve_metadata_server() so that it returns
testwhen providedmetadata_keyin the set{'instance/id', 'instance/zone'}andNonefor all other values. This can happen if a metadata server request fails. - Try to log something and see unhandled exceptions from the logging thread.
Code example
test.py
import logging
import google.cloud.logging
import google.cloud.logging_v2.handlers._monitored_resources
# Fake implementation of
# google.cloud.logging_v2._helpers.retrieve_metadata_server()
# that simulates a metadata server error for all keys except
# GCE instance ID and zone.
def _fake_retrieve_metadata_server(metadata_key, timeout=5):
if metadata_key in {'instance/id', 'instance/zone'}:
return 'test'
return None
# Patch the reference to
# google.cloud.logging_v2._helpers.retrieve_metadata_server
# imported in google.cloud.logging_v2.handlers._monitored_resources).
(google.cloud.logging_v2.handlers._monitored_resources.
retrieve_metadata_server) = _fake_retrieve_metadata_server
client = google.cloud.logging.Client()
client.setup_logging()
logging.info('hello world')Run python test.py
Stack trace
Program shutting down, attempting to send 1 queued log entries to Cloud Logging...
Waiting up to 5 seconds.
Failed to submit 1 logs.
Traceback (most recent call last):
File "/home/stewart/src/agentic/service/.venv/lib/python3.8/site-packages/google/protobuf/json_format.py", line 550, in _ConvertFieldValuePair
self._ConvertMapFieldValue(value, message, field)
File "/home/stewart/src/agentic/service/.venv/lib/python3.8/site-packages/google/protobuf/json_format.py", line 697, in _ConvertMapFieldValue
getattr(message, field.name)[key_value] = _ConvertScalarFieldValue(
File "/home/stewart/src/agentic/service/.venv/lib/python3.8/site-packages/google/protobuf/json_format.py", line 733, in _ConvertScalarFieldValue
if _UNPAIRED_SURROGATE_PATTERN.search(value):
TypeError: expected string or bytes-like object
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/stewart/src/agentic/service/.venv/lib/python3.8/site-packages/google/protobuf/json_format.py", line 580, in _ConvertFieldValuePair
self.ConvertMessage(value, sub_message)
File "/home/stewart/src/agentic/service/.venv/lib/python3.8/site-packages/google/protobuf/json_format.py", line 476, in ConvertMessage
self._ConvertFieldValuePair(value, message)
File "/home/stewart/src/agentic/service/.venv/lib/python3.8/site-packages/google/protobuf/json_format.py", line 594, in _ConvertFieldValuePair
raise ParseError('Failed to parse {0} field: {1}.'.format(name, e))
google.protobuf.json_format.ParseError: Failed to parse labels field: expected string or bytes-like object.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/stewart/src/agentic/service/.venv/lib/python3.8/site-packages/google/cloud/logging_v2/handlers/transports/background_thread.py", line 115, in _safely_commit_batch
batch.commit()
File "/home/stewart/src/agentic/service/.venv/lib/python3.8/site-packages/google/cloud/logging_v2/logger.py", line 467, in commit
client.logging_api.write_entries(
File "/home/stewart/src/agentic/service/.venv/lib/python3.8/site-packages/google/cloud/logging_v2/_gapic.py", line 154, in write_entries
log_entry_pbs = [_log_entry_mapping_to_pb(entry) for entry in entries]
File "/home/stewart/src/agentic/service/.venv/lib/python3.8/site-packages/google/cloud/logging_v2/_gapic.py", line 154, in <listcomp>
log_entry_pbs = [_log_entry_mapping_to_pb(entry) for entry in entries]
File "/home/stewart/src/agentic/service/.venv/lib/python3.8/site-packages/google/cloud/logging_v2/_gapic.py", line 564, in _log_entry_mapping_to_pb
ParseDict(mapping, entry_pb)
File "/home/stewart/src/agentic/service/.venv/lib/python3.8/site-packages/google/protobuf/json_format.py", line 445, in ParseDict
parser.ConvertMessage(js_dict, message)
File "/home/stewart/src/agentic/service/.venv/lib/python3.8/site-packages/google/protobuf/json_format.py", line 476, in ConvertMessage
self._ConvertFieldValuePair(value, message)
File "/home/stewart/src/agentic/service/.venv/lib/python3.8/site-packages/google/protobuf/json_format.py", line 588, in _ConvertFieldValuePair
raise ParseError('Failed to parse {0} field: {1}.'.format(name, e))
google.protobuf.json_format.ParseError: Failed to parse resource field: Failed to parse labels field: expected string or bytes-like object..
Sent all pending logs.