Skip to content

Transient error StatusCode.UNAVAILABLE encountered while exporting logs #3261

Closed

Description

@gilbertobr

I have an Opentelemetry that is running on Kubernetes on port 80 or 443.
https://artifacthub.io/packages/helm/opentelemetry-helm/opentelemetry-collector

But when I try to send a log via grpc using the python sdk (https://github.com/open-telemetry/opentelemetry-python), I am getting this warning, and the log is not sent:

WARNING:opentelemetry.exporter.otlp.proto.grpc.exporter:Transient error StatusCode.UNAVAILABLE encountered while exporting logs, retrying in 1s.
WARNING:opentelemetry.exporter.otlp.proto.grpc.exporter:Transient error StatusCode.UNAVAILABLE encountered while exporting logs, retrying in 2s.
WARNING:opentelemetry.exporter.otlp.proto.grpc.exporter:Transient error StatusCode.UNAVAILABLE encountered while exporting logs, retrying in 4s.
WARNING:opentelemetry.exporter.otlp.proto.grpc.exporter:Transient error StatusCode.UNAVAILABLE encountered while exporting logs, retrying in 8s.
WARNING:opentelemetry.exporter.otlp.proto.grpc.exporter:Transient error StatusCode.UNAVAILABLE encountered while exporting logs, retrying in 16s.

When I send the LOG via POSTMAN, I can send it normally.
Have you been through this?

script test:

import logging

from opentelemetry import trace
from opentelemetry._logs import set_logger_provider
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import (
    OTLPLogExporter,
)
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import (
    BatchSpanProcessor,
    ConsoleSpanExporter,
)

trace.set_tracer_provider(TracerProvider())
trace.get_tracer_provider().add_span_processor(
    BatchSpanProcessor(ConsoleSpanExporter())
)

logging.basicConfig(level=logging.DEBUG)

logger_provider = LoggerProvider(
    resource=Resource.create(
        {
            "service.name": "shoppingcart",
            "service.instance.id": "instance-12",
        }
    ),
)
set_logger_provider(logger_provider)

exporter = OTLPLogExporter(endpoint="grpc.otel-collector.example.com:80", insecure=True, timeout=20)
logger_provider.add_log_record_processor(BatchLogRecordProcessor(exporter))
handler = LoggingHandler(level=logging.NOTSET, logger_provider=logger_provider)

# Attach OTLP handler to root logger
logging.getLogger().addHandler(handler)

# Log directly
logging.info("Jackdaws love my big sphinx of quartz.")

# Create different namespaced loggers
logger1 = logging.getLogger("myapp.area1")
logger2 = logging.getLogger("myapp.area2")

logger1.debug("Quick zephyrs blow, vexing daft Jim.")
logger1.info("How quickly daft jumping zebras vex.")
logger2.warning("Jail zesty vixen who grabbed pay from quack.")
logger2.error("The five boxing wizards jump quickly.")


# Trace context correlation
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("foo"):
    # Do something
    logger2.error("Hyderabad, we have a major problem.")

logger_provider.shutdown()

pip show

backoff                                2.2.1
grpcio                                 1.53.0
grpcio-tools                           1.53.0
opentelemetry-api                      1.17.0
opentelemetry-exporter-otlp            1.17.0
opentelemetry-exporter-otlp-proto-grpc 1.17.0
opentelemetry-exporter-otlp-proto-http 1.17.0
opentelemetry-instrumentation          0.38b0
opentelemetry-instrumentation-grpc     0.38b0
opentelemetry-proto                    1.17.0
opentelemetry-sdk                      1.17.0
opentelemetry-semantic-conventions     0.38b0

Activity

  1. srikanthccv commented on Apr 15, 2023

    @srikanthccv
    Member

    But when I try to send a log via grpc using the python sdk (https://github.com/open-telemetry/opentelemetry-python), I am getting this warning, and the log is not sent

    When I send the LOG via POSTMAN, I can send it normally.
    Have you been through this?

    When you send from POSTMAN you are most likely sending using HTTP. The example shared uses gRPC. Please make sure you use the gRPC port.

  2. added
    questionFurther information is requested
    and removed
    bugSomething isn't working
    on Apr 15, 2023
  3. gilbertobr commented on Apr 17, 2023

    @gilbertobr
    Author

    Exactly, for those who need to do GRPC tests via POSTMAN, follow below how to configure.

    https://blog.postman.com/postman-now-supports-grpc/

  4. srikanthccv commented on Apr 17, 2023

    @srikanthccv
    Member

    Can I assume your question is resolved and close this?

  5. gilbertobr commented on Apr 18, 2023

    @gilbertobr
    Author

    I believe the problem is not that.
    And yes in the route that the sdk is making the request.
    But I'm still investigating.

    I noticed that there is a closed PR in the opentelemetry-collector project, but not resolved.
    The problem has followed with other people.

  6. srikanthccv commented on Apr 19, 2023

    @srikanthccv
    Member

    The example uses the gRPC exporter, which will not work with the HTTP endpoint. Either use the HTTP exporter or give a valid gRPC endpoint. This is not an issue with the SDK/Exporter but rather config/ network. Since this is not a bug and there is no action item, I am converting this to a GitHub discussion.

  7. locked and limited conversation to collaborators on Apr 19, 2023
  8. converted this issue into a discussion #3266 on Apr 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    questionFurther information is requested

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions