2.21.2

Bug Fixes

Internal:

• Fix performance overhead of Python distribution parsing for internal telemetry.

ASM:

• This fix resolves an issue where blocking mechanism could partially fail with a 500 error on fastapi with python>=3.11 with a custom middleware.

Library Injection:

• Address partial library copy failures causing failed application startups in k8s.
• Removes boto3 from the minimum versions list to avoid blocking SSI on boto3 version.

2.20.2

Bug Fixes

• ASM

  • Ensures that no module from ASM are loaded when ASM is disabled or unavailable.
  • Resolves an issue where IAST modules could be loaded, even if disabled, which could create an ImportError exception on Windows.

• Code Security

  • SCA: Ensures that no module from IAST are loaded when IAST is disabled or unavailable.
  • Runtime Code Analysis (IAST): Avoids imports of IAST native module when IAST is not enabled.

• Tracing

  • internal: Fixes performance overhead of Python distribution parsing for internal telemetry.

2.21.1

Bug Fixes

• ASM

  • Ensure that no module from ASM are loaded when ASM is disabled or unavailable.

• Code Security

  • Runtime Code Analysis (IAST): Avoid imports of IAST native module when IAST is not enabled.
  • SCA: This ensures that no module from IAST are loaded when IAST is disabled or unavailable.
  • Resolve an issue where IAST modules could be loaded, even if disabled, which could create an ImportError exception on Windows.

• Profiling

  • Fixes an issue where the profiler unnecessarily outputs log messages when it was unable to record a lock acquire event.

• Tracing

  • aiohttp: Adds the environment variable DD_AIOHTTP_CLIENT_DISABLE_STREAM_TIMING_FOR_MEM_LEAK to address a potential memory leak in the aiohttp integration. When set to true, this flag may cause streamed response span timing to be inaccurate. The flag defaults to false.
  • span: Fix issue where spans weren't being handled correctly and were not being sent when using a custom Exception class that raises an exception in __str__.

2.19.4

Bug Fixes

• ASM: This ensures that no module from ASM are loaded when ASM is disabled or unavailable.

• SCA: This ensures that no module from IAST are loaded when IAST is disabled or unavailable.

• Runtime Code Analysis (IAST): Avoid imports of IAST native module when IAST is not enabled.

• ASM: This fix resolves an issue where IAST modules could be loaded, even if disabled,
  which could create an ImportError exception on Windows.

3.0.0

Migration Guide: Upgrading from dd-trace-py v2 to v3

We are happy to announce the release of v3.0.0 of ddtrace. This release drops support for Python 3.7 and many previously-deprecated parts of the library interface.

The 2.x release line enters Maintenance Mode with the release of 3.0.0, meaning that it will only receive bugfix changes on its last few minor releases, and 2.21 will be the last 2.x minor version. See the versioning policy for support level definitions.

---

Summary of Significant Changes

• Removed support for Python 3.7 from the library

CI Visibility

• The new pytest plugin is now the default.
• Module, suite, and test names are now parsed from item.nodeid.
• Test names now include the class name (TestClass::test_method) for class-based tests.
• Test skipping is now done at the suite level.
• The DD_PYTEST_USE_NEW_PLUGIN_BETA environment variable is no longer used.

LLM Observability

• Removed support for DD_LLMOBS_APP_NAME.
• The parameters argument in LLMObs.annotate() has been removed. Use metadata instead.
• Dropped support for OpenAI 0.x and Langchain v0.0.x.

Tracing

• Removed all deprecated interfaces from integrations.
• DD_TRACE_PROPAGATION_STYLE=b3 single header is no longer supported. Use b3 instead.
• Dropped support for multiple Tracer instances. The global tracer instance ddtrace.tracer must be used.
• Removed deprecated parameters from Tracer.configure(...).
• Dropped support for several deprecated tracing configurations. See “Environment Variables” section below for details.

Recommended Pre-Upgrade Steps

Before upgrading to v3, check for deprecation warnings in your application using the latest v2 release (2.21.0):

Detect Deprecation Warnings in Tests

Enable warnings as errors in pytest:

pytest -W "error::ddtrace.DDTraceDeprecationWarning" tests.py

Detect Deprecation Warnings in Applications

Set the following environment variable to raise exceptions for deprecated usage:

DD_TRACE_RAISE_DEPRECATIONWARNING=1 ddtrace-run python app.py

Alternatively, enable all deprecation warnings:

PYTHONWARNINGS=all python app.py

Environment Variables

The following environment variables have been removed or replaced in v3. Use the table below to update your configuration:

v2	Removed in v3	Replaced/Changed in v3
LLM Observability
DD_LLMOBS_APP_NAME		Use DD_LLMOBS_ML_APP instead
_DD_LLMOBS_EVALUATOR_SAMPLING_RULES		Use DD_LLMOBS_EVALUATOR_SAMPLING_RULES instead
_DD_LLMOBS_EVALUATORS		Use DD_LLMOBS_EVALUATORS instead
Tracing
DD_ANALYTICS_ENABLED	No replacement, this setting is now a no-op. Datadog Analytics is no longer supported. See the ingestion controls documentation for more detail.
DD_HTTP_CLIENT_TAG_QUERY_STRING		Use DD_TRACE_HTTP_CLIENT_TAG_QUERY_STRING.
DD_TRACE_ANALYTICS_ENABLED	No replacement, this setting is now a no-op. Datadog Analytics is no longer supported. See the ingestion controls documentation for more detail.
DD_TRACE_METHODS (using [] notation)		Must use : notation instead, For example: mod.submod:method1,method2;mod.submod:Class.method1. (docs)
DD_TRACE_PROPAGATION_STYLE=”b3 single header”		Use DD_TRACE_PROPAGATION_STYLE=b3 for identical behavior
DD_TRACE_SAMPLE_RATE		Use DD_TRACE_SAMPLING_RULES (docs).
DD_TRACE_SPAN_AGGREGATOR_RLOCK	No replacement (feature removed). Config variable is now a no-op.
CI Visibility
DD_PYTEST_USE_NEW_PLUGIN_BETA	No replacement (new pytest plugin is not in beta anymore).

Use the following patterns to identify the deprecated environment variables in a code base:

git grep -P -e "DD_LLMOBS_APP_NAME" \
  -e "_DD_LLMOBS_EVALUATOR_SAMPLING_RULES" \
  -e "_DD_LLMOBS_EVALUATORS" \
  -e "DD_TRACE_PROPAGATION_STYLE=.*b3 single header" \
  -e "DD_TRACE_SAMPLE_RATE" \
  -e "DD_TRACE_API_VERSION=v0.3" \
  -e "DD_ANALYTICS_ENABLED" \
  -e "DD_TRACE_ANALYTICS_ENABLED" \
  -e "DD_HTTP_CLIENT_TAG_QUERY_STRING" \
  -e "DD_TRACE_SPAN_AGGREGATOR_RLOCK" \
  -e "DD_TRACE_METHODS=.*\[\]"

Replace deprecated settings with the recommended alternatives where applicable.

NOTE: The changes to environment variables apply only to the configuration of the dd-trace-py library and not the Datadog Agent.

Legacy Interfaces

The following methods and module attributes have been removed or changed. Unless noted, removal means that the functionality pointed to by the removed interface has been disabled.

v2	v3
Tracing
Tracer.configure( enabled=foo, hostname=foo, port=foo, uds_path=foo, https=foo, sampler=foo, wrap_executor=foo, priority_sampling=foo, settings=foo, dogstatsd_url=foo, writer=foo, partial_flush_enabled=foo, partial_flush_min_spans=foo, api_version=foo )	Tracer.configure()
ddtrace.opentracer.tracer.Tracer()._dd_tracer	ddtrace.tracer
ddtrace.tracer.sampler.rules[].choose_matcher(lambda: True)	ddtrace.tracer.sampler.rules[].choose_matcher(“foo”) Callables and regex patterns are no longer allowed as arguments to choose_matcher
Span.sampled	Span.context.sampling_priority > 0
LLM Observability
LLMObs.annotate(parameters=foo)	LLMObs.annotate(metadata=foo)

Python 3.13 Support

Much of the library’s functionality has been compatible with Python 3.13 since version 2.20. Some pieces of functionality do not yet work with Python 3.13. Here’s what does not work under Python 3.13 as of dd-trace-py 3.0. 'Not tested' indicates that while the feature may be compatible with Python 3.13, its functionality has not been verified, as it is not included in our automated test coverage.

• dd-trace-py doesn’t work on Windows with Python 3.13
• Appsec Threat Detection is not tested against Django, Flask, or FastAPI with 3.13
• Automatic Service Naming is not tested with 3.13
• The following products are not tested with 3.13:
  • Code Coverage
  • Appsec IAST
  • Data Streams Monitoring
  • CI Visibility
  • Continuous Profiling
• The following integrations are not tested with 3.13:
  • anthropic
  • consul
  • freezegun
  • gevent
  • google_generativeai
  • gunicorn
  • langchain
  • mysqlclient
  • opentracing
  • psycopg
  • psycopg2
  • pymysql
  • pytest
  • pytest-bdd
  • pytest-benchmark
  • sanic
  • selenium
  • sqlite3
  • starlette
  • tornado
  • vertexai

---

By following these guidelines, you can transition to dd-trace-py v3 gradually, ensuring minimal disruption to your services while taking advantage of the latest features and improvements. Happy migrating!

---

New Features

• Removes code and tests related to Python 3.7, breaking the library's compatibility with this Python version.

• ASM

  • Introduces use of meta struct for appsec event reports. This will improve performances. ASM will now require at least agent version 7.35.0 to work as expected.

• Code Security (IAST)

  • Introduces Code Security support to PyMySQL and MySQLDB integrations.
  • XSS detection for Django applications and Jinja2 (Flask and FastAPI applications), which will be displayed on your DataDog Vulnerability Explorer dashboard. See the Application Vulnerability Management documentation for more information about this feature.
  • Implements the detection of the Stacktrace-Leak vulnerability for Django, Flask and FastAPI.

• LLM Observability

  • valkey: Adds automatic instrumentation of the Valkey package. Thank you AhmadMasry!

  • openai: Introduces tracing support to the OpenAI integration for Python versions 3.12 and 3.13.

  • Introduces an integration with the RAGAS evaluation framework to continuously monitor the performance of context-augmented LLM generations in production.
    The integration supports evaluating LLM inferences with the following RAGAS metrics:

    • Faithfulness: measures if the LLM response is faithful to the provided context.
    • Answer Relevancy: measures how relevant the LLM response is to the user input.
    • Context Precision: measures how effectively the context is used in the generated response.

    To learn more, see the LLM Observability evaluations guide.

Upgrade Notes

• CI Visibility
  • Official release of the new version of the pytest plugin, introducing the following features:
    • Auto Test Retries
    • Early Flake Detection
    • Improved coverage collection for ...

3.0.0rc3

Bug Fixes

• ASM

  • Ensures that no module from ASM are loaded when ASM is disabled or unavailable.

• Code Security

  • SCA: This ensures that no module from IAST are loaded when IAST is disabled or unavailable.

• Tracing

  • internal: Fixes performance overhead of Python distribution parsing for internal telemetry.

2.21.0

New Features

• LLM Observability
  • openai: Introduces tracing support to the OpenAI integration for Python versions 3.12 and 3.13.

Deprecation Notes

• Tracing
  • Moves ddtrace.provider.BaseContextProvider to ddtrace.trace.BaseContextProvider. The ddtrace.provider module is deprecated and will be removed in v3.0.0.
  • Deprecates the following constants in ddtrace.constants module:
    • ANALYTICS_SAMPLE_RATE_KEY
    • SAMPLE_RATE_METRIC_KEY
    • SAMPLING_PRIORITY_KEY
    • SAMPLING_AGENT_DECISION
    • SAMPLING_RULE_DECISION
    • SAMPLING_LIMIT_DECISION
    • ORIGIN_KEY
    • USER_ID_KEY
    • HOSTNAME_KEY
    • RUNTIME_FAMILY
    • BASE_SERVICE_KEY
    • SPAN_MEASURED_KEY
    • KEEP_SPANS_RATE_KEY
    • MULTIPLE_IP_HEADERS
    • CONFIG_ENDPOINT_ENV
    • CONFIG_ENDPOINT_RETRIES_ENV
    • CONFIG_ENDPOINT_TIMEOUT_ENV
  • Internalizes the ddtrace.settings.config module and deprecates the following ddtrace.config attributes:
    • http, use DD_TRACE_HEADER_TAGS environment variable instead.
    • http_server, use DD_TRACE_HTTP_SERVER_ERROR_STATUSES environment variable instead.
    • trace_headers, this attribute is internal to the tracer.
    • header_is_traced, this attribute is internal to the tracer.
    • convert_rc_trace_sampling_rules, this attribute is internal to the tracer.
    • enable_remote_configuration, use DD_REMOTE_CONFIGURATION_ENABLED environment variable instead.
    • get_from, use ddtrace.trace.Pin to set instance level configurations.

Bug Fixes

• CI Visibility

  • Fixes an issue where Auto Test Retries with pytest would always consider retries of tests defined inside unittest classes to be successful.

• Code Security

  • Resolves an issue where the usage of callonce decorator could trigger an import loop

• LLM Observability

  • Resolves an issue where explicitly only using LLMObs.enable() to configure LLM Observability without environment variables would not automatically propagate distributed tracing headers.
  • Resolves an issue where annotating a span with non latin-1 (but valid utf-8) input/output values resulted in encoding errors.
  • Resolves an issue where extracting token metadata from openai streamed chat completion token chunks caused an IndexError
  • openai: Fixes a patching issue where asynchronous moderation endpoint calls resulted in coroutine scheduling errors.
  • vertexai: Resolves an issue with chat.send_message() where the content keyword argument was not parsed correctly.

• Profiling

  • Fixes SystemError from the memory profiler returning NULL when collecting events

Other Changes

• Tracing
  • Ensures the ddtrace library does not use deprecated APIs internally. Deprecation warnings should only be logged when the user's code is using deprecated APIs.
  • cassandra, cherrypy, flask_cache, starlette: Ensures a deprecation warning is not raised when patching these integrations via ddtrace-run and import ddtrace.auto.

2.21.0rc1

New Features

• LLM Observability
  • openai: Introduces tracing support to the OpenAI integration for Python versions 3.12 and 3.13.

Deprecation Notes

• Tracing
  • Moves ddtrace.provider.BaseContextProvider to ddtrace.trace.BaseContextProvider. The ddtrace.provider module is deprecated and will be removed in v3.0.0.
  • Deprecates the following constants in ddtrace.constants module:
    • ANALYTICS_SAMPLE_RATE_KEY
    • SAMPLE_RATE_METRIC_KEY
    • SAMPLING_PRIORITY_KEY
    • SAMPLING_AGENT_DECISION
    • SAMPLING_RULE_DECISION
    • SAMPLING_LIMIT_DECISION
    • ORIGIN_KEY
    • USER_ID_KEY
    • HOSTNAME_KEY
    • RUNTIME_FAMILY
    • BASE_SERVICE_KEY
    • SPAN_MEASURED_KEY
    • KEEP_SPANS_RATE_KEY
    • MULTIPLE_IP_HEADERS
    • CONFIG_ENDPOINT_ENV
    • CONFIG_ENDPOINT_RETRIES_ENV
    • CONFIG_ENDPOINT_TIMEOUT_ENV
  • Internalizes the ddtrace.settings.config module and deprecates the following ddtrace.config attributes:
    • http, use DD_TRACE_HEADER_TAGS environment variable instead.
    • http_server, use DD_TRACE_HTTP_SERVER_ERROR_STATUSES environment variable instead.
    • trace_headers, this attribute is internal to the tracer.
    • header_is_traced, this attribute is internal to the tracer.
    • convert_rc_trace_sampling_rules, this attribute is internal to the tracer.
    • enable_remote_configuration, use DD_REMOTE_CONFIGURATION_ENABLED environment variable instead.
    • get_from, use ddtrace.trace.Pin to set instance level configurations.

Bug Fixes

• CI Visibility

  • Fixes an issue where Auto Test Retries with pytest would always consider retries of tests defined inside unittest classes to be successful.

• Code security

  • Resolves an issue where the usage of callonce decorator could trigger an import loop

• LLM Observability

  • Resolves an issue where explicitly only using LLMObs.enable() to configure LLM Observability without environment variables would not automatically propagate distributed tracing headers.
  • Resolves an issue where annotating a span with non latin-1 (but valid utf-8) input/output values resulted in encoding errors.
  • Resolves an issue where extracting token metadata from openai streamed chat completion token chunks caused an IndexError.
  • openai: Fixes a patching issue where asynchronous moderation endpoint calls resulted in coroutine scheduling errors.
  • vertexai: Resolves an issue with chat.send_message() where the content keyword argument was not parsed correctly.

• Profiling

  • Fixes SystemError from the memory profiler returning NULL when collecting events

Other Changes

• Tracing
  • Ensures the ddtrace library does not use deprecated APIs internally. Deprecation warnings should only be logged when the user's code is using deprecated APIs.
• cassandra,cherrypy,flask_cache,starlette: Ensures a deprecation warning is not raised when patching these integrations via ddtrace-run and import ddtrace.auto.

2.20.1

Deprecation Notes

• Tracing
  • Moves ddtrace.provider.BaseContextProvider to ddtrace.trace.BaseContextProvider. The ddtrace.provider module is deprecated and will be removed in v3.0.0.
  • Deprecates the following constants in ddtrace.constants module:
    • ANALYTICS_SAMPLE_RATE_KEY
    • SAMPLE_RATE_METRIC_KEY
    • SAMPLING_PRIORITY_KEY
    • SAMPLING_AGENT_DECISION
    • SAMPLING_RULE_DECISION
    • SAMPLING_LIMIT_DECISION
    • ORIGIN_KEY
    • USER_ID_KEY
    • HOSTNAME_KEY
    • RUNTIME_FAMILY
    • BASE_SERVICE_KEY
    • SPAN_MEASURED_KEY
    • KEEP_SPANS_RATE_KEY
    • MULTIPLE_IP_HEADERS
    • CONFIG_ENDPOINT_ENV
    • CONFIG_ENDPOINT_RETRIES_ENV
    • CONFIG_ENDPOINT_TIMEOUT_ENV
  • Internalizes the ddtrace.settings.config module and deprecates the following ddtrace.config attributes:
    • http, use DD_TRACE_HEADER_TAGS environment variable instead.
    • http_server, use DD_TRACE_HTTP_SERVER_ERROR_STATUSES environment variable instead.
    • trace_headers, this attribute is internal to the tracer.
    • header_is_traced, this attribute is internal to the tracer.
    • convert_rc_trace_sampling_rules, this attribute is internal to the tracer.
    • enable_remote_configuration, use DD_REMOTE_CONFIGURATION_ENABLED environment variable instead.
    • get_from, use ddtrace.trace.Pin to set instance level configurations.

Bug Fixes

• CI Visibility
  • Fixes an issue where Auto Test Retries with pytest would always consider retries of tests defined inside unittest classes to be successful.
• LLM Observability
  • Resolves an issue where explicitly only using LLMObs.enable() to configure LLM Observability without environment variables would not automatically propagate distributed tracing headers.
  • This fix resolves an issue where annotating a span with non latin-1 (but valid utf-8) input/output values resulted in encoding errors.
  • This fix resolves an issue where extracting token metadata from openai streamed chat completion token chunks caused an IndexError.
• Tracing
  • vertexai: Resolves an issue with chat.send_message() where the content keyword argument was not parsed correctly.
• Profiling
  • Fix SystemError from the memory profiler returning NULL when collecting events

Other Changes

• Tracing
  • Ensures the ddtrace library does not use deprecated APIs internally. Deprecation warnings should only be logged when the user's code is using deprecated APIs.
  • cassandra, cherrypy ,flask_cache, starlette: Ensures a deprecation warning is not raised when patching these integrations via ddtrace-run and import ddtrace.auto.

3.0.0rc2

Upgrade Notes

• LLM Observability

  • langchain: Removes the langchain.tokens.total_cost span metric for OpenAI calls. For continued cost estimation of OpenAI calls, enable LLM Observability.
  • langchain: Removes prompt-completion log sampling from the LangChain integration. To continue logging prompt completions, enable LLM Observability.
  • langchain: Removes integration metrics from the LangChain integration. To continue tracking operational metrics from the OpenAI integration, enable LLM Observability or use trace metrics instead.
  • openai: Removes prompt-completion log sampling from the OpenAI integration. To continue logging prompt completions, enable LLM Observability.
  • openai: Removes integration metrics from the OpenAI integration. To continue tracking operational metrics from the OpenAI integration, enable LLM Observability or use trace metrics instead.

• Tracing

  • Drops support for multiple Tracer instances in the same process. Use ddtrace.trace.tracer to access the global tracer instance.
  • Drops support for multiple tracer instances, ddtrace.trace.Tracer can not be reinitialized.
  • Removes deprecated parameters from Tracer.configure(...) method and removes the Tracer.sampler attribute.
  • configurations: Drops support for deprecated tracing configurations. The following configurations are no longer supported:
    • DD_TRACE_SAMPLE_RATE, use DD_TRACE_SAMPLING_RULES instead.
    • DD_TRACE_API_VERSION=v0.3, the default v0.5 version is used instead.
    • DD_ANALYTICS_ENABLED, Datadog Analytics is no longer supported.
    • DD_TRACE_ANALYTICS_ENABLED, Datadog Analytics is no longer supported.
    • DD_HTTP_CLIENT_TAG_QUERY_STRING, DD_TRACE_HTTP_CLIENT_TAG_QUERY_STRING should be used instead.
    • DD_TRACE_SPAN_AGGREGATOR_RLOCK, disabling the span aggregator rlock is no longer supported.
  • configurations: Drops support for configuring DD_TRACE_METHODS with the [] notation. Ensure DD_TRACE_METHODS use the : notation instead.
  • opentracing: Removes the deprecated ddtracer parameter from ddtrace.opentracer.tracer.Tracer().
  • sampling: Drops support for configuring sampling rules using functions and regex in the ddtrace.tracer.sampler.rules[].choose_matcher(...) method and removes the timestamp_ns parameter from ddtrace.internal.rate_limiter.RateLimiter.is_allowed().
  • span: Removes the deprecated Span.sampled property.

New Features

• Tracing
  • valkey: Adds automatic instrumentation of the Valkey package. Thank you AhmadMasry!

Bug Fixes

• Tracing
  • aiohttp: Adds the environment variable DD_AIOHTTP_CLIENT_DISABLE_STREAM_TIMING_FOR_MEM_LEAK to address a potential memory leak in the aiohttp integration. When set to true, this flag may cause streamed response span timing to be inaccurate. The flag defaults to false.
  • logging: Resolves an an unneeded info log being logged on process exit due to a forksafe hook being unregistered that was never registered to begin with.