pax_global_header00006660000000000000000000000064136315355760014530gustar00rootroot0000000000000052 comment=6b3dab3f4ab5c3b0c9e8885660202f065e7f2c69 datadog-lambda-python-14/000077500000000000000000000000001363153557600154565ustar00rootroot00000000000000datadog-lambda-python-14/.circleci/000077500000000000000000000000001363153557600173115ustar00rootroot00000000000000datadog-lambda-python-14/.circleci/config.yml000066400000000000000000000025121363153557600213010ustar00rootroot00000000000000version: 2.1 jobs: verify_build: working_directory: ~/repo docker: - image: circleci/python:<< parameters.python >> parameters: python: type: string steps: - checkout - restore_cache: keys: - v1-dependencies-<< parameters.python >>-{{ checksum "setup.py" }} # fallback to using the latest cache if no exact match is found - v1-dependencies-<< parameters.python >>-- - run: name: Install Dependencies command: | pip install virtualenv virtualenv venv source venv/bin/activate pip install .[dev] - run: name: Run Unit Tests command: | source venv/bin/activate nose2 -v - run: name: Lint command: | source venv/bin/activate flake8 datadog_lambda/ - save_cache: paths: - ./venv key: v1-dependencies-<< parameters.python >>-{{ checksum "setup.py" }} workflows: python-v2.7: jobs: - verify_build: python: "2.7" python-v3.6: jobs: - verify_build: python: "3.6" python-v3.7: jobs: - verify_build: python: "3.7" python-v3.8: jobs: - verify_build: python: "3.8" datadog-lambda-python-14/.flake8000066400000000000000000000000361363153557600166300ustar00rootroot00000000000000[flake8] max-line-length = 100datadog-lambda-python-14/.github/000077500000000000000000000000001363153557600170165ustar00rootroot00000000000000datadog-lambda-python-14/.github/ISSUE_TEMPLATE/000077500000000000000000000000001363153557600212015ustar00rootroot00000000000000datadog-lambda-python-14/.github/ISSUE_TEMPLATE/bug_report_template.md000066400000000000000000000004321363153557600255650ustar00rootroot00000000000000--- name: Bug Report about: Create a report to help us improve --- ## Expected Behavior ## Actual Behavior ## Steps to Reproduce the Problem 1. 1. 1. ## Specifications - Datadog Lambda Layer version: - Python version: ## Stacktrace ``` Paste here ```datadog-lambda-python-14/.github/ISSUE_TEMPLATE/feature_request_template.md000066400000000000000000000004141363153557600266200ustar00rootroot00000000000000--- name: Feature Request about: Suggest an idea --- ## Expected Behavior ## Actual Behavior ## Steps to Reproduce the Problem 1. 1. 1. ## Specifications - Datadog Lambda Layer version: - Python version: ## Stacktrace ``` Paste here ```datadog-lambda-python-14/.github/pull_request_template.md000066400000000000000000000007231363153557600237610ustar00rootroot00000000000000*Note: Please remember to review the [contribution guidelines](https://github.com/DataDog/datadog-lambda-layer-python/blob/master/CONTRIBUTING.md) if you have not yet done so.* ### What does this PR do? A brief description of the change being made with this pull request. ### Motivation What inspired you to submit this pull request? ### Testing Guidelines How did you test this pull request? ### Additional Notes Anything else we should know when reviewing? datadog-lambda-python-14/.gitignore000066400000000000000000000004601363153557600174460ustar00rootroot00000000000000*.py[cod] # Layer zips .layers # C extensions *.so # Packages *.egg *.egg-info dist build eggs parts bin var sdist develop-eggs .installed.cfg lib lib64 __pycache__ # Installer logs pip-log.txt # Unit test / coverage reports .coverage .tox nosetests.xml #Misc .cache/ .DS_Store .eggs/ .env/ .idea/ datadog-lambda-python-14/CODEOWNERS000066400000000000000000000000251363153557600170460ustar00rootroot00000000000000* @DataDog/serverlessdatadog-lambda-python-14/CONTRIBUTING.md000066400000000000000000000010351363153557600177060ustar00rootroot00000000000000# Contributing We love pull requests. Here's a quick guide. Fork, then clone: ```bash git clone git@github.com:your-username/datadog-lambda-layer-python.git ``` Make your change, update tests and ensure the tests pass (install Docker if you haven't): ```bash ./scripts/run_tests.sh ``` Push to your fork and [submit a pull request][pr]. [pr]: https://github.com/your-username/datadog-lambda-layer-python/compare/DataDog:master...master At this point you're waiting on us. We may suggest some changes or improvements or alternatives. datadog-lambda-python-14/Dockerfile000066400000000000000000000010611363153557600174460ustar00rootroot00000000000000ARG image FROM $image ARG runtime # Create the directory structure required for AWS Lambda Layer RUN mkdir -p /build/python/lib/$runtime/site-packages WORKDIR /build # Install datadog_lambda and dependencies from local COPY . . RUN pip install . -t ./python/lib/$runtime/site-packages # Remove *.pyc files RUN find ./python/lib/$runtime/site-packages -name \*.pyc -delete # Remove botocore (40MB) to reduce package size. aws-xray-sdk # installs it, while it's already provided by the Lambda Runtime. RUN rm -rf ./python/lib/$runtime/site-packages/botocore*datadog-lambda-python-14/LICENSE000066400000000000000000000261631363153557600164730ustar00rootroot00000000000000 Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "{}" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright 2018 Datadog, Inc. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. SPDX-License-Identifier: Apache-2.0datadog-lambda-python-14/LICENSE-3rdparty.csv000066400000000000000000000010411363153557600210170ustar00rootroot00000000000000Component,Origin,License,Copyright aws-xray-sdk-python,github.com/aws/aws-xray-sdk-python,Apache-2.0, flake8,gitlab.com/pycqa/flake8,MIT,"Copyright (C) 2011-2013 Tarek Ziade . Copyright (C) 2012-2016 Ian Cordasco ." nose2,github.com/nose-devs/nose2,BSD-2-Clause,"Copyright (c) 2012, Jason Pellerin. All rights reserved." requests,github.com/kennethreitz/requests,Apache-2.0,"Copyright 2018 Kenneth Reitz" wrapt,github.com/GrahamDumpleton/wrapt,BSD-2-Clause,"Copyright (c) 2013-2019, Graham Dumpleton"datadog-lambda-python-14/NOTICE000066400000000000000000000002231363153557600163570ustar00rootroot00000000000000Datadog datadog-lambda-layer-python Copyright 2019 Datadog, Inc. This product includes software developed at Datadog (https://www.datadoghq.com/).datadog-lambda-python-14/README.md000066400000000000000000000276111363153557600167440ustar00rootroot00000000000000# datadog-lambda-layer-python [![CircleCI](https://img.shields.io/circleci/build/github/DataDog/datadog-lambda-layer-python)](https://circleci.com/gh/DataDog/datadog-lambda-layer-python) [![PyPI](https://img.shields.io/pypi/v/datadog-lambda)](https://pypi.org/project/datadog-lambda/) ![PyPI - Python Version](https://img.shields.io/pypi/pyversions/datadog-lambda) [![Slack](https://img.shields.io/badge/slack-%23serverless-blueviolet?logo=slack)](https://datadoghq.slack.com/channels/serverless/) [![License](https://img.shields.io/badge/license-Apache--2.0-blue)](https://github.com/DataDog/datadog-lambda-layer-python/blob/master/LICENSE) Datadog Lambda Layer for Python (2.7, 3.6, 3.7 and 3.8) enables custom metric submission from AWS Lambda functions, and distributed tracing between serverful and serverless environments. ## IMPORTANT NOTE AWS Lambda is expected to recieve a [breaking change](https://aws.amazon.com/blogs/compute/upcoming-changes-to-the-python-sdk-in-aws-lambda/) on **March 30th, 2020**. If you are using Datadog Python Lambda layer version 7 or below, please upgrade to version 11. ## Installation Datadog Lambda Layer can be added to a Lambda function via AWS Lambda console, [AWS CLI](https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html#configuration-layers-using) or [Serverless Framework](https://serverless.com/framework/docs/providers/aws/guide/layers/#using-your-layers) using the following ARN. ``` arn:aws:lambda::464622532012:layer:Datadog-: ``` Replace `` with the AWS region where your Lambda function is published to. Replace `` with one of the following that matches your Lambda's Python runtime: - `Datadog-Python27` - `Datadog-Python36` - `Datadog-Python37` - `Datadog-Python38` Replace `` with the latest layer version that can be found from [CHANGELOG](CHANGELOG.md). For example, ``` arn:aws:lambda:us-east-1:464622532012:layer:Datadog-Python37:1 ``` ### PyPI When developing your Lambda function locally where AWS Layer doesn't work, the Datadog Lambda layer can be installed from [PyPI](https://pypi.org/project/datadog-lambda/) by `pip install datadog-lambda` or adding `datadog-lambda` to your project's `requirements.txt`. The minor version of the `datadog-lambda` package always match the layer version. E.g., datadog-lambda v0.5.0 matches the content in layer version 5. ### The Serverless Framework [The Datadog Serverless Framework Plugin](https://github.com/DataDog/serverless-plugin-datadog) makes it easy to manage the Datadog instrumentation for all of your Lambda functions in one place. Instead of the plugin, you can also use the sample `serverless.yml` below as a reference for manually including the Lambda Layer, enable AWS X-Ray tracing, and set up environment variables. ```yaml provider: name: aws runtime: python3.7 tracing: lambda: true apiGateway: true functions: hello: handler: handler.hello events: - http: path: hello method: get layers: - arn:aws:lambda::464622532012:layer:Datadog-: environment: DD_FLUSH_TO_LOG: true ``` ## Environment Variables ### DD_FLUSH_TO_LOG Set to `true` (recommended) to send custom metrics asynchronously (with no added latency to your Lambda function executions) through CloudWatch Logs with the help of [Datadog Forwarder](https://github.com/DataDog/datadog-serverless-functions/tree/master/aws/logs_monitoring). If set to `false`, you also need to set `DD_API_KEY` and `DD_SITE`. ### DD_API_KEY If `DD_FLUSH_TO_LOG` is set to false (not recommended), the Datadog API Key must be defined as one of the following environment variables: - DD_API_KEY - the Datadog API Key in plain-text, NOT recommended - DD_KMS_API_KEY - the KMS-encrypted API Key, requires the `kms:Decrypt` permission - DD_API_KEY_SECRET_ARN - the Secret ARN to fetch API Key from the Secrets Manager, requires the `secretsmanager:GetSecretValue` permission (and `kms:Decrypt` if using a customer managed CMK) You can also supply or override the API key at runtime (not recommended): ```python # Override DD API Key after importing datadog_lambda packages from datadog import api api._api_key = "MY_API_KEY" ``` ### DD_SITE If `DD_FLUSH_TO_LOG` is set to false (not recommended), and your data need to be sent to the Datadog EU site, you must set `DD_SITE` to `datadoghq.eu`. ### DD_LOGS_INJECTION Inject Datadog trace id into logs for [correlation](https://docs.datadoghq.com/tracing/connect_logs_and_traces/?tab=python). Defaults to true. ### DD_LOG_LEVEL Set to `debug` enable debug los from the Datadog Lambda Layer. ### DD_ENHANCED_METRICS Generate enhanced Datadog Lambda integration metrics, such as, `aws.lambda.enhanced.invocations` and `aws.lambda.enhanced.errors`. Defaults to true. ## Basic Usage ```python import requests from datadog_lambda.wrapper import datadog_lambda_wrapper from datadog_lambda.metric import lambda_metric @datadog_lambda_wrapper def lambda_handler(event, context): lambda_metric("my_metric", 10, tags=['tag:value']) requests.get("https://www.datadoghq.com") ``` ## Custom Metrics Custom metrics can be submitted using `lambda_metric` and the Lambda handler function needs to be decorated with `@datadog_lambda_wrapper`. The metrics are submitted as [distribution metrics](https://docs.datadoghq.com/graphing/metrics/distributions/). **IMPORTANT NOTE:** If you have already been submitting the same custom metric as non-distribution metric (e.g., gauge, count, or histogram) without using the Datadog Lambda Layer, you MUST pick a new metric name to use for `lambda_metric`. Otherwise that existing metric will be converted to a distribution metric and the historical data prior to the conversion will be no longer queryable. ```python from datadog_lambda.wrapper import datadog_lambda_wrapper from datadog_lambda.metric import lambda_metric @datadog_lambda_wrapper def lambda_handler(event, context): lambda_metric( "coffee_house.order_value", # metric 12.45, # value tags=['product:latte', 'order:online'] # tags ) ``` ### VPC If your Lambda function is associated with a VPC, you need to ensure it has [access to the public internet](https://aws.amazon.com/premiumsupport/knowledge-center/internet-access-lambda-function/). ## Distributed Tracing [Distributed tracing](https://docs.datadoghq.com/tracing/guide/distributed_tracing/?tab=python) allows you to propagate a trace context from a service running on a host to a service running on AWS Lambda, and vice versa, so you can see performance end-to-end. Linking is implemented by injecting Datadog trace context into the HTTP request headers. Distributed tracing headers are language agnostic, e.g., a trace can be propagated between a Java service running on a host to a Lambda function written in Python. Because the trace context is propagated through HTTP request headers, the Lambda function needs to be triggered by AWS API Gateway or AWS Application Load Balancer. To enable this feature, you simple need to decorate your Lambda handler function with `@datadog_lambda_wrapper`. ```python import requests from datadog_lambda.wrapper import datadog_lambda_wrapper @datadog_lambda_wrapper def lambda_handler(event, context): requests.get("https://www.datadoghq.com") ``` Note, the Datadog Lambda Layer is only needed to enable _distributed_ tracing between Lambda and non-Lambda services. For standalone Lambda functions, traces can be found in Datadog APM after configuring [the X-Ray integration](https://docs.datadoghq.com/integrations/amazon_xray/). ### Patching By default, widely used HTTP client libraries, such as `requests`, `urllib2` and `urllib.request` are patched automatically to inject Datadog trace context into outgoing requests. You can also manually retrieve the Datadog trace context (i.e., http headers in a Python dict) and inject it to request headers when needed. ```python import requests from datadog_lambda.wrapper import datadog_lambda_wrapper from datadog_lambda.tracing import get_dd_trace_context @datadog_lambda_wrapper def lambda_handler(event, context): headers = get_dd_trace_context() requests.get("https://www.datadoghq.com", headers=headers) ``` ### Sampling The traces for your Lambda function are converted by Datadog from AWS X-Ray traces. X-Ray needs to sample the traces that the Datadog tracing agent decides to sample, in order to collect as many complete traces as possible. You can create X-Ray sampling rules to ensure requests with header `x-datadog-sampling-priority:1` or `x-datadog-sampling-priority:2` via API Gateway always get sampled by X-Ray. These rules can be created using the following AWS CLI command. ```bash aws xray create-sampling-rule --cli-input-json file://datadog-sampling-priority-1.json aws xray create-sampling-rule --cli-input-json file://datadog-sampling-priority-2.json ``` The file content for `datadog-sampling-priority-1.json`: ```json { "SamplingRule": { "RuleName": "Datadog-Sampling-Priority-1", "ResourceARN": "*", "Priority": 9998, "FixedRate": 1, "ReservoirSize": 100, "ServiceName": "*", "ServiceType": "AWS::APIGateway::Stage", "Host": "*", "HTTPMethod": "*", "URLPath": "*", "Version": 1, "Attributes": { "x-datadog-sampling-priority": "1" } } } ``` The file content for `datadog-sampling-priority-2.json`: ```json { "SamplingRule": { "RuleName": "Datadog-Sampling-Priority-2", "ResourceARN": "*", "Priority": 9999, "FixedRate": 1, "ReservoirSize": 100, "ServiceName": "*", "ServiceType": "AWS::APIGateway::Stage", "Host": "*", "HTTPMethod": "*", "URLPath": "*", "Version": 1, "Attributes": { "x-datadog-sampling-priority": "2" } } } ``` ### Non-proxy integration If your Lambda function is triggered by API Gateway via [the non-proxy integration](https://docs.aws.amazon.com/apigateway/latest/developerguide/getting-started-lambda-non-proxy-integration.html), then you have to [set up a mapping template](https://aws.amazon.com/premiumsupport/knowledge-center/custom-headers-api-gateway-lambda/), which passes the Datadog trace context from the incoming HTTP request headers to the Lambda function via the `event` object. If your Lambda function is deployed by the Serverless Framework, such a mapping template gets created by default. ## Log and Trace Correlations By default, the Datadog trace id gets automatically injected into the logs for correlation, if using the standard python `logging` library. If you use a custom logger handler to log in json, you can inject the ids using the helper function `get_correlation_ids` [manually](https://docs.datadoghq.com/tracing/connect_logs_and_traces/?tab=python#manual-trace-id-injection). Set the environment variable `DD_LOGS_INJECTION` to `false` to disable this feature. ```python from datadog_lambda.wrapper import datadog_lambda_wrapper from ddtrace.helpers import get_correlation_ids @datadog_lambda_wrapper def lambda_handler(event, context): trace_id, span_id = get_correlation_ids() logger.info({ "message": "hello world", "dd": { "trace_id": trace_id, "span_id": span_id } }) ``` ## Opening Issues If you encounter a bug with this package, we want to hear about it. Before opening a new issue, search the existing issues to avoid duplicates. When opening an issue, include the Datadog Lambda Layer version, Python version, and stack trace if available. In addition, include the steps to reproduce when appropriate. You can also open an issue for a feature request. ## Contributing If you find an issue with this package and have a fix, please feel free to open a pull request following the [procedures](CONTRIBUTING.md). ## License Unless explicitly stated otherwise all files in this repository are licensed under the Apache License Version 2.0. This product includes software developed at Datadog (https://www.datadoghq.com/). Copyright 2019 Datadog, Inc. datadog-lambda-python-14/datadog_lambda/000077500000000000000000000000001363153557600203615ustar00rootroot00000000000000datadog-lambda-python-14/datadog_lambda/__init__.py000066400000000000000000000004461363153557600224760ustar00rootroot00000000000000# The minor version corresponds to the Lambda layer version. # E.g.,, version 0.5.0 gets packaged into layer version 5. __version__ = "2.14.0" import os import logging logger = logging.getLogger(__name__) logger.setLevel(logging.getLevelName(os.environ.get("DD_LOG_LEVEL", "INFO").upper())) datadog-lambda-python-14/datadog_lambda/cold_start.py000066400000000000000000000011641363153557600230730ustar00rootroot00000000000000_cold_start = True _lambda_container_initialized = False def set_cold_start(): """Set the value of the cold start global This should be executed once per Lambda execution before the execution """ global _cold_start global _lambda_container_initialized _cold_start = not _lambda_container_initialized _lambda_container_initialized = True def is_cold_start(): """Returns the value of the global cold_start """ return _cold_start def get_cold_start_tag(): """Returns the cold start tag to be used in metrics """ return "cold_start:{}".format(str(is_cold_start()).lower()) datadog-lambda-python-14/datadog_lambda/constants.py000066400000000000000000000013141363153557600227460ustar00rootroot00000000000000# Unless explicitly stated otherwise all files in this repository are licensed # under the Apache License Version 2.0. # This product includes software developed at Datadog (https://www.datadoghq.com/). # Copyright 2019 Datadog, Inc. # Datadog trace sampling priority class SamplingPriority(object): USER_REJECT = -1 AUTO_REJECT = 0 AUTO_KEEP = 1 USER_KEEP = 2 # Datadog trace headers class TraceHeader(object): TRACE_ID = 'x-datadog-trace-id' PARENT_ID = 'x-datadog-parent-id' SAMPLING_PRIORITY = 'x-datadog-sampling-priority' # X-Ray subsegment to save Datadog trace metadata class XraySubsegment(object): NAME = 'datadog-metadata' KEY = 'trace' NAMESPACE = 'datadog' datadog-lambda-python-14/datadog_lambda/metric.py000066400000000000000000000107551363153557600222260ustar00rootroot00000000000000# Unless explicitly stated otherwise all files in this repository are licensed # under the Apache License Version 2.0. # This product includes software developed at Datadog (https://www.datadoghq.com/). # Copyright 2019 Datadog, Inc. import os import json import time import base64 import logging import boto3 from datadog import api from datadog.threadstats import ThreadStats from datadog_lambda.tags import get_enhanced_metrics_tags, tag_dd_lambda_layer ENHANCED_METRICS_NAMESPACE_PREFIX = "aws.lambda.enhanced" logger = logging.getLogger(__name__) lambda_stats = ThreadStats() lambda_stats.start() def lambda_metric(metric_name, value, timestamp=None, tags=None): """ Submit a data point to Datadog distribution metrics. https://docs.datadoghq.com/graphing/metrics/distributions/ When DD_FLUSH_TO_LOG is True, write metric to log, and wait for the Datadog Log Forwarder Lambda function to submit the metrics asynchronously. Otherwise, the metrics will be submitted to the Datadog API periodically and at the end of the function execution in a background thread. """ tags = tag_dd_lambda_layer(tags) if os.environ.get("DD_FLUSH_TO_LOG", "").lower() == "true": write_metric_point_to_stdout(metric_name, value, timestamp, tags) else: logger.debug("Sending metric %s to Datadog via lambda layer", metric_name) lambda_stats.distribution(metric_name, value, timestamp=timestamp, tags=tags) def write_metric_point_to_stdout(metric_name, value, timestamp=None, tags=[]): """Writes the specified metric point to standard output """ logger.debug( "Sending metric %s value %s to Datadog via log forwarder", metric_name, value ) print( json.dumps( { "m": metric_name, "v": value, "e": timestamp or int(time.time()), "t": tags, } ) ) def are_enhanced_metrics_enabled(): """Check env var to find if enhanced metrics should be submitted Returns: boolean for whether enhanced metrics are enabled """ # DD_ENHANCED_METRICS defaults to true return os.environ.get("DD_ENHANCED_METRICS", "true").lower() == "true" def submit_enhanced_metric(metric_name, lambda_context): """Submits the enhanced metric with the given name Args: metric_name (str): metric name w/o enhanced prefix i.e. "invocations" or "errors" lambda_context (dict): Lambda context dict passed to the function by AWS """ if not are_enhanced_metrics_enabled(): logger.debug( "Not submitting enhanced metric %s because enhanced metrics are disabled", metric_name, ) return # Enhanced metrics are always written to logs write_metric_point_to_stdout( "{}.{}".format(ENHANCED_METRICS_NAMESPACE_PREFIX, metric_name), 1, tags=get_enhanced_metrics_tags(lambda_context), ) def submit_invocations_metric(lambda_context): """Increment aws.lambda.enhanced.invocations by 1, applying runtime, layer, and cold_start tags Args: lambda_context (dict): Lambda context dict passed to the function by AWS """ submit_enhanced_metric("invocations", lambda_context) def submit_errors_metric(lambda_context): """Increment aws.lambda.enhanced.errors by 1, applying runtime, layer, and cold_start tags Args: lambda_context (dict): Lambda context dict passed to the function by AWS """ submit_enhanced_metric("errors", lambda_context) # Set API Key and Host in the module, so they only set once per container if not api._api_key: DD_API_KEY_SECRET_ARN = os.environ.get("DD_API_KEY_SECRET_ARN", "") DD_KMS_API_KEY = os.environ.get("DD_KMS_API_KEY", "") DD_API_KEY = os.environ.get("DD_API_KEY", os.environ.get("DATADOG_API_KEY", "")) if DD_API_KEY_SECRET_ARN: api._api_key = boto3.client("secretsmanager").get_secret_value( SecretId=DD_API_KEY_SECRET_ARN )["SecretString"] elif DD_KMS_API_KEY: api._api_key = boto3.client("kms").decrypt( CiphertextBlob=base64.b64decode(DD_KMS_API_KEY) )["Plaintext"] else: api._api_key = DD_API_KEY logger.debug("Setting DATADOG_API_KEY of length %d", len(api._api_key)) # Set DATADOG_HOST, to send data to a non-default Datadog datacenter api._api_host = os.environ.get( "DATADOG_HOST", "https://api." + os.environ.get("DD_SITE", "datadoghq.com") ) logger.debug("Setting DATADOG_HOST to %s", api._api_host) datadog-lambda-python-14/datadog_lambda/patch.py000066400000000000000000000046651363153557600220450ustar00rootroot00000000000000# Unless explicitly stated otherwise all files in this repository are licensed # under the Apache License Version 2.0. # This product includes software developed at Datadog (https://www.datadoghq.com/). # Copyright 2019 Datadog, Inc. import sys import logging from wrapt import wrap_function_wrapper as wrap from datadog_lambda.tracing import get_dd_trace_context logger = logging.getLogger(__name__) if sys.version_info >= (3, 0, 0): httplib_module = 'http.client' else: httplib_module = 'httplib' _httplib_patched = False _requests_patched = False def patch_all(): """ Patch the widely-used HTTP clients to automatically inject Datadog trace context. """ _patch_httplib() _patch_requests() def _patch_httplib(): """ Patch the Python built-in `httplib` (Python 2) or `http.client` (Python 3) module. """ global _httplib_patched if not _httplib_patched: _httplib_patched = True wrap( httplib_module, 'HTTPConnection.request', _wrap_httplib_request ) logger.debug('Patched %s', httplib_module) def _patch_requests(): """ Patch the high-level HTTP client module `requests` if it's installed. """ global _requests_patched if not _requests_patched: _requests_patched = True try: wrap( 'requests', 'Session.request', _wrap_requests_request ) logger.debug('Patched requests') except Exception: logger.debug('Failed to patch requests', exc_info=True) def _wrap_requests_request(func, instance, args, kwargs): """ Wrap `requests.Session.request` to inject the Datadog trace headers into the outgoing requests. """ context = get_dd_trace_context() if 'headers' in kwargs: kwargs['headers'].update(context) elif len(args) >= 5: args[4].update(context) else: kwargs['headers'] = context return func(*args, **kwargs) def _wrap_httplib_request(func, instance, args, kwargs): """ Wrap `httplib` (python2) or `http.client` (python3) to inject the Datadog trace headers into the outgoing requests. """ context = get_dd_trace_context() if 'headers' in kwargs: kwargs['headers'].update(context) elif len(args) >= 4: args[3].update(context) else: kwargs['headers'] = context return func(*args, **kwargs) datadog-lambda-python-14/datadog_lambda/tags.py000066400000000000000000000036661363153557600217040ustar00rootroot00000000000000import sys from platform import python_version_tuple from datadog_lambda import __version__ from datadog_lambda.cold_start import get_cold_start_tag def _format_dd_lambda_layer_tag(): """ Formats the dd_lambda_layer tag, e.g., 'dd_lambda_layer:datadog-python27_1' """ runtime = "python{}{}".format(sys.version_info[0], sys.version_info[1]) return "dd_lambda_layer:datadog-{}_{}".format(runtime, __version__) def tag_dd_lambda_layer(tags): """ Used by lambda_metric to insert the dd_lambda_layer tag """ dd_lambda_layer_tag = _format_dd_lambda_layer_tag() if tags: return tags + [dd_lambda_layer_tag] else: return [dd_lambda_layer_tag] def parse_lambda_tags_from_arn(arn): """Generate the list of lambda tags based on the data in the arn Args: arn (str): Lambda ARN. ex: arn:aws:lambda:us-east-1:123597598159:function:my-lambda[:optional-version] """ # Cap the number of times to split split_arn = arn.split(":") # If ARN includes version / alias at the end, drop it if len(split_arn) > 7: split_arn = split_arn[:7] _, _, _, region, account_id, _, function_name = split_arn return [ "region:{}".format(region), "account_id:{}".format(account_id), "functionname:{}".format(function_name), ] def get_runtime_tag(): """Get the runtime tag from the current Python version """ major_version, minor_version, _ = python_version_tuple() return "runtime:python{major}.{minor}".format( major=major_version, minor=minor_version ) def get_enhanced_metrics_tags(lambda_context): """Get the list of tags to apply to enhanced metrics """ return parse_lambda_tags_from_arn(lambda_context.invoked_function_arn) + [ get_cold_start_tag(), "memorysize:{}".format(lambda_context.memory_limit_in_mb), get_runtime_tag(), _format_dd_lambda_layer_tag(), ] datadog-lambda-python-14/datadog_lambda/tracing.py000066400000000000000000000135671363153557600223760ustar00rootroot00000000000000# Unless explicitly stated otherwise all files in this repository are licensed # under the Apache License Version 2.0. # This product includes software developed at Datadog (https://www.datadoghq.com/). # Copyright 2019 Datadog, Inc. import logging from aws_xray_sdk.core import xray_recorder from aws_xray_sdk.core.lambda_launcher import LambdaContext from ddtrace import patch, tracer from datadog_lambda.constants import ( SamplingPriority, TraceHeader, XraySubsegment, ) logger = logging.getLogger(__name__) dd_trace_context = {} def _convert_xray_trace_id(xray_trace_id): """ Convert X-Ray trace id (hex)'s last 63 bits to a Datadog trace id (int). """ return str(0x7FFFFFFFFFFFFFFF & int(xray_trace_id[-16:], 16)) def _convert_xray_entity_id(xray_entity_id): """ Convert X-Ray (sub)segement id (hex) to a Datadog span id (int). """ return str(int(xray_entity_id, 16)) def _convert_xray_sampling(xray_sampled): """ Convert X-Ray sampled (True/False) to its Datadog counterpart. """ return str(SamplingPriority.USER_KEEP) if xray_sampled \ else str(SamplingPriority.USER_REJECT) def extract_dd_trace_context(event): """ Extract Datadog trace context from the Lambda `event` object. Write the context to a global `dd_trace_context`, so the trace can be continued on the outgoing requests with the context injected. Save the context to an X-Ray subsegment's metadata field, so the X-Ray trace can be converted to a Datadog trace in the Datadog backend with the correct context. """ global dd_trace_context headers = event.get('headers', {}) lowercase_headers = {k.lower(): v for k, v in headers.items()} trace_id = lowercase_headers.get(TraceHeader.TRACE_ID) parent_id = lowercase_headers.get(TraceHeader.PARENT_ID) sampling_priority = lowercase_headers.get(TraceHeader.SAMPLING_PRIORITY) if trace_id and parent_id and sampling_priority: logger.debug('Extracted Datadog trace context from headers') dd_trace_context = { 'trace-id': trace_id, 'parent-id': parent_id, 'sampling-priority': sampling_priority, } xray_recorder.begin_subsegment(XraySubsegment.NAME) subsegment = xray_recorder.current_subsegment() subsegment.put_metadata( XraySubsegment.KEY, dd_trace_context, XraySubsegment.NAMESPACE ) xray_recorder.end_subsegment() else: # AWS Lambda runtime caches global variables between invocations, # reset to avoid using the context from the last invocation. dd_trace_context = {} logger.debug('extracted dd trace context %s', dd_trace_context) def get_dd_trace_context(): """ Return the Datadog trace context to be propogated on the outgoing requests. If the Lambda function is invoked by a Datadog-traced service, a Datadog trace context may already exist, and it should be used. Otherwise, use the current X-Ray trace entity. Most of widely-used HTTP clients are patched to inject the context automatically, but this function can be used to manually inject the trace context to an outgoing request. """ if not is_lambda_context(): logger.debug('get_dd_trace_context is only supported in LambdaContext') return {} global dd_trace_context xray_trace_entity = xray_recorder.get_trace_entity() # xray (sub)segment if dd_trace_context: return { TraceHeader.TRACE_ID: dd_trace_context['trace-id'], TraceHeader.PARENT_ID: _convert_xray_entity_id( xray_trace_entity.id), TraceHeader.SAMPLING_PRIORITY: dd_trace_context['sampling-priority'], } else: return { TraceHeader.TRACE_ID: _convert_xray_trace_id( xray_trace_entity.trace_id), TraceHeader.PARENT_ID: _convert_xray_entity_id( xray_trace_entity.id), TraceHeader.SAMPLING_PRIORITY: _convert_xray_sampling( xray_trace_entity.sampled), } def set_correlation_ids(): """ Create a dummy span, and overrides its trace_id and span_id, to make ddtrace.helpers.get_correlation_ids() return the correct ids for both auto and manual log correlations. TODO: Remove me when Datadog tracer is natively supported in Lambda. """ if not is_lambda_context(): logger.debug('set_correlation_ids is only supported in LambdaContext') return context = get_dd_trace_context() span = tracer.trace('dummy.span') span.trace_id = context[TraceHeader.TRACE_ID] span.span_id = context[TraceHeader.PARENT_ID] logger.debug('correlation ids set') def inject_correlation_ids(): """ Override the formatter of LambdaLoggerHandler to inject datadog trace and span id for log correlation. For manual injections to custom log handlers, use `ddtrace.helpers.get_correlation_ids` to retrieve correlation ids (trace_id, span_id). """ # Override the log format of the AWS provided LambdaLoggerHandler root_logger = logging.getLogger() for handler in root_logger.handlers: if handler.__class__.__name__ == 'LambdaLoggerHandler': handler.setFormatter(logging.Formatter( '[%(levelname)s]\t%(asctime)s.%(msecs)dZ\t%(aws_request_id)s\t' '[dd.trace_id=%(dd.trace_id)s dd.span_id=%(dd.span_id)s]\t%(message)s\n', '%Y-%m-%dT%H:%M:%S' )) # Patch `logging.Logger.makeRecord` to actually inject correlation ids patch(logging=True) logger.debug('logs injection configured') def is_lambda_context(): """ Return True if the X-Ray context is `LambdaContext`, rather than the regular `Context` (e.g., when testing lambda functions locally). """ return type(xray_recorder.context) == LambdaContext datadog-lambda-python-14/datadog_lambda/wrapper.py000066400000000000000000000064751363153557600224270ustar00rootroot00000000000000# Unless explicitly stated otherwise all files in this repository are licensed # under the Apache License Version 2.0. # This product includes software developed at Datadog (https://www.datadoghq.com/). # Copyright 2019 Datadog, Inc. import os import logging import traceback from datadog_lambda.cold_start import set_cold_start from datadog_lambda.metric import ( lambda_stats, submit_invocations_metric, submit_errors_metric, ) from datadog_lambda.patch import patch_all from datadog_lambda.tracing import ( extract_dd_trace_context, set_correlation_ids, inject_correlation_ids, ) logger = logging.getLogger(__name__) """ Usage: import requests from datadog_lambda.wrapper import datadog_lambda_wrapper from datadog_lambda.metric import lambda_metric @datadog_lambda_wrapper def my_lambda_handle(event, context): lambda_metric("my_metric", 10) requests.get("https://www.datadoghq.com") """ class _NoopDecorator(object): def __init__(self, func): self.func = func def __call__(self, *args, **kwargs): return self.func(*args, **kwargs) class _LambdaDecorator(object): """ Decorator to automatically initialize Datadog API client, flush metrics, and extracts/injects trace context. """ _force_new = False def __new__(cls, func): """ If the decorator is accidentally applied to the same function multiple times, only the first one takes effect. If _force_new, always return a real decorator, useful for unit tests. """ if cls._force_new or not getattr(func, "_dd_wrapped", False): wrapped = super(_LambdaDecorator, cls).__new__(cls) wrapped._dd_wrapped = True return wrapped else: return _NoopDecorator(func) def __init__(self, func): """Executes when the wrapped function gets wrapped""" self.func = func self.flush_to_log = os.environ.get("DD_FLUSH_TO_LOG", "").lower() == "true" self.logs_injection = ( os.environ.get("DD_LOGS_INJECTION", "true").lower() == "true" ) # Inject trace correlation ids to logs if self.logs_injection: inject_correlation_ids() # Patch HTTP clients to propagate Datadog trace context patch_all() logger.debug("datadog_lambda_wrapper initialized") def __call__(self, event, context, **kwargs): """Executes when the wrapped function gets called""" self._before(event, context) try: return self.func(event, context, **kwargs) except Exception: submit_errors_metric(context) raise finally: self._after(event, context) def _before(self, event, context): set_cold_start() try: submit_invocations_metric(context) # Extract Datadog trace context from incoming requests extract_dd_trace_context(event) # Set log correlation ids using extracted trace context set_correlation_ids() except Exception: traceback.print_exc() def _after(self, event, context): try: if not self.flush_to_log: lambda_stats.flush(float("inf")) except Exception: traceback.print_exc() datadog_lambda_wrapper = _LambdaDecorator datadog-lambda-python-14/scripts/000077500000000000000000000000001363153557600171455ustar00rootroot00000000000000datadog-lambda-python-14/scripts/build_layers.sh000077500000000000000000000031451363153557600221650ustar00rootroot00000000000000#!/bin/sh # Unless explicitly stated otherwise all files in this repository are licensed # under the Apache License Version 2.0. # This product includes software developed at Datadog (https://www.datadoghq.com/). # Copyright 2019 Datadog, Inc. # Builds Datadogpy layers for lambda functions, using Docker set -e LAYER_DIR=".layers" LAYER_FILES_PREFIX="datadog_lambda_py" PYTHON_VERSIONS=("2.7" "3.6" "3.7" "3.8") function make_path_absolute { echo "$(cd "$(dirname "$1")"; pwd)/$(basename "$1")" } function docker_build_zip { # Args: [python version] [zip destination] destination=$(make_path_absolute $2) # Install datadogpy in a docker container to avoid the mess from switching # between different python runtimes. temp_dir=$(mktemp -d) docker build -t datadog-lambda-layer-python:$1 . --no-cache \ --build-arg image=python:$1 \ --build-arg runtime=python$1 # Run the image by runtime tag, tar its generatd `python` directory to sdout, # then extract it to a temp directory. docker run datadog-lambda-layer-python:$1 tar cf - python | tar -xf - -C $temp_dir # Zip to destination, and keep directory structure as based in $temp_dir (cd $temp_dir && zip -q -r $destination ./) rm -rf $temp_dir echo "Done creating archive $destination" } rm -rf $LAYER_DIR mkdir $LAYER_DIR for python_version in "${PYTHON_VERSIONS[@]}" do echo "Building layer for python${python_version}" docker_build_zip ${python_version} $LAYER_DIR/${LAYER_FILES_PREFIX}${python_version}.zip done echo "Done creating layers:" ls $LAYER_DIR | xargs -I _ echo "$LAYER_DIR/_" datadog-lambda-python-14/scripts/list_layers.sh000077500000000000000000000035761363153557600220510ustar00rootroot00000000000000#!/bin/bash # Unless explicitly stated otherwise all files in this repository are licensed # under the Apache License Version 2.0. # This product includes software developed at Datadog (https://www.datadoghq.com/). # Copyright 2019 Datadog, Inc. # Lists most recent layers ARNs across regions to STDOUT # Optionals args: [layer-name] [region] LAYER_NAMES=("Datadog-Python27" "Datadog-Python36" "Datadog-Python37" "Datadog-Python38") AVAILABLE_REGIONS=(us-east-2 us-east-1 us-west-1 us-west-2 ap-south-1 ap-northeast-2 ap-southeast-1 ap-southeast-2 ap-northeast-1 ca-central-1 eu-north-1 eu-central-1 eu-west-1 eu-west-2 eu-west-3 sa-east-1) # Check region arg if [ -z "$2" ]; then >&2 echo "Region parameter not specified, running for all available regions." REGIONS=("${AVAILABLE_REGIONS[@]}") else >&2 echo "Region parameter specified: $2" if [[ ! " ${AVAILABLE_REGIONS[@]} " =~ " ${2} " ]]; then >&2 echo "Could not find $2 in available regions: ${AVAILABLE_REGIONS[@]}" >&2 echo "" >&2 echo "EXITING SCRIPT." return 1 fi REGIONS=($2) fi # Check region arg if [ -z "$1" ]; then >&2 echo "Layer parameter not specified, running for all layers " LAYERS=("${LAYER_NAMES[@]}") else >&2 echo "Layer parameter specified: $1" if [[ ! " ${LAYER_NAMES[@]} " =~ " ${1} " ]]; then >&2 echo "Could not find $1 in layers: ${LAYER_NAMES[@]}" >&2 echo "" >&2 echo "EXITING SCRIPT." return 1 fi LAYERS=($1) fi for region in "${REGIONS[@]}" do for layer_name in "${LAYERS[@]}" do last_layer_arn=$(aws lambda list-layer-versions --layer-name $layer_name --region $region | jq -r ".LayerVersions | .[0] | .LayerVersionArn") if [ -z $last_layer_arn ]; then >&2 echo "No layer found for $region, $layer_name" else echo $last_layer_arn fi done done datadog-lambda-python-14/scripts/publish_layers.sh000077500000000000000000000072361363153557600225410ustar00rootroot00000000000000#!/bin/bash # Unless explicitly stated otherwise all files in this repository are licensed # under the Apache License Version 2.0. # This product includes software developed at Datadog (https://www.datadoghq.com/). # Copyright 2019 Datadog, Inc. # Publish the datadog python lambda layer across regions, using the AWS CLI # Usage: publish_layer.sh [region] [layer] # Specifying the region and layer arg will publish the specified layer to the specified region set -e # Makes sure any subprocesses will be terminated with this process trap "pkill -P $$; exit 1;" INT PYTHON_VERSIONS_FOR_AWS_CLI=("python2.7" "python3.6" "python3.7" "python3.8") LAYER_PATHS=(".layers/datadog_lambda_py2.7.zip" ".layers/datadog_lambda_py3.6.zip" ".layers/datadog_lambda_py3.7.zip" ".layers/datadog_lambda_py3.8.zip") AVAILABLE_LAYER_NAMES=("Datadog-Python27" "Datadog-Python36" "Datadog-Python37" "Datadog-Python38") AVAILABLE_REGIONS=(us-east-2 us-east-1 us-west-1 us-west-2 ap-south-1 ap-northeast-2 ap-southeast-1 ap-southeast-2 ap-northeast-1 ca-central-1 eu-north-1 eu-central-1 eu-west-1 eu-west-2 eu-west-3 sa-east-1) # Check that the layer files exist for layer_file in "${LAYER_PATHS[@]}" do if [ ! -f $layer_file ]; then echo "Could not find $layer_file." exit 1 fi done # Check region arg if [ -z "$1" ]; then echo "Region parameter not specified, running for all available regions." REGIONS=("${AVAILABLE_REGIONS[@]}") else echo "Region parameter specified: $1" if [[ ! " ${AVAILABLE_REGIONS[@]} " =~ " ${1} " ]]; then echo "Could not find $1 in available regions: ${AVAILABLE_REGIONS[@]}" echo "" echo "EXITING SCRIPT." exit 1 fi REGIONS=($1) fi echo "Publishing layers for regions: ${REGIONS[*]}" # Check layer_name arg if [ -z "$2" ]; then echo "Layer name parameter not specified, running for all layer names." LAYER_NAMES=("${AVAILABLE_LAYER_NAMES[@]}") else echo "Layer name parameter specified: $2" if [[ ! " ${AVAILABLE_LAYER_NAMES[@]} " =~ " ${2} " ]]; then echo "Could not find $2 in available layer names: ${AVAILABLE_LAYER_NAMES[@]}" echo "" echo "EXITING SCRIPT." exit 1 fi LAYER_NAMES=($2) fi echo "Publishing layers: ${LAYER_NAMES[*]}" publish_layer() { region=$1 layer_name=$2 aws_version_key=$3 layer_path=$4 version_nbr=$(aws lambda publish-layer-version --layer-name $layer_name \ --description "Datadog Lambda Layer for Python" \ --zip-file "fileb://$layer_path" \ --region $region \ --compatible-runtimes $aws_version_key \ | jq -r '.Version') aws lambda add-layer-version-permission --layer-name $layer_name \ --version-number $version_nbr \ --statement-id "release-$version_nbr" \ --action lambda:GetLayerVersion --principal "*" \ --region $region echo "Published layer for region $region, python version $aws_version_key, layer_name $layer_name, layer_version $version_nbr" } BATCH_SIZE=60 PIDS=() wait_for_processes() { for pid in "${PIDS[@]}"; do wait $pid done PIDS=() } for region in "${REGIONS[@]}" do echo "Starting publishing layer for region $region..." # Publish the layers for each version of python i=0 for layer_name in "${LAYER_NAMES[@]}"; do aws_version_key="${PYTHON_VERSIONS_FOR_AWS_CLI[$i]}" layer_path="${LAYER_PATHS[$i]}" publish_layer $region $layer_name $aws_version_key $layer_path & PIDS+=($!) if [ ${#PIDS[@]} -eq $BATCH_SIZE ]; then wait_for_processes fi i=$(expr $i + 1) done done wait_for_processes echo "Done !" datadog-lambda-python-14/scripts/pypi.sh000077500000000000000000000011721363153557600204660ustar00rootroot00000000000000#!/bin/sh # Unless explicitly stated otherwise all files in this repository are licensed # under the Apache License Version 2.0. # This product includes software developed at Datadog (https://www.datadoghq.com/). # Copyright 2019 Datadog, Inc. # Builds the lambda layer and upload to Pypi set -e # Clear previously built distributions if [ -d "dist" ]; then echo "Removing folder 'dist' to clear previously built distributions" rm -rf dist; fi # Install build tools pip install --upgrade setuptools wheel twine # Build distributions python setup.py sdist bdist_wheel # Upload distributions python -m twine upload dist/* datadog-lambda-python-14/scripts/run_tests.sh000077500000000000000000000016471363153557600215420ustar00rootroot00000000000000#!/bin/bash # Unless explicitly stated otherwise all files in this repository are licensed # under the Apache License Version 2.0. # This product includes software developed at Datadog (https://www.datadoghq.com/). # Copyright 2019 Datadog, Inc. # Run unit tests in Docker set -e PYTHON_VERSIONS=("2.7" "3.6" "3.7" "3.8") for python_version in "${PYTHON_VERSIONS[@]}" do echo "Running tests against python${python_version}" docker build -t datadog-lambda-layer-python-test:$python_version \ -f tests/Dockerfile . \ --build-arg python_version=$python_version docker run -v `pwd`:/datadog-lambda-layer-python \ -w /datadog-lambda-layer-python \ datadog-lambda-layer-python-test:$python_version \ nose2 -v docker run -v `pwd`:/datadog-lambda-layer-python \ -w /datadog-lambda-layer-python \ datadog-lambda-layer-python-test:$python_version \ flake8 done datadog-lambda-python-14/setup.py000066400000000000000000000024341363153557600171730ustar00rootroot00000000000000from setuptools import setup from os import path from io import open from datadog_lambda import __version__ here = path.abspath(path.dirname(__file__)) with open(path.join(here, 'README.md'), encoding='utf-8') as f: long_description = f.read() setup( name='datadog_lambda', version=__version__, description='The Datadog AWS Lambda Layer', long_description=long_description, long_description_content_type='text/markdown', url='https://github.com/DataDog/datadog-lambda-layer-python', author='Datadog, Inc.', author_email='dev@datadoghq.com', classifiers=[ 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3.6', 'Programming Language :: Python :: 3.7', 'Programming Language :: Python :: 3.8', ], keywords='datadog aws lambda layer', packages=['datadog_lambda'], python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*, <4', install_requires=[ 'aws-xray-sdk==2.4.3', 'datadog==0.32.0', 'ddtrace==0.31.0', 'wrapt==1.11.2', 'setuptools==42.0.2', ], extras_require={ 'dev': [ 'nose2==0.9.1', 'flake8==3.7.9', 'requests==2.22.0', 'boto3==1.10.33', ] } ) datadog-lambda-python-14/tests/000077500000000000000000000000001363153557600166205ustar00rootroot00000000000000datadog-lambda-python-14/tests/Dockerfile000066400000000000000000000003151363153557600206110ustar00rootroot00000000000000ARG python_version FROM python:$python_version ENV PYTHONDONTWRITEBYTECODE True RUN mkdir -p /test WORKDIR /test # Install datadog-lambda with dev dependencies from local COPY . . RUN pip install .[dev]datadog-lambda-python-14/tests/__init__.py000066400000000000000000000000001363153557600207170ustar00rootroot00000000000000datadog-lambda-python-14/tests/test_metric.py000066400000000000000000000023721363153557600215200ustar00rootroot00000000000000import os import unittest try: from unittest.mock import patch, call except ImportError: from mock import patch, call from datadog_lambda.metric import lambda_metric from datadog_lambda.tags import _format_dd_lambda_layer_tag class TestLambdaMetric(unittest.TestCase): def setUp(self): patcher = patch("datadog_lambda.metric.lambda_stats") self.mock_metric_lambda_stats = patcher.start() self.addCleanup(patcher.stop) def test_lambda_metric_tagged_with_dd_lambda_layer(self): lambda_metric("test", 1) lambda_metric("test", 1, 123, []) lambda_metric("test", 1, tags=["tag1:test"]) expected_tag = _format_dd_lambda_layer_tag() self.mock_metric_lambda_stats.distribution.assert_has_calls( [ call("test", 1, timestamp=None, tags=[expected_tag]), call("test", 1, timestamp=123, tags=[expected_tag]), call("test", 1, timestamp=None, tags=["tag1:test", expected_tag]), ] ) def test_lambda_metric_flush_to_log(self): os.environ["DD_FLUSH_TO_LOG"] = "True" lambda_metric("test", 1) self.mock_metric_lambda_stats.distribution.assert_not_called() del os.environ["DD_FLUSH_TO_LOG"] datadog-lambda-python-14/tests/test_patch.py000066400000000000000000000025731363153557600213370ustar00rootroot00000000000000import sys import unittest try: from unittest.mock import patch except ImportError: from mock import patch from datadog_lambda.patch import ( _patch_httplib, _patch_requests, ) from datadog_lambda.constants import TraceHeader class TestPatchHTTPClients(unittest.TestCase): def setUp(self): patcher = patch('datadog_lambda.patch.get_dd_trace_context') self.mock_get_dd_trace_context = patcher.start() self.mock_get_dd_trace_context.return_value = { TraceHeader.TRACE_ID: '123', TraceHeader.PARENT_ID: '321', TraceHeader.SAMPLING_PRIORITY: '2', } self.addCleanup(patcher.stop) def test_patch_httplib(self): _patch_httplib() if sys.version_info >= (3, 0, 0): import urllib.request as urllib else: import urllib2 as urllib urllib.urlopen("https://www.datadoghq.com/") self.mock_get_dd_trace_context.assert_called() def test_patch_requests(self): _patch_requests() import requests r = requests.get("https://www.datadoghq.com/") self.mock_get_dd_trace_context.assert_called() self.assertEqual(r.request.headers[TraceHeader.TRACE_ID], '123') self.assertEqual(r.request.headers[TraceHeader.PARENT_ID], '321') self.assertEqual(r.request.headers[TraceHeader.SAMPLING_PRIORITY], '2') datadog-lambda-python-14/tests/test_tags.py000066400000000000000000000026461363153557600211770ustar00rootroot00000000000000import unittest try: from unittest.mock import patch except ImportError: from mock import patch from datadog_lambda.tags import parse_lambda_tags_from_arn, get_runtime_tag class TestMetricTags(unittest.TestCase): def setUp(self): patcher = patch("datadog_lambda.tags.python_version_tuple") self.mock_python_version_tuple = patcher.start() self.addCleanup(patcher.stop) def test_parse_lambda_tags_from_arn(self): self.assertListEqual( parse_lambda_tags_from_arn( "arn:aws:lambda:us-east-1:1234597598159:function:swf-hello-test" ), [ "region:us-east-1", "account_id:1234597598159", "functionname:swf-hello-test", ], ) self.assertListEqual( parse_lambda_tags_from_arn( "arn:aws:lambda:us-west-1:1234597598159:function:other-function:function-alias" ), [ "region:us-west-1", "account_id:1234597598159", "functionname:other-function", ], ) def test_get_runtime_tag(self): self.mock_python_version_tuple.return_value = ("2", "7", "10") self.assertEqual(get_runtime_tag(), "runtime:python2.7") self.mock_python_version_tuple.return_value = ("3", "7", "2") self.assertEqual(get_runtime_tag(), "runtime:python3.7") datadog-lambda-python-14/tests/test_tracing.py000066400000000000000000000127741363153557600216730ustar00rootroot00000000000000import unittest try: from unittest.mock import MagicMock, patch except ImportError: from mock import MagicMock, patch from ddtrace.helpers import get_correlation_ids from datadog_lambda.constants import ( SamplingPriority, TraceHeader, XraySubsegment, ) from datadog_lambda.tracing import ( extract_dd_trace_context, get_dd_trace_context, set_correlation_ids, _convert_xray_trace_id, _convert_xray_entity_id, _convert_xray_sampling, ) class TestExtractAndGetDDTraceContext(unittest.TestCase): def setUp(self): patcher = patch('datadog_lambda.tracing.xray_recorder') self.mock_xray_recorder = patcher.start() self.mock_xray_recorder.get_trace_entity.return_value = MagicMock( id='ffff', trace_id='1111', sampled=True, ) self.mock_current_subsegment = MagicMock() self.mock_xray_recorder.current_subsegment.return_value = \ self.mock_current_subsegment self.addCleanup(patcher.stop) patcher = patch('datadog_lambda.tracing.is_lambda_context') self.mock_is_lambda_context = patcher.start() self.mock_is_lambda_context.return_value = True self.addCleanup(patcher.stop) def test_without_datadog_trace_headers(self): extract_dd_trace_context({}) self.assertDictEqual( get_dd_trace_context(), { TraceHeader.TRACE_ID: '4369', TraceHeader.PARENT_ID: '65535', TraceHeader.SAMPLING_PRIORITY: '2', } ) def test_with_incomplete_datadog_trace_headers(self): extract_dd_trace_context({ 'headers': { TraceHeader.TRACE_ID: '123', TraceHeader.PARENT_ID: '321', } }) self.assertDictEqual( get_dd_trace_context(), { TraceHeader.TRACE_ID: '4369', TraceHeader.PARENT_ID: '65535', TraceHeader.SAMPLING_PRIORITY: '2', } ) def test_with_complete_datadog_trace_headers(self): extract_dd_trace_context({ 'headers': { TraceHeader.TRACE_ID: '123', TraceHeader.PARENT_ID: '321', TraceHeader.SAMPLING_PRIORITY: '1', } }) self.assertDictEqual( get_dd_trace_context(), { TraceHeader.TRACE_ID: '123', TraceHeader.PARENT_ID: '65535', TraceHeader.SAMPLING_PRIORITY: '1', } ) self.mock_xray_recorder.begin_subsegment.assert_called() self.mock_xray_recorder.end_subsegment.assert_called() self.mock_current_subsegment.put_metadata.assert_called_with( XraySubsegment.KEY, { 'trace-id': '123', 'parent-id': '321', 'sampling-priority': '1', }, XraySubsegment.NAMESPACE ) def test_with_complete_datadog_trace_headers_with_mixed_casing(self): extract_dd_trace_context({ 'headers': { 'X-Datadog-Trace-Id': '123', 'X-Datadog-Parent-Id': '321', 'X-Datadog-Sampling-Priority': '1', } }) self.assertDictEqual( get_dd_trace_context(), { TraceHeader.TRACE_ID: '123', TraceHeader.PARENT_ID: '65535', TraceHeader.SAMPLING_PRIORITY: '1', } ) class TestXRayContextConversion(unittest.TestCase): def test_convert_xray_trace_id(self): self.assertEqual( _convert_xray_trace_id('00000000e1be46a994272793'), '7043144561403045779' ) self.assertEqual( _convert_xray_trace_id('bd862e3fe1be46a994272793'), '7043144561403045779' ) self.assertEqual( _convert_xray_trace_id('ffffffffffffffffffffffff'), '9223372036854775807' # 0x7FFFFFFFFFFFFFFF ) def test_convert_xray_entity_id(self): self.assertEqual( _convert_xray_entity_id('53995c3f42cd8ad8'), '6023947403358210776' ) self.assertEqual( _convert_xray_entity_id('1000000000000000'), '1152921504606846976' ) self.assertEqual( _convert_xray_entity_id('ffffffffffffffff'), '18446744073709551615' ) def test_convert_xray_sampling(self): self.assertEqual( _convert_xray_sampling(True), str(SamplingPriority.USER_KEEP) ) self.assertEqual( _convert_xray_sampling(False), str(SamplingPriority.USER_REJECT) ) class TestLogsInjection(unittest.TestCase): def setUp(self): patcher = patch('datadog_lambda.tracing.get_dd_trace_context') self.mock_get_dd_trace_context = patcher.start() self.mock_get_dd_trace_context.return_value = { TraceHeader.TRACE_ID: '123', TraceHeader.PARENT_ID: '456', } self.addCleanup(patcher.stop) patcher = patch('datadog_lambda.tracing.is_lambda_context') self.mock_is_lambda_context = patcher.start() self.mock_is_lambda_context.return_value = True self.addCleanup(patcher.stop) def test_set_correlation_ids(self): set_correlation_ids() trace_id, span_id = get_correlation_ids() self.assertEqual(trace_id, '123') self.assertEqual(span_id, '456') datadog-lambda-python-14/tests/test_wrapper.py000066400000000000000000000232741363153557600217210ustar00rootroot00000000000000import os import unittest try: from unittest.mock import patch, call, ANY, MagicMock except ImportError: from mock import patch, call, ANY, MagicMock from datadog_lambda.wrapper import datadog_lambda_wrapper from datadog_lambda.metric import lambda_metric def get_mock_context( aws_request_id="request-id-1", memory_limit_in_mb="256", invoked_function_arn="arn:aws:lambda:us-west-1:123457598159:function:python-layer-test", ): lambda_context = MagicMock() lambda_context.aws_request_id = aws_request_id lambda_context.memory_limit_in_mb = memory_limit_in_mb lambda_context.invoked_function_arn = invoked_function_arn return lambda_context class TestDatadogLambdaWrapper(unittest.TestCase): def setUp(self): # Force @datadog_lambda_wrapper to always create a real # (not no-op) wrapper. datadog_lambda_wrapper._force_new = True patcher = patch("datadog_lambda.metric.lambda_stats") self.mock_metric_lambda_stats = patcher.start() self.addCleanup(patcher.stop) patcher = patch("datadog_lambda.wrapper.lambda_stats") self.mock_wrapper_lambda_stats = patcher.start() self.addCleanup(patcher.stop) patcher = patch("datadog_lambda.metric.lambda_metric") self.mock_lambda_metric = patcher.start() self.addCleanup(patcher.stop) patcher = patch("datadog_lambda.wrapper.extract_dd_trace_context") self.mock_extract_dd_trace_context = patcher.start() self.addCleanup(patcher.stop) patcher = patch("datadog_lambda.wrapper.set_correlation_ids") self.mock_set_correlation_ids = patcher.start() self.addCleanup(patcher.stop) patcher = patch("datadog_lambda.wrapper.inject_correlation_ids") self.mock_inject_correlation_ids = patcher.start() self.addCleanup(patcher.stop) patcher = patch("datadog_lambda.wrapper.patch_all") self.mock_patch_all = patcher.start() self.addCleanup(patcher.stop) patcher = patch("datadog_lambda.cold_start.is_cold_start") self.mock_is_cold_start = patcher.start() self.mock_is_cold_start.return_value = True self.addCleanup(patcher.stop) patcher = patch("datadog_lambda.tags.python_version_tuple") self.mock_python_version_tuple = patcher.start() self.mock_python_version_tuple.return_value = ("2", "7", "10") self.addCleanup(patcher.stop) patcher = patch("datadog_lambda.metric.write_metric_point_to_stdout") self.mock_write_metric_point_to_stdout = patcher.start() self.addCleanup(patcher.stop) patcher = patch("datadog_lambda.tags._format_dd_lambda_layer_tag") self.mock_format_dd_lambda_layer_tag = patcher.start() # Mock the layer version so we don't have to update tests on every version bump self.mock_format_dd_lambda_layer_tag.return_value = ( "dd_lambda_layer:datadog-python27_0.1.0" ) self.addCleanup(patcher.stop) def test_datadog_lambda_wrapper(self): @datadog_lambda_wrapper def lambda_handler(event, context): lambda_metric("test.metric", 100) lambda_event = {} lambda_handler(lambda_event, get_mock_context()) self.mock_metric_lambda_stats.distribution.assert_has_calls( [call("test.metric", 100, timestamp=None, tags=ANY)] ) self.mock_wrapper_lambda_stats.flush.assert_called() self.mock_extract_dd_trace_context.assert_called_with(lambda_event) self.mock_set_correlation_ids.assert_called() self.mock_inject_correlation_ids.assert_called() self.mock_patch_all.assert_called() def test_datadog_lambda_wrapper_flush_to_log(self): os.environ["DD_FLUSH_TO_LOG"] = "True" @datadog_lambda_wrapper def lambda_handler(event, context): lambda_metric("test.metric", 100) lambda_event = {} lambda_handler(lambda_event, get_mock_context()) self.mock_metric_lambda_stats.distribution.assert_not_called() self.mock_wrapper_lambda_stats.flush.assert_not_called() del os.environ["DD_FLUSH_TO_LOG"] def test_datadog_lambda_wrapper_inject_correlation_ids(self): os.environ["DD_LOGS_INJECTION"] = "True" @datadog_lambda_wrapper def lambda_handler(event, context): lambda_metric("test.metric", 100) lambda_event = {} lambda_handler(lambda_event, get_mock_context()) self.mock_set_correlation_ids.assert_called() self.mock_inject_correlation_ids.assert_called() del os.environ["DD_LOGS_INJECTION"] def test_invocations_metric(self): @datadog_lambda_wrapper def lambda_handler(event, context): lambda_metric("test.metric", 100) lambda_event = {} lambda_handler(lambda_event, get_mock_context()) self.mock_write_metric_point_to_stdout.assert_has_calls( [ call( "aws.lambda.enhanced.invocations", 1, tags=[ "region:us-west-1", "account_id:123457598159", "functionname:python-layer-test", "cold_start:true", "memorysize:256", "runtime:python2.7", "dd_lambda_layer:datadog-python27_0.1.0", ], ) ] ) def test_errors_metric(self): @datadog_lambda_wrapper def lambda_handler(event, context): raise RuntimeError() lambda_event = {} with self.assertRaises(RuntimeError): lambda_handler(lambda_event, get_mock_context()) self.mock_write_metric_point_to_stdout.assert_has_calls( [ call( "aws.lambda.enhanced.invocations", 1, tags=[ "region:us-west-1", "account_id:123457598159", "functionname:python-layer-test", "cold_start:true", "memorysize:256", "runtime:python2.7", "dd_lambda_layer:datadog-python27_0.1.0", ], ), call( "aws.lambda.enhanced.errors", 1, tags=[ "region:us-west-1", "account_id:123457598159", "functionname:python-layer-test", "cold_start:true", "memorysize:256", "runtime:python2.7", "dd_lambda_layer:datadog-python27_0.1.0", ], ), ] ) def test_enhanced_metrics_cold_start_tag(self): @datadog_lambda_wrapper def lambda_handler(event, context): lambda_metric("test.metric", 100) lambda_event = {} lambda_handler(lambda_event, get_mock_context()) self.mock_is_cold_start.return_value = False lambda_handler( lambda_event, get_mock_context(aws_request_id="second-request-id") ) self.mock_write_metric_point_to_stdout.assert_has_calls( [ call( "aws.lambda.enhanced.invocations", 1, tags=[ "region:us-west-1", "account_id:123457598159", "functionname:python-layer-test", "cold_start:true", "memorysize:256", "runtime:python2.7", "dd_lambda_layer:datadog-python27_0.1.0", ], ), call( "aws.lambda.enhanced.invocations", 1, tags=[ "region:us-west-1", "account_id:123457598159", "functionname:python-layer-test", "cold_start:false", "memorysize:256", "runtime:python2.7", "dd_lambda_layer:datadog-python27_0.1.0", ], ), ] ) def test_no_enhanced_metrics_without_env_var(self): os.environ["DD_ENHANCED_METRICS"] = "false" @datadog_lambda_wrapper def lambda_handler(event, context): raise RuntimeError() lambda_event = {} with self.assertRaises(RuntimeError): lambda_handler(lambda_event, get_mock_context()) self.mock_write_metric_point_to_stdout.assert_not_called() del os.environ["DD_ENHANCED_METRICS"] def test_only_one_wrapper_in_use(self): patcher = patch("datadog_lambda.wrapper.submit_invocations_metric") self.mock_submit_invocations_metric = patcher.start() self.addCleanup(patcher.stop) @datadog_lambda_wrapper def lambda_handler(event, context): lambda_metric("test.metric", 100) # Turn off _force_new to emulate the nested wrapper scenario, # the second @datadog_lambda_wrapper should actually be no-op. datadog_lambda_wrapper._force_new = False lambda_handler_double_wrapped = datadog_lambda_wrapper(lambda_handler) lambda_event = {} lambda_handler_double_wrapped(lambda_event, get_mock_context()) self.mock_patch_all.assert_called_once() self.mock_wrapper_lambda_stats.flush.assert_called_once() self.mock_submit_invocations_metric.assert_called_once()