Description

A user was reporting 2x-3x memory consumption when using the Node.js SDK. I had a chat with them and will summarize the setup and findings here.

• SDK setup is a very ordinary express setup with mostly SDK defaults (traces sample rate: 0.5, 0.1 - doesn't matter, leaks more or less equally)
• Requests to the service are transient
• No long-running requests or anything
• Large objects are stored on the res object (in particular large array of strings containing identifiers)
• The memory snapshot shows that NonRecordingSpans and Spans are holding references to the res object via the _isolationScope field.
• The majority of these references seems to come from NonRecordingSpans.
• Interestingly, and I don't know if this is somehow relevant, the NonRecordingSpans are held by very deep chains of Timeout objects. The user does not define any timeouts manually.

My hypothesis: We are not cleaning up NonRecordingSpans "naturally" and keeping them in the exporter or somewhere and they end up being evicted by a timeout policy. This may be in our code, or this may be in OTELs code.