@splunk/otel

The Splunk distribution of OpenTelemetry Node Instrumentation provides a Node agent that automatically instruments your Node application to capture and report distributed traces to Splunk APM.

Usage no npm install needed!

<script type="module">
  import splunkOtel from 'https://cdn.skypack.dev/@splunk/otel';
</script>

README

Beta GitHub release (latest by date) npm node-current Codecov GitHub branch checks state

Splunk distribution of OpenTelemetry JS for NodeJS

The Splunk distribution of OpenTelemetry JS automatically instruments your Node application to capture and report distributed traces to Splunk APM.

This Splunk distribution comes with the following defaults:

If you're currently using the SignalFx Tracing Library for Node and want to migrate to the Splunk Distribution of OpenTelemetry Node, see Migrate from the SignalFx Tracing Library for JS.

:construction: This project is currently in BETA. It is officially supported by Splunk. However, breaking changes MAY be introduced.

Getting Started

Assuming the default Splunk APM setup with OpenTelemetry Collector running on localhost. If you're running a different setup, refer to the configuration options below to customize trace export endpoint and other behaviour.

  1. Install @splunk/otel package
npm install @splunk/otel --save
  1. Install instrumentation packages
npm install @opentelemetry/instrumentation-http --save

Note: If you are using NPM 6 or older, it'll warn you about missing peer dependencies. All of these dependencies are instrumentation packages and are completely optional. You can install the ones you need and ignore the rest. NPM 7+ supports optional peer dependencies feature and will not complain about this.

You can find a list of instrumentation packages supported out of the box here.

You can also install additional packages and use them as described here.

  1. Run node app with -r @splunk/otel/instrument CLI argument
export OTEL_SERVICE_NAME=my-node-svc
node -r @splunk/otel/instrument app.js

That's it - the telemetry data is not sent to the locally running Opentelemetry Collector! You can also instrument your app with code as described here.

Splunk APM

In order to send traces directly to Splunk APM, you need to:

  1. Set OTEL_EXPORTER_OTLP_ENDPOINT to https://ingest.<realm>.signalfx.com/v2/trace/otlp where realm is your Splunk APM realm e.g, https://ingest.us0.signalfx.com/v2/trace/otlp.
  2. Set SPLUNK_ACCESS_TOKEN to your Splunk APM access token.

Configuration options

Environment variable Config Option Default value Notes
OTEL_EXPORTER_OTLP_ENDPOINT endpoint http://localhost:55681/v1/traces The OTLP endpoint to export to. Only OTLP over HTTP is supported.
OTEL_TRACES_EXPORTER tracesExporter otlp Chooses the exporter. Shortcut for setting spanExporterFactory. One of [otlp, jaeger-thrift-http, jaeger-thrift-splunk]. See TracesExporter.
OTEL_PROPAGATORS propagators tracecontext,baggage Comma-delimited list of propagators to use. Valid keys: baggage, tracecontext, b3multi, b3.
OTEL_SERVICE_NAME serviceName unnamed-node-service The service name of this Node service.
SPLUNK_ACCESS_TOKEN acceessToken The optional access token for exporting signal data directly to SignalFx API.
SPLUNK_MAX_ATTR_LENGTH maxAttrLength 1200 Maximum length of string attribute value in characters. Longer values are truncated.
SPLUNK_TRACE_RESPONSE_HEADER_ENABLED serverTimingEnabled true Enable injection of Server-Timing header to HTTP responses.
SPLUNK_LOGS_INJECTION logInjectionEnabled false Enable injecting of trace ID, span ID and service name to log records. Please note that the corresponding logging library instrumentation needs to be installed.
OTEL_RESOURCE_ATTRIBUTES Comma-separated list of resource attributes added to every reported span.
Examplekey1=val1,key2=val2
OTEL_TRACE_ENABLED true Globally enables tracer creation and auto-instrumentation.

More details on config options can be seen here

Automatically instrument an application

You can use node's -r CLI flag to pre-load the instrumentation module and automatically instrument your NodeJS application. For example, if you normally started your application as follows:

node index.js

Then you can automatically instrument your application by running

node -r @splunk/otel/instrument index.js

Manually instrument an application

You can also manually instrument your application by adding the following lines before everything else in your application.

const { startTracing } = require('@splunk/otel');

startTracing();

// rest of your application entry point script

startTracing() accept an optional Options argument. It can be used to customize many aspects of the tracing pipeline. For example:

startTracing({
  serviceName: 'my-node-service',
});

Please note that startTracing is destructive to Open Telemetry API globals. We provide the stopTracing method, but it won't revert to OTel API globals set before startTracing was run, it will only disable globals, which startTracing set.

All config options

startTracing() accepts an optional argument to pass down configuration. The argument must be an Object and may contain any of the following keys.

  • endpoint: corresponds to the OTEL_EXPORTER_OTLP_ENDPOINT environment variable. Defaults to http://localhost:55681/v1/traces. Configures the http endpoint to which all spans will be exported.

  • serviceName: corresponds to the OTEL_SERVICE_NAME environment variable. Defaults to unnamed-node-service. Configures the service name of the instrumented node service. The name is added to all spans as an attribute.

  • accessToken: corresponds to the SPLUNK_ACCESS_TOKEN environment variable. Configures the access token that should be used to authenticate with the span exporter http endpoint. Used when exporting directly to Splunk APM from a Node service.

  • maxAttrLength: corresponds to the SPLUNK_MAX_ATTR_LENGTH environment variable. Defaults to 1200. Configures the maximum length any span attribute value can have. Values longer than the specified length will be truncated.

  • serverTimingEnabled: corresponds to the SPLUNK_SERVER_TIMING_ENABLED environment variable. Defaults to false. Enables injection of Server-Timing header to responses.

  • logInjectionEnabled: corresponds to the SPLUNK_LOGS_INJECTION environment variable. Defaults to false. Injects trace ID, span ID and service name to the log records. Service version or deployment environment will be injected if available in the configured resource. Supported logging libraries: bunyan, pino, winston.

  • tracerConfig: a JS object that is merged into the default tracer config replacing any existing keys and is passed on to the tracer provider during initialization. This can be used to customize the tracer provider or tracer. Must satisfy TracerConfig interface

  • spanExporterFactory: A function that accepts the options passed to startTracing function and returns a new instance of SpanExporter. When set, this function will be used to create a new exporter and the returned exporter will be used in the pipeline.

  • spanProcessorFactory: A function that accepts the options passed to startTracing function and returns a SpanProcessor instance or an array of SpanProcessor instances. When set, this function is be used to create one or more span processor(s). The returned processors are added to the global tracer provider and configured to process all spans generated by any tracer provider by the global provider. The function is responsible for creating a new span exporter and using it with each processor it creates. It may call options.spanExporterFactory(options) to create a new exporter as configured by the user.

  • propagatorFactory: A function that accepts the options passed to startTracing function and returns a new instance of a TextMapPropagator. Defaults to a composite propagator comprised of W3C Trace Context and Baggage propagators.

  • instrumentations: defaults to the list of instrumentation listed below. Can be used to enable additional instrumentation packages. Refer examples here

Using additional instrumentation plugins

If you setup tracing manually by calling the startTracing() method, you can use custom or 3rd party instrumentations as long as they implement the OpenTelemetry JS Instrumentation interface. Custom instrumentations can be enabled by passing them to the startTracing() method as follows:

const { startTracing } = require('@splunk/otel');

startTracing({
  instrumentations: [
    new MyCustomInstrumentation(),
    new AnotherInstrumentation(),
  ]
});

You can also add the default set of instrumentation to the list as follows:

const { startTracing } = require('@splunk/otel');
const { getInstrumentations } = require('@splunk/otel/lib/instrumentations');

startTracing({
  instrumentations: [
    ...getInstrumentations(),
    new MyCustomInstrumentation(),
    new AnotherInstrumentation(),
  ]
});

Default Instrumentation Packages

By default the following instrumentations will automatically be enabled if they are installed. In order to use any of these instrumentations, you'll need to install them with npm and then run your app with -r @splunk/otel/instrument flag as described above.

@opentelemetry/instrumentation-dns
@opentelemetry/instrumentation-express
@opentelemetry/instrumentation-graphql
@opentelemetry/instrumentation-grpc
@opentelemetry/instrumentation-hapi
@opentelemetry/instrumentation-http
@opentelemetry/instrumentation-ioredis
@opentelemetry/instrumentation-koa
@opentelemetry/instrumentation-mongodb
@opentelemetry/instrumentation-mysql
@opentelemetry/instrumentation-net
@opentelemetry/instrumentation-pg
opentelemetry-instrumentation-amqplib
opentelemetry-instrumentation-aws-sdk
opentelemetry-instrumentation-elasticsearch
opentelemetry-instrumentation-kafkajs
opentelemetry-instrumentation-mongoose
opentelemetry-instrumentation-sequelize
opentelemetry-instrumentation-typeorm

If log injection is enabled, the corresponding logging library package will need to be installed beforehand. Supported logging library instrumentations:

@opentelemetry/instrumentation-bunyan
@opentelemetry/instrumentation-pino
@opentelemetry/instrumentation-winston

You can find more instrumentation packages over at the OpenTelemetry Registry and enable them manually as described above.

Troubleshooting

TODO:

License and versioning

The Splunk distribution of OpenTelemetry JS Instrumentation is a distribution of the OpenTelemetry JS project. It is released under the terms of the Apache Software License version 2.0. See the license file for more details.