influx db subscriptions

replicating data to multiple hosts or instances

InfluxDB subscriptions are local or remote endpoints to which all data written to InfluxDB is copied. Subscriptions are primarily used with Kapacitor, but any endpoint able to accept UDP, HTTP, or HTTPS connections can subscribe and receive a copy of all data as it is written.

The number of writer goroutines is defined by the write-concurrency configuration. setting the subscriber write-concurrency to greater than 1 increases subscriber write throughput, it can result in out-of-order writes under high ingest rates. Setting write-concurrency to 1 ensures writes are passed to subscriber endpoints sequentially, but can create a bottleneck under high ingest rates. InfluxQL subscription statements

SHOW SUBSCRIPTIONS outputs all subscriber URL in plain text, including authentication credentials.
Example output: 
 
time influx -format csv -database _internal -execute 'show subscriptions'

name,retention_policy,name,mode,destinations
_internal,monitor,kapacitor-UUID,ANY,[http://localhost:9092]

ruuvi,autogen,kapacitor-UUID,ANY,[http://localhost:9092]
ruuvi,half_yr,kapacitor-UUID,ANY,[http://localhost:9092]

telegraf,autogen,kapacitor-UUID,ANY,[http://localhost:9092]

chronograf,autogen,kapacitor-UUID,ANY,[http://localhost:9092]

   199.4 wall time;  95.275 user; 1.214 sys; 48.37%
CREATE SUBSCRIPTION subscription_name
ON db_name.retention_policy DESTINATIONS ALL|ANY host[, …] 		ANY : Round-robin writes between multiple hosts

host: [HTTP| HTTPS| UDP ] ://hostname:pport

Examples:

  • CREATE SUBSCRIPTION "sub0" ON "mydb"."autogen" DESTINATIONS ALL 'http://example.com:9090'
  • round-robins the data to 'h1ost:9090' and 'h2ost:9090'
    CREATE SUBSCRIPTION "sub0" ON "mydb"."autogen" DESTINATIONS ANY 'udp://h1ost:9090', 'udp://h2ost:9090'
  • send data to another InfluxDB on 'host.com:8086' with authentication
    CREATE SUBSCRIPTION "sub0" ON "mydb"."autogen" DESTINATIONS ALL 'http://subscriber:secret@host.com:8086'
    If creating a Kapacitor subscription, defined by the subscription-protocol in the [[influxdb]] section of kapacitor.conf.
    • DROP SUBSCRIPTION subscription_name
    ON db_name.retention_policy     		DROP SUBSCRIPTION "sub0" ON "mydb"."autogen"

    Drop all subscriptions

    bash script loops through all subscriptions, and removes them. 
    # Environment variable exports:
# comment these if INFLUXUSER and INFLUXPASS are globally set.
export INFLUXUSER=influxdb-username
export INFLUXPASS=influxdb-password


IFS=$'\n'
 for i in $(influx -format csv -username $INFLUXUSER -password $INFLUXPASS -database _internal -execute 'show subscriptions' | tail -n +2 | grep -v name)

 do 
influx -format csv -username $INFLUXUSER -password $INFLUXPASS -database _internal \
-execute "drop subscription \"$(echo "$i" | cut -f 3 -d ',')\" ON \"$(echo "$i" | cut -f 1 -d ',')\".\"$(echo "$i" | cut -f 2 -d ',')\""

 done

    Configure InfluxDB subscriptions

    InfluxDB subscription configuration options are available in the [subscriber] section of the influxdb.conf.
    To use subcriptions, the enabled option in the [subscriber] section must be set to true.

    example influxdb.conf subscriber configuration: 

     
[subscriber]
  enabled = true
  http-timeout = "30s"
  insecure-skip-verify = false
  ca-certs = ""
  write-concurrency = 40
  write-buffer-size = 1000
    Descriptions of [subscriber] configuration options are available in the Configuring InfluxDB documentation.

    Troubleshooting

    Inaccessible or decommissioned subscription endpoints InfluxDB assumes the endpoint should receive data and will continue to attempt to send data. If an endpoint host is inaccessible or has been decommissioned, errors like
    # Some message content omitted (...) for the sake of brevity 
    "Post http://x.y.z.a:9092/write?consistency=...: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" ... service=subscriber
"Post http://x.y.z.a:9092/write?consistency=...: dial tcp x.y.z.a:9092: getsockopt: connection refused" ... service=subscriber
"Post http://x.y.z.a:9092/write?consistency=...: dial tcp 172.31.36.5:9092: getsockopt: no route to host" ... service=subscriber
    This may be caused by a networking error preventing a successful connection to the subscription endpoint. or the subscription endpoint no longer exists and the subscription hasn't been dropped.

    InfluxDB does not know if a subscription endpoint will become accessible again, subscriptions are not automatically dropped when an endpoint becomes inaccessible.

    Log entries like:

    2019-12-05 16:31:12  daemon.info influxd[15135]: ts=2019-12-05T21:31:12.382670Z lvl=info 
    msg="Post http://localhost:9092/write?consistency=&db=_internal&precision=ns&rp=monitor: 
    dial tcp [::1]:9092: connect: connection refused" log_id=0JXX36AG000 service=subscriber
    Are due to subscription Edit /etc/influxdb/influxdb.conf and set subscriotion enable = false