ECK Filebeat Daemonset Forwarding To Remote Cluster – Nginx

I wish to forward logs from remote EKS clusters to a centralised EKS cluster hosting ECK.

Versions in use:

• EKS v1.20.7
• Elasticsearch v7.7.0
• Kibana v7.7.0
• Filebeat v7.10.0

The setup is using a AWS NLB to forward requests to Nginx ingress, using host based routing.

When the DNS lookup (filebeat test output) for the Elasticsearch is tested on Filebeat, it validates the request.

But the logs for Filebeat are telling a different story.

    2021-10-05T10:39:00.202Z        ERROR   [publisher_pipeline_output]     
pipeline/output.go:154  Failed to connect to backoff(elasticsearch(https://elasticsearch.dev.example.com:9200)): 
Get "https://elasticsearch.dev.example.com:9200": Bad Request

The Filebeat agents can connect to the remote Elasticsearch via the NLB, when using a curl request.

The config is below. NB: dev.example.com is the remote cluster hosing ECK.

app:
  name: "filebeat"
  configmap:
    enabled: true
    filebeatConfig:
      filebeat.yml: |-
        filebeat.autodiscover:
          providers:
            - type: kubernetes
              node: ${NODE_NAME}
              hints.enabled: true
              templates:
                - config:
                    - type: container
                      paths:
                        - /var/lib/docker/containers/*/${data.kubernetes.container.id}-json.log
                      exclude_lines: ["^\s+[\-`('.|_]"] 
                      processors:
                        - drop_event.when.not.or:
                            - contains.kubernetes.namespace: "apps-"
                            - equals.kubernetes.namespace: "cicd"
                        - decode_json_fields:
                            fields: ["message"]
                            target: ""
                            process_array: true
                            overwrite_keys: true
                        - add_fields:
                            fields:
                              kubernetes.cluster.name: dev-eks-cluster
                            target: ""

        processors:
          - add_cloud_metadata: ~
          - add_host_metadata: ~

        cloud:
          id: '${ELASTIC_CLOUD_ID}'
        cloud:
          auth: '${ELASTIC_CLOUD_AUTH}'

        output:
          elasticsearch:
            enabled: true
            hosts: "elasticsearch.dev.example.com"
            username: '${ELASTICSEARCH_USERNAME}'
            password: '${ELASTICSEARCH_PASSWORD}'
            protocol: https
            ssl:
              verification_mode: "none"
            headers:
              Host: "elasticsearch.dev.example.com"
            proxy_url: "https://example.elb.eu-west-2.amazonaws.com"
            proxy_disable: false

  daemonset:
    enabled: true
    version: 7.10.0
    image:
      repository: "docker.elastic.co/beats/filebeat"
      tag: "7.10.0"
      pullPolicy: Always
    extraenvs:
      - name: ELASTICSEARCH_HOST
        value: "https://elasticsearch.dev.example.com"
      - name: ELASTICSEARCH_PORT
        value: "9200"
      - name: ELASTICSEARCH_USERNAME
        value: "elastic"
      - name: ELASTICSEARCH_PASSWORD
        value: "remote-cluster-elasticsearch-es-elastic-user-password"
    resources:
      limits:
        memory: 200Mi
      requests:
        cpu: 100m
        memory: 100Mi
  clusterrolebinding:
    enabled: true
    namespace: monitoring
  clusterrole:
    enabled: true
  serviceaccount:
    enabled: true
    namespace: monitoring
  deployment:
    enabled: false
    configmap:
      enabled: false

Any tips or suggestions on how to enable Filebeat forwarding, would be much appreciated 🙂

#1 Missing ports:

Even with the ports added in as suggested. Filebeat is erroring with:

2021-10-06T08:34:41.355Z ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://elasticsearch.dev.example.com:9200)): Get "https://elasticsearch.dev.example.com:9200": Bad Request