How to Collect Neo4j's `query.log`/Stream from Log4j Inside a Container Using Azure Monitor Agent in AKS

Jeffrey_Mutethia
October 27, 2024
146 views
1 vote
2 Answers

I’m running a Neo4j database inside a Kubernetes cluster on Azure Kubernetes Service (AKS). The Neo4j pod writes its logs, including query.log, to the filesystem inside the container at /logs/query.log.

I want to collect the query.log (or stream from Log4j) and send it to Azure Log Analytics using the Azure Monitor Agent (AMA) for centralized logging and monitoring.

I’ve tried the following steps:

Enabled Azure Monitor for Containers:
- Enabled monitoring on my AKS cluster via the Azure Portal.
- Deployed the Azure Monitor Agent as a DaemonSet in the kube-system namespace.

Created a ConfigMap to Configure AMA:

Created a ConfigMap named container-azm-ms-agentconfig in the kube-system namespace with the following content:

apiVersion: v1
kind: ConfigMap
metadata:
  name: container-azm-ms-agentconfig
  namespace: kube-system
data:
  config.yaml: |
    schema-version: v1
    config-version: 1.0
    logs:
      - name: neo4j-query-log
        enabled: true
        namespace: graph
        containerNames:
          - neo4j
        filePaths:
          - /logs/query.log

Applied the ConfigMap:
- Used kubectl apply -f ama-neo4j-config.yaml to apply the configuration.

However, after waiting and checking, I do not see the logs from query.log appearing in Azure Log Analytics. The agent seems to be working, and other logs are being collected, but not the ones from inside the container.

I’ve learned that the Azure Monitor Agent cannot access files inside a container’s filesystem directly.

Question:

How can I configure the Azure Monitor Agent to collect the query.log or stream logs from Log4j inside the Neo4j container and send them to Azure Log Analytics? Is there a recommended approach to achieve this?

Additional Information:

Neo4j is running in a container in the graph namespace.
The container name is neo4j.
The query.log file is located at /logs/query.log inside the container.
I prefer not to modify the Neo4j container image if possible.
I’m open to using sidecar containers or other methods that align with best practices.

Answers

As mentioned in comment a sidecar container can be used to read the query.log file from within the Neo4j container and make it accessible to AMA, which can then collect it from a shared node directory. Edit your Neo4j deployment file to add a sidecar container that reads query.log and outputs it to a shared node directory.

Example

apiVersion: apps/v1
kind: Deployment
metadata:
  name: neo4j
  namespace: graph
spec:
  replicas: 1
  selector:
    matchLabels:
      app: neo4j
  template:
    metadata:
      labels:
        app: neo4j
  spec:
    containers:
      - name: neo4j
        image: neo4j:latest
        volumeMounts:
          - name: logs
            mountPath: /logs
      - name: log-forwarder
        image: busybox
        command: ["/bin/sh", "-c", "tail -F /logs/query.log > /node-logs/query.log"]
        volumeMounts:
          - name: logs
            mountPath: /logs  # Neo4j log directory
          - name: node-logs
            mountPath: /node-logs  # Shared directory on the node
            readOnly: false
    volumes:
      - name: logs
        emptyDir: {}  # Log storage within the pod
      - name: node-logs
        hostPath:
          path: /var/log/neo4j
          type: DirectoryOrCreate

Now the sidecar is forwarding query.log to /var/log/neo4j on the host node, configure AMA to collect logs from this location.

Update your ConfigMap as below

apiVersion: v1
kind: ConfigMap
metadata:
  name: container-azm-ms-agentconfig
  namespace: kube-system
data:
  config.yaml: |
    schema-version: v1
    config-version: 1.0
    logs:
      - name: neo4j-query-log
        enabled: true
        filePaths:
          - /var/log/neo4j/query.log
        namespace: graph

Enable monitoring on the cluster and link it to your Log Analytics workspace

az aks enable-addons --resource-group arkorg --name Neo4jakscluster --addons monitoring --workspace-resource-id "/subscriptions/abcd-efg-hijk-lmnop-912345bc7d/resourceGroups/arkorg/providers/Microsoft.OperationalInsights/workspaces/Neo4jloganalytics"

This approach ensures that query.log is ingested into Azure Log Analytics for centralized monitoring without modifying the Neo4j container image.

An alternative is writing the logs to the container stream instead of on the filesystem.

Here is a log4j2 configuration that works well for us on EKS :

neo4j:
  image: #
    # ...
  neo4j:
    name: neo4j-dev
    # ...
  logging:
    serverLogsXml: |-
      <?xml version="1.0" encoding="UTF-8"?>
      <Configuration status="ERROR" monitorInterval="30" packages="org.neo4j.logging.log4j">
          <Appenders>
              <!-- Default debug.log, please keep -->
              <RollingRandomAccessFile name="DebugLog" fileName="${config:server.directories.logs}/debug.log"
                                      filePattern="$${config:server.directories.logs}/debug.log.%02i">
                  <Neo4jDebugLogLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSSZ}{GMT+0} %-5p [%c{1.}] %m%n"/>
                  <Policies>
                      <SizeBasedTriggeringPolicy size="20 MB"/>
                  </Policies>
                  <DefaultRolloverStrategy fileIndex="min" max="7"/>
              </RollingRandomAccessFile>

              <RollingRandomAccessFile name="HttpLog" fileName="${config:server.directories.logs}/http.log"
                                      filePattern="$${config:server.directories.logs}/http.log.%02i">
                  <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSSZ}{GMT+0} %-5p %m%n"/>
                  <Policies>
                      <SizeBasedTriggeringPolicy size="20 MB"/>
                  </Policies>
                  <DefaultRolloverStrategy fileIndex="min" max="5"/>
              </RollingRandomAccessFile>

              <Console name="ConsoleAppender" target="SYSTEM_OUT">
                  <JsonTemplateLayout eventTemplateUri="classpath:org/neo4j/logging/QueryLogJsonLayout.json"/>
              </Console>
          </Appenders>

          <Loggers>
              <!-- Log levels. One of DEBUG, INFO, WARN, ERROR or OFF -->

              <!-- The debug log is used as the root logger to catch everything -->
              <Root level="INFO">
                  <AppenderRef ref="DebugLog"/> <!-- Keep this -->
              </Root>

              <!-- The query log, must be named "QueryLogger" -->
              <Logger name="QueryLogger" level="INFO" additivity="false">
                  <AppenderRef ref="ConsoleAppender"/>
              </Logger>

              <!-- The http request log, must be named "HttpLogger" -->
              <Logger name="HttpLogger" level="INFO" additivity="false">
                  <AppenderRef ref="HttpLog"/>
              </Logger>

              <!-- The security log, must be named "SecurityLogger" -->
              <Logger name="SecurityLogger" level="INFO" additivity="false">
                  <AppenderRef ref="ConsoleAppender"/>
              </Logger>
          </Loggers>
      </Configuration>
# End of Neo4j's chart configuration

Please signup or login to give your own answer.

Click here to cancel reply.