I have a problem about generating log info when I work in Docker. There is no issue to write logs in log file in localhost.
I cannot see any new logs when I implement the CRUD process during docker.
How can I connect log file(Springboot-Elk.log) to Docker?
How can I fix it?
Here is the file showing screenshots : Link
Here is my project link : My Project
Here is the docker-compose.yml shown below
version: '3.8'
services:
logstash:
image: docker.elastic.co/logstash/logstash:7.15.2
user: root
command: -f /etc/logstash/conf.d/
volumes:
- ./elk/logstash/:/etc/logstash/conf.d/
- ./Springboot-Elk.log:/tmp/logs/Springboot-Elk.log
ports:
- "5000:5000"
environment:
LS_JAVA_OPTS: "-Xmx256m -Xms256m"
depends_on:
- elasticsearch
filebeat:
build:
context: ./filebeat
dockerfile: Dockerfile
links:
- "logstash:logstash"
volumes:
- /var/run/docker.sock:/host_docker/docker.sock
- /var/lib/docker:/host_docker/var/lib/docker
depends_on:
- logstash
kibana:
image: docker.elastic.co/kibana/kibana:7.15.2
user: root
volumes:
- ./elk/kibana/:/usr/share/kibana/config/
ports:
- "5601:5601"
depends_on:
- elasticsearch
entrypoint: ["./bin/kibana", "--allow-root"]
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.15.2
user: root
volumes:
- ./elk/elasticsearch/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
ports:
- "9200:9200"
- "9300:9300"
environment:
ES_JAVA_OPTS: "-Xmx256m -Xms256m"
app:
image: 'springbootelk:latest'
build:
context: .
dockerfile: Dockerfile
container_name: SpringBootElk
depends_on:
- db
- logstash
ports:
- '8077:8077'
environment:
- SPRING_DATASOURCE_URL=jdbc:mysql://db:3306/springbootexample?useSSL=false&allowPublicKeyRetrieval=true&serverTimezone=Turkey
- SPRING_DATASOURCE_USERNAME=springexample
- SPRING_DATASOURCE_PASSWORD=111111
- SPRING_JPA_HIBERNATE_DDL_AUTO=update
db:
container_name: db
image: 'mysql:latest'
ports:
- "3366:3306"
restart: always
environment:
MYSQL_DATABASE: ${MYSQL_DATABASE}
MYSQL_USER: ${MYSQL_USER}
MYSQL_PASSWORD: ${MYSQL_PASSWORD}
MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
volumes:
- db-data:/var/lib/mysql
# Volumes
volumes:
db-data:
Here is logstash.conf shown below
input {
beats {
port => 5000
}
file {
path => "/tmp/logs/Springboot-Elk.log"
sincedb_path => "/dev/null"
start_position => "beginning"
}
}
output {
stdout{
codec => rubydebug
}
elasticsearch {
hosts => "elasticsearch:9200"
index => "dockerlogs"
}
}
filebeat.yml file is shown below.
filebeat.inputs:
- type: docker
enabled: true
containers:
ids:
- "*"
path: "/host_docker/var/lib/docker/containers"
processors:
- add_docker_metadata:
host: "unix:///host_docker/docker.sock"
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
output.logstash:
hosts: ["logstash:5000"]
log files:
logging.level: info
logging.to_files: false
logging.to_syslog: false
loggins.metrice.enabled: false
logging.files:
path: /var/log/filebeat
name: filebeat
keepfiles: 7
permissions: 0644
ssl.verification_mode: none
Here is the Dockerfile of filebeat.yml
FROM docker.elastic.co/beats/filebeat:7.15.2
COPY filebeat.yml /usr/share/filebeat/filebeat.yml
USER root
RUN mkdir /usr/share/filebeat/dockerlogs
RUN chown -R root /usr/share/filebeat/
RUN chmod -R go-w /usr/share/filebeat/
As I want to see logs in logstash , I run this command docker container logs -f .
I cannot see any logs defined in PersonController and PersonService there.
Here is the screenshot
4
Answers
Here is my answer shown below.
After I revised logstash.conf file shown below, my issue was dissappeared.
When working with docker, its good to write all logs to the console. This will allow it to expose logs when run in Kubernetes or other ochestrators. On spring framework, you can achieve this by changing to ConsoleAppender. The example bellow shows how to achieve this in log4j.xml. Place the file to your resources folder and add the log4j dependencies(Ref: https://www.baeldung.com/spring-boot-logging ):
You can still configure log to disk by adding another appender in the configuration above, but you need to add a mount point to your docker-compose file to point to the logs directory on your application.
It is good to note that Docker is stateless and therefore logs are lost when you restart the container.
For me it works. But my ELK Stack is running not in docker.
This is my logstash config (same config for TCP and UDP):
And you have to setup the logback index in Kibana too.
This is my logback-spring.xml:
I’m logging directly (UDP) to ELK Stack.
And you need
According to your repository you have
logback
configuration for logging, but excudedlogback
and addedlog4j
support in your maven dependencies. To makelogback
work you just need not to exclude default logging library, and add newer version of thelogback
dependency (because specified by spring boot dependency version of logback doesn’t contain specified in your configuration logstash appender – net.logstash.logback.appender.LogstashTcpSocketAppender).F.e.:
But it is better to find a good guide on the internet how to setup logging of your spring boot application to logstash properly. And I don’t suggest you to use configuration in your repository for production purposes.