CentOS Question posted in
The official documentation can be found here.

Openshift OKD Excessive Logging – CentOS

So I installed a single host Openshift OKD v3.11 cluster. I installed it on a VM running Centos 7.8.2003.

It seems to have installed ok except that it continually streams verbose logs to /var/log/messages. Around 5 logs per second and all seem to be about throttling requests. Example of a typical log message:

******Jun 13 15:49:13 centos7 journal: I0613 14:49:13.011402 1 request.go:485] Throttling request took 196.341689ms, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-service-cert-signer/serviceaccounts/service-serving-cert-signer-sa*****

The only reference I have managed to find is a question here but the access to the discussion is only available to those with deep pockets.
https://access.redhat.com/solutions/3348921

I assume these logs are nothing to worry about and so my main question is what is the “best”/cleanest/simplest/easiest way to ensure the Openshift cluster doesn’t continue to fill up /var/log/messages but will still log any important messages there?

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    Simon's answer was helpful but I've finally got to the bottom of this.

    The problem was simply that the version of Docker I had installed was old. At the time of writing the latest version of Centos is 7.8.2003 and if you install that and then simply run "yum install docker" hoping that you'll get something at least reasonably new and certainly compatible with the rest of the linux installation, you'll probably be making a mistake.

    The right thing to do is to follow the simple steps here: https://docs.docker.com/engine/install/centos/

    The reason I found the problem was because excessive logging of my openshift cluster wasn't the only issue. I started seeing strange behaviour of other containers. A process of trial and error narrowed down the issue to the default Centos version of docker. Once I followed the page above all my problem vanished including the original problem of /var/log/message getting hammered by openshift containers.

    The main reason I decided to answer my own question was because surely someone else is going to be as impatient/thick as me and simply install Centos7 then try "yum install docker" without knowing they're about to enter a world of pain.


  2. 0

    I would recommend looking at the root cause for this behavior. These messages indicate that there are a lot of requests coming to your API. Typically this is due to some application performing calls in a tight loop leading to this many messages. In your case check your openshift-service-cert-signer if you can see any warnings or an abnormal amount of log messages.

    If you want to get rid of the throttling messages, you can increase the amount of Queries per second (QPS) for the API server: Recommended Practices for OKD Master Hosts (lower part).

    The only reference I have managed to find is a question here but the access to the discussion is only available to those with deep pockets. https://access.redhat.com/solutions/3348921

    I do not understand why you’re saying that, as I can access that document with my free Red Hat account without any subscriptions. Have you tried with a free account as it says on the site?

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search