CentOS Question posted in
The official documentation can be found here.

Node-RED docker problem reading directory contents – CentOS

I have a Node-RED app running in a docker container, with the aim to periodically read contents of a directory where .csv files are constantly updated and new .csv files are sometimes added. The point is to read new entries periodically, parse data, and send it onward.

I have not utilized the numerous ‘contrib’ nodes, as I have enabled the NodeJS ‘fs’ module and played with it. Additionally the built-in ‘file’ and ‘file in’ Node-RED modules are useful when reading the .csv files’ contents, so that is not an issue.

The problem comes with the new .csv files being added into the directory where all the .csv files are. I want be able to read all the file names and subsequently read all the .csv files.

I have mounted the .csv file directory into the docker container, and when testing whether I’m able to read the file names, weird things happen. Even though the files are visible in the container (viewed using docker exec -it CONTAINER /bin/bash) a piece of code containing fs.readdir does not list the files. When I try the fs.readdir too see the contents of /data directory, which is mounted into the container, it lists the contents like 10 % of the time (injecting a timestamp into the node to run it)

snip-of-nodered-flow

As you can see from the image, the contents of the directorty in question are not listed on every execution of the node. The contents of the mounted directory containing the .csv files are never listed upon running this node with the correct path as parameter.

The operating system is CentOS 7, where I am not a sudoer. I have managed to make it so that none of the mounted files or directories are owned by root, so they are owned by user node-red within the container. I managed to pull this directory file listing through on my ubuntu where I am a sudoer, but as none of the stuff is root-owned there either, I am not sure if that is the problem. I have a feeling this might be an operating system -relating thing.

Notes:

  • All relevant files and directories have permissions rwxr-xr-x
  • I have tried to mount the .csv files containing directory under /data directory, and as its own directory directly under root as /files
  • I am able to read the file contents with the Node-RED file nodes, just not the directories. Reading static file names is not enough as the directory contents keep changing
  • I have enabled NodeJS ‘fs’ module from the settings.js file which is mounted into the container
  • The Node-RED node (in image) does not output any errors (I tried this by adding an error return to the function in the image)
  • I have tried to run the Node-RED container as root user and without defining the user
  • I am running the Node-RED container using docker-compose

I hope this was not too much text or too unclear, I just wanted to make sure at least most of the stuff I have tried would be written here. If someone has some insight on the workings of Node-RED under docker and using the NodeJS fs module, it would be most appreciated 🙂

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    To answer my question of why Node-RED was unable to read directory contents most of the time, it was because of using the asynchronous fs.readdir module. When I switched to using the synchronous version fs.readdirSync, Node-RED was able to read directory contents without problems.


  2. 0

    The core Watch node should do all of this for you, no need to write function nodes.

    enter image description here

    If you want walk subdirectories make sure you tick the right box in the config.

    From the Sidebar docs for the watch node:

    The full filename of the file that actually changed is put into
    msg.payload and msg.filename, while a stringified version of the watch
    list is returned in msg.topic.

    msg.file contains just the short filename of the file that changed.
    msg.type has the type of thing changed, usually file or directory,
    while msg.size holds the file size in bytes.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search