Question 117929 in Mongodb Question posted in
The official documentation can be found here.

Mongodb – Version controll for multiple files on an embedded system

I have an esp32-wrover-e ( 16mb flash and 8mb external ram )
I’m coding it with arduino framework.

On the esp32, there is a webserver and it has multiple files for login, the main page and things like that. I want to upgrade every file on the file system from a node js server remotelly. The firmware upgrade is nearly done.

For the firmware there is an object which contains the firmware version the date it is updated and two more arrays.

Like this:

var firmwareInfos = {
    version : 1.1,
    date : new Date().getTime() / 1000,
    downloads: [],
    queries: []
};

When the esp ask for a new version, it performs an HTTP GET request to the server. The server sends this object to the esp and it will decide if the version is bigger or not. If it is bigger on the server, it will download the new firmware ( with a separate HTTP request ). This is okay, and it works.

Now i want to check for files and their versions like this too. When the ESP asks for the firmware version, the server could send all the file versions along with the firmware too.

For this to work, the server would have an object containing all the file path’s and their versions, and the esp32 should hold this object on it’s file system too. This could work, but it can not scale.

If i want to add new files along with the new firmware to the esp, i have to manually add the new file path and it’s version to both the esp and the server’s version object.

I started on the server side with this object:

var versions = {
    firmware : {
        version : 1.1,
        date : new Date().getTime() / 1000,
        downloads: [],
        queries: []
    },
    main: {
        script:{
            version : "1.0",
            date : new Date().getTime() / 1000,
            downloads: [],
            queries: []
        },
        style:{
            version : "1.0",
            date : new Date().getTime() / 1000,
            downloads: [],
            queries: []
        },
        index:{
            version : "1.0",
            date : new Date().getTime() / 1000,
            downloads: [],
            queries: []
        },
    },
    login: {
        script:{
            version : "1.0",
            date : new Date().getTime() / 1000,
            downloads: [],
            queries: []
        },
        style:{
            version : "1.0",
            date : new Date().getTime() / 1000,
            downloads: [],
            queries: []
        },
        index:{
            version : "1.0",
            date : new Date().getTime() / 1000,
            downloads: [],
            queries: []
        },
    },
    admin: {
        script:{
            version : "1.0",
            date : new Date().getTime() / 1000,
            downloads: [],
            queries: []
        },
        style:{
            version : "1.0",
            date : new Date().getTime() / 1000,
            downloads: [],
            queries: []
        },
        index:{
            version : "1.0",
            date : new Date().getTime() / 1000,
            downloads: [],
            queries: []
        },
    }
};

This would have be the object what the esp is asking for. It isn’t complete, I haven’t thought about it all yet, because as you can see there are multiple key pairs in the object and i think it could be done much more simply. But again, this is the best approach i can think of, and it can not scale, because this object has to exist on the esp too, and i have to manually add new files to it if i want to extend the esp’s file system.

The other solution:

The esp would loop trought it’s file system, gather all the file names which is in there, put it into a json object and send it to the server asking for new versions for all of the files. The server would loop trought this object and it would check if there are objects with the same name in it’s database ( which is mongo db ) and it would create a new object with the versions for the files what it found. For this i’m thinking about an approach like this:

void fSystem::checkEntireFS(){
    SpiRamJsonDocument doc(15000);
    JsonArray rootArray = doc.to<JsonArray>();
    Serial.println("FS - Start checking entire FS");
    gatherFiles("/", 5, rootArray);
    serializeJsonPretty(doc, Serial);
    Serial.println("FS - End checking entire FS");
}

void fSystem::gatherFiles(const char* dirname, uint8_t levels, JsonArray &rootArray){
    File root = LittleFS.open(dirname);
    if(!root){
        Serial.println("- failed to open directory");
        return;
    }
    if(!root.isDirectory()){
        Serial.println(" - not a directory");
        return;
    }

    File file = root.openNextFile();
    while(file){
        String name = file.name();
        JsonObject entryObject = rootArray.createNestedObject();
        entryObject["path"] = dirname;
        entryObject["name"] = name;
        if(file.isDirectory()){
            entryObject["type"] = "dir";
            if(levels){
                gatherFiles(file.path(), levels -1, rootArray);
            }
        } else {
            entryObject["type"] = "file";
            entryObject["size"] = file.size();
            entryObject["date"] = file.getLastWrite();
        }
        vTaskDelay(1);
        file = root.openNextFile();
    }
}

Using ArduinoJson and the built in LittleFS for this to create an array of my files.

That way i could use the getLastWrite() or the size to determine if the file on the nodejs server is newer or not, and not have to rely on version number.

I’m also confused on the version variable. If i want three digit variable, i must use a char array. If i use long or double, i have only a two digit varaible like 1.1.

But with char array, i must loop trought the char array and compare each number. There must be a better way for this.

So the question is, what could be the best approach? How do you guys do multiple file version controlls? Can it be done more simple?

Thank you for your answers.

Edit*

The esp32 is inside a network behind a firewall far away from the server. The server is on a public network but the esp is not. The esp is not reachable from outside, only the server. Unfortunatelly i have very limited resources on the esp, like 100kb free heap is left. I could pull all the files from the node js server and refresh everything regardless of the versions, but there are a bunch of files and it would take so much time to download all of them on every request. There are multiple language files which is 15kb each and there are images which can be 70-100kb each. I want to download only the relevant files, which needs update.

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    The final solution has been the following.

    On the server from an admin UI the admin can upload any kind of file to the server. ( there is an exact copy of the esp's file system structure on the server, with the same paths ) On success upload, the server will create an object from this file, containing it's path, name and upload date.

    The server keeps track of these uploads, saves and updates them in mongoDB.

    When the esp asks for new files, the server gives this object to the esp. The esp loops trought this object and checks it's own file system for files based on the path of the object. If it finds this path, checks the last modified date for it. If it is lower, puts this file in a separate array and continues the loop. If the esp can not find a file on the provided path, it assumes that it is a new file, and puts this path to the separate array too.

    When it finished looping, it will ask the server to download these new files which are inside the separate array. And that's how i solved the multiple file update from server.

    File upload looks like this:

    var uploader    = require("express-fileupload");
var moduleHandles   = require("./moduleRoutes.js");

function uploadFile(file,path,res,key){
    file.mv(`${path}/${file.name}`,function(err){
        if(err){
            res.status(400).json({status: "error", message: `${file.name} upload failed!`});
        }else{
            moduleHandles.addNewFileInfo(`${path}/${file.name}`);
            res.status(200).json({status: "success", message: `${file.name} upload success!`});
        }
    });
}

module.exports = {
    initPaths: function(app){
        app.use(uploader({ createParentPath: true }));

        app.post('/fileUpload', function (req, res) {
            let path        = req.query.path;
            let fileKeys    = Object.keys(req.files);
            if( fileKeys.length > 0 ){
                fileKeys.forEach(function(key) {
                    uploadFile(req.files[key],path,res,key);
                });
            }else{
                res.status(400).json({status: "error", message: "Please choose at least one file to be able to upload!"});
            }
        });
    }
}

    In the module handles it looks like this:

    addNewFileInfo: function(filePath){
        // Removing any unnecessary path for the esp
        let pathKey = filePath.replace("./HsH_Files","");
        let fileInfo = {
            dateSec : parseInt(new Date().getTime() / 1000),
            name    : pathKey.split("/").pop(),
        };
        fileInfos[pathKey] = fileInfo;
        updateFileInfosInDB();
    },
    deleteFileInfoByPath: function(filePath){
        let elemPath = filePath.replace("./HsH_Files","");
        if( fileInfos.hasOwnProperty(elemPath) ){
            delete fileInfos[elemPath];
            asyncFileInfoUpdateInDB();
        }
    },

    It is not done yet on the esp side, but it would look something like this:

    void pSystem::checkNewFiles(){
    HTTPClient http;
    char fileCheckURL[200];
    http.begin( fileCheckURL );
    int httpCode = http.GET();

    if (httpCode > 0) {
        SpiRamJsonDocument infoJsonDoc(FILE_INFO_SIZE);
        DeserializationError error = deserializeJson(infoJsonDoc, http.getStream());
        if( !error ){
            JsonObject infoDoc = infoJsonDoc.as<JsonObject>();
            for (JsonPair infoRef : infoDoc) {
                const char* filePath    = infoRef.key().c_str();
                JsonObject fileInfo     = infoDoc[filePath];

                SpiRamJsonDocument fileDetailsDoc(FILE_DETAILS_SIZE);
                JsonObject fileDetails = fileDetailsDoc.to<JsonObject>();
                hsh_fileSystem.getFileDetailsOnPath(filePath, fileDetails);

                if( fileDetails["result"] ){
                    if( fileDetails["lastModified"].as<long>() < fileInfo["date"].as<long>() ){
                        // push it to downloadable files array.
                    }
                }else if( !fileDetails["result"] ){
                    if( !fileDetails["exists"] ){
                        hsh_fileSystem.createPath(filePath);
                        // push it to downloadable files array.
                    }
                }
            }
        }
    }
    http.end();
}

  2. 0

    I assume your motivation for this is a scalable/sparse update.

    If you really want to continue with your own version control solution, the char*, can be passed to a std::string constructor and then use the == operator for comparison. For detecting changes, I would use the file hash as file versions, but the date could work too for reasonably synchronized system clocks. If your files are only ever pulled from the server (and never written on the esp), then you already know the files the esp needs to download beforehand and avoid the file system code on the esp. I.e if the server gets a global version 5 in the request and the current global version is 8, send back a compressed archive with the changed files between versions 5 and 8. This is optimal in terms of delay and bandwidth and you can do all the heavy lifting on your server. It also reduces the protocol itself to a minimum. This can be extended to a continuous kind of versioning where you just pass the last pull time to the server and it sends back the files changed since then.

    Edit: git is not an option on ESP32-Arduino

    However, all this (and much more) is included in git. I would just install it on both machines and automate the calls through your code (child_process.exec on node.js and system() on c++). Basically, you need a git push on the server and a git pull on the robot. Transfer is quick and scalable. You may even self-host the git repository remote on your server machine, e.g. if you want to stay in a local network.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search