How do I set a secret variable in pipelines library using pipelines and/or powershell (On premise installation of Azure DevOps Server 2020)

You can set variables via api requests. I have done a similar setup to upload secure files, which is discussed here. Copying my solution from that thread:

        ### The variables secureFileFolder & secureFileName references variables I have defined higher up in the pipeline.
        - task: PowerShell@2
          enabled: true
          displayName: 'Upload Secure File'
          inputs:
            targetType: 'inline'
            script: |
              $header = @{ Authorization = "Bearer $env:ACCESSTOKEN" }
              $infile = "$(secureFileFolder)/$(secureFileName)"
              $organization = (Split-Path $(System.CollectionUri) -leaf)
              $uri = "https://dev.azure.com/$organization/$(System.TeamProject)/_apis/distributedtask/securefiles?api-version=5.0-preview.1&name=$(secureFileName)"
              
              Invoke-RestMethod -Uri $uri -Method Post -ContentType "application/octet-stream" -Headers $header -InFile "$infile"
          env:
            ACCESSTOKEN: $(System.AccessToken)

You should be able to do pretty much the same, but alter the uri to target the libraries instead. See Microsoft documentation about updating variables in a library here. Basically, call PUT https://dev.azure.com/{organization}/_apis/distributedtask/variablegroups/{groupId}?api-version=7.0 and set the body to match the variables you want to update.