Question 364878 in Azure Question posted in
The official documentation to get you started can be found here.

Azure – Install a private package with credentials from requirements.txt

I want to a private package via

%pip install -r /dbfs/requirements.txt

Inside this requirements.txt is give:

git+https://<user>:$token@<gitprovider>.com/<path/to/repo>

If i give user and token inside the requirements.txt it works like a charm, but I want to pass user and token via dbutils.secrets. Is this doable without manipulating the requirements.txt before?

I thought it would be possible to create environment variables, but this did not work for me.

%env GIT_USERNAME= ...
%env GIT_TOKEN= ...
Tags:

2

Answers


  1. 0

    The way is using python

    1. Getting information
    2. Store the information in a temporary file so that its details will not be found in the run log or records after running
    3. Run
    import dbutils

username = dbutils.secrets.get(scope="your_scope", key="git_username")
token = dbutils.secrets.get(scope="your_scope", key="git_token")

repo_url = f"git+https://{username}:{token}@gitprovider.com/path/to/repo"

requirements_path = "/dbfs/tmp/requirements.txt"

with open(requirements_path, "w") as file:
    file.write(repo_url + "n")

dbutils.notebook.run("%pip install -r " + requirements_path, 60)
    Login or Signup to reply.
  2. 0

    I believe the simplest way would be to use a ~/.netrc file, that way you don’t pollute your requirements.txt with usernames/passwords.

    If you’ve never used a ~/.netrc file, then touch ~/.netrc first, otherwise add this line (either programmatically or manually)

    machine <gitprovider>.com login <user> password <token>

    Now your requests should pick up the authentication without you having to manually specify it, i.e. this is what your requirement.txt line is

    git+https://<gitprovider>.com/<path/to/repo>

    Note: If this is running in a pipeline, etc. you can also use a ~/.netrc, and clear it up when the pipeline ends or rely on whatever sort of garbage collection exists.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search