skip to Main Content

I’m trying to debug the following build error in our CI where "A depends on B which can’t build because it depends on C." I’m building my data service which doesn’t directly depend on kafkaAvailMonitor.go which makes this error hard to trace. In other words:

data (what I’m building) depends on (?) which depends on
kafkaAvailMonitor.go

It may seem trivial to fix for a developer they just do "go get whatever" but I can’t do that as part of the release process – I have to find the person that added the dependency and ask them to fix it.

I’m aware that there are tools to visualize the dependency tree and other more sophisticated build systems, but this seems like a pretty basic issue: is there any way I can view the full dependency tree to see what’s causing the build issue?

go build -a -v

../../../msgq/kafkaAvailMonitor.go:8:2: cannot find package 
  "github.com/Shopify/sarama/tz/breaker" in any of:
  /usr/lib/go-1.6/src/github.com/Shopify/sarama/tz/breaker (from $GOROOT)
  /home/jenkins/go/src/github.com/Shopify/sarama/tz/breaker (from $GOPATH)
  /home/jenkins/vendor-library/src/github.com/Shopify/sarama/tz/breaker
  /home/jenkins/go/src/github.com/Shopify/sarama/tz/breaker
  /home/jenkins/vendor-library/src/github.com/Shopify/sarama/tz/breaker

6

Answers


  1. Chosen as BEST ANSWER

    The above answer still doesn't show me a dependency tree so I've taken the time to write a Python script to do what I need - hopefully that helps other people.

    The issue with the above solution (the others proposed like go list) is that it only tells me the top level. They don't "traverse the tree." This is the output I get - which doesn't help any more than what go build gives me.

    .../npd/auth/
       .../mon/mlog
       .../auth/service
    

    This is what I'm trying to get - I know that auth is broken (top) and that breaker is broken (bottom) from go build but I have no idea what's in between - my script below gives me this output.

    .../npd/auth/
        .../npd/auth/service                
            .../npd/auth/resource
                .../npd/auth/storage
                   .../npd/middleware
                      .../npd/metrics/persist
                        .../npd/kafka
                            .../vendor-library/src/github.com/Shopify/sarama
                                .../vendor-library/src/github.com/Shopify/sarama/vz/breaker
    

    My Python script:

    import subprocess
    import os
    
    folder_locations=['.../go/src','.../vendor-library/src']
    
    def getImports(_cwd):
        #When the commands were combined they overflowed the bugger and I couldn't find a workaround
        cmd1 = ["go", "list", "-f", " {{.ImportPath}}","./..."]
        cmd2 = ["go", "list", "-f", " {{.Imports}}","./..."]
    
        process = subprocess.Popen(' '.join(cmd1), cwd=_cwd,shell=True,
                                   stdout=subprocess.PIPE,
                                   stderr=subprocess.PIPE)
    
        out1, err = process.communicate()
    
        process = subprocess.Popen(' '.join(cmd2), cwd=_cwd,shell=True,
                                   stdout=subprocess.PIPE,
                                   stderr=subprocess.PIPE)
    
        out2, err = process.communicate()
        out2clean=str(out2).replace("b'",'').replace('[','').replace(']','').replace("'",'')
    
        return str(out1).split('\n'),out2clean.split('\n')
    
    def getFullPath(rel_path):
        for i in folder_locations:
            if os.path.exists(i+'/'+rel_path):
                return i+'/'+rel_path
        return None
    
    def getNextImports(start,depth):
    
        depth=depth+1
        indent = 't'*(depth+1)
    
        for i,val in enumerate(start.keys()):
    
            if depth==1:
                print (val)
    
            out1,out2=getImports(val)
    
            noDeps=True
            for j in out2[i].split(' '):
                noDeps=False
        
                _cwd2=getFullPath(j)
                new_tree = {_cwd2:[]}
                not_exists = (not _cwd2 in alltmp)
        
                if not_exists:
                    print(indent+_cwd2)
                    start[val].append(new_tree)
                    getNextImports(new_tree,depth)
                    alltmp.append(_cwd2)
    
            if noDeps:
                print(indent+'No deps')
    
    _cwd = '/Users/.../npd/auth'
    
    alltmp=[]
    start_root={_cwd:[]}
    getNextImports(start_root,0)
    

  2. if the following isn’t a stack trace what is it?

    It is the list of path where Go is looking for your missing package.

    I have no idea who is importing kafkaAvailMonitor.go

    It is not “imported”, just part of your sources and compiled.
    Except it cannot compile, because it needs github.com/Shopify/sarama/tz/breaker, which is not in GOROOT or GOPATH.

    Still, check what go list would return on your direct package, to see if kafkaAvailMonitor is mentioned.

    go list can show both the packages that your package directly depends, or its complete set of transitive dependencies.

    % go list -f '{{ .Imports }}' github.com/davecheney/profile
    [io/ioutil log os os/signal path/filepath runtime runtime/pprof]
    % go list -f '{{ .Deps }}' github.com/davecheney/profile
    [bufio bytes errors fmt io io/ioutil log math os os/signal path/filepath reflect run
    

    You can then script go list in order to list all dependencies.
    See this bash script for instance, by Noel Cower (nilium)

    #!/usr/bin/env bash
    # Usage: lsdep [PACKAGE...]
    #
    # Example (list github.com/foo/bar and package dir deps [the . argument])
    # $ lsdep github.com/foo/bar .
    #
    # By default, this will list dependencies (imports), test imports, and test
    # dependencies (imports made by test imports).  You can recurse further by
    # setting TESTIMPORTS to an integer greater than one, or to skip test
    # dependencies, set TESTIMPORTS to 0 or a negative integer.
    
    : "${TESTIMPORTS:=1}"
    
    lsdep_impl__ () {
        local txtestimps='{{range $v := .TestImports}}{{print . "n"}}{{end}}'
        local txdeps='{{range $v := .Deps}}{{print . "n"}}{{end}}'
    
        {
            go list -f "${txtestimps}${txdeps}" "$@"
            if [[ -n "${TESTIMPORTS}" ]] && [[ "${TESTIMPORTS:-1}" -gt 0 ]]
            then
                go list -f "${txtestimps}" "$@" |
                sort | uniq |
                comm -23 - <(go list std | sort) |
                    TESTIMPORTS=$((TESTIMPORTS - 1)) xargs bash -c 'lsdep_impl__ "$@"' "$0"
            fi
        } |
        sort | uniq |
        comm -23 - <(go list std | sort)
    }
    export -f lsdep_impl__
    
    lsdep_impl__ "$@"
    
    Login or Signup to reply.
  3. When using modules you may be able to get what you need from go mod graph.

    usage: go mod graph
    
    Graph prints the module requirement graph (with replacements applied)
    in text form. Each line in the output has two space-separated fields: a module
    and one of its requirements. Each module is identified as a string of the form
    path@version, except for the main module, which has no @version suffix.
    

    I.e., for the original question, run go mod graph | grep github.com/Shopify/sarama then look more closely at each entry on the left-hand side.

    Login or Signup to reply.
  4. I just want to mention here that go mod why can also help. Anyway you cannot get and display the whole tree. But you can trace back one single branch of a child dependency until its parent root.

    Example:

    $ go mod why github.com/childdep
    # github.com/childdep
    github.com/arepo.git/service
    github.com/arepo.git/service.test
    github.com/anotherrepo.git/mocks
    github.com/childdep
    

    That means, you have imported ‘childdep’ finally in ‘anotherrepo.git/mocks’.

    Login or Signup to reply.
  5. can try this https://github.com/vc60er/deptree

     redis git:(master) go mod graph | deptree -d 3
    package: github.com/go-redis/redis/v9
    dependence tree:
    
    ┌── github.com/cespare/xxhash/[email protected]
    ├── github.com/dgryski/[email protected]
    ├── github.com/fsnotify/[email protected]
    │    └── golang.org/x/[email protected]
    ├── github.com/nxadm/[email protected]
    │    ├── github.com/fsnotify/[email protected]
    │    │    └── golang.org/x/[email protected]
    │    └── gopkg.in/[email protected]
    ├── github.com/onsi/[email protected]
    │    ├── github.com/go-task/[email protected]
    │    │    ├── github.com/davecgh/[email protected]
    │    │    └── github.com/stretchr/[email protected]
    │    │         └── ...
    
    Login or Signup to reply.
  6. The dependency of a Go project is a directional graph. This graph consists of multiple layers, ranging from several to hundreds or thousands. Here is an dependency graph of redis. The cascaded tree can be difficult to understand due to the presence of many duplicated subtrees. To make the layout easier to view, the tree can be flattened via gomoddeps to fit the width of the screen.

    tzhang:~/github.com/redis/go-redis$ go mod graph | gomoddeps
    ├─ github.com/redis/go-redis/v9
    │  └─ dependencies
    │     ├─ github.com/bsm/ginkgo/[email protected]
    │     ├─ github.com/bsm/[email protected]
    │     ├─ github.com/cespare/xxhash/[email protected]
    │     ├─ github.com/davecgh/[email protected]
    │     ├─ github.com/dgryski/[email protected]
    │     ├─ github.com/pmezard/[email protected]
    │     ├─ github.com/stretchr/[email protected]
    │     └─ gopkg.in/[email protected]
    │  
    ├─ github.com/bsm/ginkgo/[email protected]
    │  └─ dependents
    │     └─ github.com/redis/go-redis/v9
    │  
    ├─ github.com/bsm/[email protected]
    │  └─ dependents
    │     └─ github.com/redis/go-redis/v9
    │  
    ├─ github.com/cespare/xxhash/[email protected]
    │  └─ dependents
    │     └─ github.com/redis/go-redis/v9
    │  
    ├─ github.com/davecgh/[email protected]
    │  └─ dependents
    │     ├─ github.com/redis/go-redis/v9
    │     ├─ github.com/stretchr/[email protected]
    │     ├─ github.com/stretchr/[email protected]
    │     └─ github.com/stretchr/[email protected]
    ...
    
    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search