There are some problems in your approach…

<li> height problem (and being display:inline by default)

First of all you set the height of the li elements.. despite they might have inline contents (spans) that won’t consume actual height so their container won’t grow accordingly.

You have to change that! Otherwhise you will have list items with a short fixed height, containing further lists inside that will overlap.

To make sure such problem won’t occur, just style the <li> elements with display: inline-block and don’t set any height. If you want to control the height of the label of each list item, just style the label not its parent <li>,

Styling the label for each <li>

But more importantly you want to style the li, but the side effect will be affecting its all content. If you instead wished to style only the name listed next to the list item, you should target that element instead in your css selectors. That way you’ll make sure to "highlight" only that specific label instead of the whole container holding deeper trees.

Use specific selectors for better parent > children relationships

Plus since the parent relationship described by ul li will match also with any tree involving stuff inbetween you want to make sure your selector match one category only, by using the direct child operator > by doing .filestab ul > li > span.

https://developer.mozilla.org/en-US/docs/Web/CSS/:first-child

The :first-child CSS pseudo-class represents the first element among a
group of sibling elements.

In the end to estabilish a very strict condition for your label to be determined given a li I would suggest it being the very first child instead of any children by doing .filestab ul > li > span:first-child.

Using :has to specifically style li > span with anchors

Then following the same route, to discriminate when such label contains an anchor you may just use the :has pseudo-class .filestab ul > li > span:first-child:has(a).

https://developer.mozilla.org/en-US/docs/Web/CSS/:has

The functional :has() CSS pseudo-class represents an element if any of
the relative selectors that are passed as an argument match at least
one element when anchored against this element.

Here follow 2 demos where all the problems described above were addressed, and two distinct strategies were used…

Demo 1 – Using your own html styled with :has

Here I highly simplified your demo.. the only side effect is not having one of your <li> engaged by the style, because its span is not the first child.

To solve that scenario, you’ll need to stick with the decided convention of having the first span as your label, and just use it as a container for all your content that in that case will include a picture before the text.

I highlighted that point in this demo by doing a comparison between Example broken and Example solved:

.filestab ul > li {
}

.filestab ul > li > span:first-child {
  cursor: pointer;
  /*this will make the container li grow with the span size*/
  display: inline-block;
  padding: .5em 1em;
  background: lightgray;
}

.filestab ul > li > span:first-child:hover  {
  background: darkgray;
  color: #fff;
}

.filestab ul > li > span:first-child:has(a) {
  background: lightgreen;
}

.filestab ul > li:hover > span:first-child:has(a):hover{
  background: darkgreen;
  color: white;
}

a:visited{
  color: white;
}

<div class="filestab">
  <button class="buttonfilesubmenu"><u>C</u>lick Me!</button>
  <ul class="filesubmenu">
    <li>
      <span>I'm an example submenu!</span>
    </li>
    <li>
      <span>Example</span>
    </li>
    <li>
      <span>
        <a href="">Example with link</a>
      </span>
    </li>
    <li>
      <span>Example</span>
      <button class="filepopout1">
        <img
          class="invert-image5"
          src="https://file.garden/ZWlUCY4S7Xz2vypS/themes/windows%2095%20paint/right%20arrow.png">
      </button>
      <ul class="filepop1">
        <li>
          <span>
            <a href="">Example with link</a>
          </span>
        </li>
        <li>
          <span>
            <a href="#">Example with link</a>
          </span>
        </li>
        <li>
          <span>Example</span>
          <button class="filepopout2">
            <img
              class="invert-image6"
              src="https://file.garden/ZWlUCY4S7Xz2vypS/themes/windows%2095%20paint/right%20arrow.png">           </button>
          <ul class="filepop2">
            <li>
              <span><a href="">Example with link</a></span>
            </li>
            <li>
              <span><a href="#">Example with link</a></span>
            </li>
            <li>
              <span>Example</span>
            </li>
            <li>
              <img src="https://file.garden/ZWlUCY4S7Xz2vypS/themes/windows%2095%20paint/paint%2016x16.gif">
              <span>Example broken</span>
            </li>
            <li>
              <span>
                <img src="https://file.garden/ZWlUCY4S7Xz2vypS/themes/windows%2095%20paint/paint%2016x16.gif">
                <span>Example solved</span>
              </span>
            </li>
          </ul>
        </li>
        <li>
          <span>Example</span>
        </li>
      </ul>
    </li>
    <li>
      <span>Example</span>
    </li>
    <li>
      <span>Example</span>
    </li>
    <li>
      <span>Example</span>
    </li>
    <li>
      <span>Example</span>
    </li>
  </ul>
</div>
<!-- FILESTAB -->

Demo 2 – Using custom class

Anyway :has was a recent addition in the browsers space and some older versions will be cut off -> https://caniuse.com/?search=%3Ahas

If you need a safer approach, you can’t rely on css to style the <span> based on a condition over its content. But yet you can just define a distinct css class needed to style the labels bound to list items having an anchor

Here I used such approach by having a special class itemAnchor that when present will style the label differently then the general rule applied for list items label.

Such strategy requires to add the class to the desider labels inside the html itself upfront! But yet if relying on Javascript is not a problem, this will be enough:

/**
Given a ul element, sets the anchorItem css class
to all the li > span:first-child having an anchor belong their children
*/
function setClassForItemAnchors(tree){
  tree.querySelectorAll('li > span:first-child')
    .forEach(el=>{ if(el.querySelector('a')) el.classList.add('anchorItem') });
}

//when document is ready...
document.addEventListener('DOMContentLoaded', ()=>{  
  //call the function passing the main #tree
  setClassForItemAnchors(document.getElementById('tree'));
})

In the example I both used the class manually in the html anchorItemCustom (just a placeholder not having any css styling rule) and also included the javascript that would add it anchorItem automatically when the page is loaded:

/**
Given a ul element, sets the anchorItem css class
to all the li > span:first-child having an anchor belong their children
*/
function setClassForItemAnchors(tree){
  tree.querySelectorAll('li > span:first-child')
    .forEach(el=>{ if(el.querySelector('a')) el.classList.add('anchorItem') });
}

//when document is ready...
document.addEventListener('DOMContentLoaded', ()=>{  
  //call the function passing the main #tree
  setClassForItemAnchors(document.getElementById('tree'));
})

li > span:first-child{
  background: lightgreen;
  
  cursor: pointer;
  display: inline-block;
  padding: .4em;  
  margin-bottom: .5em;
}

li > span:first-child:hover{
  background: darkgreen;
}

.anchorItem{
  background: pink !important;
}

.anchorItem:hover{
  background: red !important;
}

<ul id="tree">
  <li>
    <span class="anchorItemCustom"><a href="">Item (parent) as anchor</a></span>
    <ul>
      <li>
        <span>Item</span>
      </li>
      <li>
        <span>Item</span>
        <ul>
          <li>
            <span>Item</span>
          </li>
          <li>
             <span class="anchorItemCustom"><a href="">Item (children) as anchor</a></span>
          </li>
          <li>
            <span>Item</span>
          </li>
        </ul>
      </li>
      <li>
        <span>Item</span>
      </li>
      <li>
        <span>Item</span>
      </li>
    </ul>
  </li>
</ul>