Question 7770 in PHP Versions Question posted in
You can find a list of all versions here.

Why does PHPUnit in a GitHub workflow generate different test coverage XML report compared to local? – PHP Versions

I am running a code coverage workflow in GitHub action for this PHP package and it generates a different XML report than the one I get when I run the PHPUnit tests locally, resulting in lower coverage score.

Here is the workflow file:

name: Update codecov

on:
  push:
    branches: [ "master" ]
  pull_request:
    branches: [ "master" ]

permissions:
  contents: read

env:
  LANG: "sl_SI.utf8"

jobs:
  codecov:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v3
        with:
          ref: ${{ github.head_ref }}

      - name: Set up system locale
        run: |
          sudo apt-get install -y locales
          sudo locale-gen ${{ env.LANG }}

      - name: Validate composer.json and composer.lock
        run: composer validate --strict

      - name: Setup PHP
        uses: shivammathur/setup-php@v2
        with:
          php-version: 7.2
          extensions: xdebug, gettext

      - name: Install dependencies
        run: composer update --prefer-dist --no-progress --prefer-stable

      - name: Run test suite
        run: vendor/bin/phpunit

      - name: Upload to Codecov
        uses: codecov/codecov-action@v2
        with:
          files: ./build/coverage.xml
          verbose: true

Locally I get:

<?xml version="1.0" encoding="UTF-8"?>
<coverage generated="1673717717">
  <project timestamp="1673717717">
    <file name="/app/src/gettext-context.php">
      <line num="13" type="stmt" count="3"/>
      <line num="15" type="stmt" count="3"/>
      <line num="18" type="stmt" count="3"/>
      <line num="20" type="stmt" count="3"/>
      <line num="23" type="stmt" count="1"/>
      <line num="39" type="stmt" count="1"/>
      <line num="40" type="stmt" count="1"/>
      <line num="42" type="stmt" count="1"/>
      <line num="45" type="stmt" count="1"/>
      <line num="47" type="stmt" count="1"/>
      <line num="50" type="stmt" count="1"/>
      <line num="65" type="stmt" count="1"/>
      <line num="67" type="stmt" count="1"/>
      <line num="70" type="stmt" count="1"/>
      <line num="72" type="stmt" count="1"/>
      <line num="75" type="stmt" count="1"/>
      <line num="92" type="stmt" count="1"/>
      <line num="93" type="stmt" count="1"/>
      <line num="95" type="stmt" count="1"/>
      <line num="98" type="stmt" count="1"/>
      <line num="100" type="stmt" count="1"/>
      <line num="103" type="stmt" count="1"/>
      <metrics loc="105" ncloc="55" classes="0" methods="0" coveredmethods="0" conditionals="0" coveredconditionals="0" statements="22" coveredstatements="22" elements="22" coveredelements="22"/>
    </file>
    <metrics files="1" loc="105" ncloc="55" classes="0" methods="0" coveredmethods="0" conditionals="0" coveredconditionals="0" statements="22" coveredstatements="22" elements="22" coveredelements="22"/>
  </project>
</coverage>

However, the XML that is uploaded by the workflow to codecov.io is:

<?xml version="1.0" encoding="UTF-8"?>
<coverage generated="1673722112">
  <project timestamp="1673722112">
    <file name="/home/runner/work/gettext-context/gettext-context/src/gettext-context.php">
      <line num="3" type="stmt" count="0"/>
      <line num="13" type="stmt" count="3"/>
      <line num="15" type="stmt" count="3"/>
      <line num="18" type="stmt" count="3"/>
      <line num="20" type="stmt" count="3"/>
      <line num="23" type="stmt" count="1"/>
      <line num="27" type="stmt" count="0"/>
      <line num="39" type="stmt" count="1"/>
      <line num="40" type="stmt" count="1"/>
      <line num="42" type="stmt" count="1"/>
      <line num="45" type="stmt" count="1"/>
      <line num="47" type="stmt" count="1"/>
      <line num="50" type="stmt" count="1"/>
      <line num="54" type="stmt" count="0"/>
      <line num="65" type="stmt" count="1"/>
      <line num="67" type="stmt" count="1"/>
      <line num="70" type="stmt" count="1"/>
      <line num="72" type="stmt" count="1"/>
      <line num="75" type="stmt" count="1"/>
      <line num="79" type="stmt" count="0"/>
      <line num="92" type="stmt" count="1"/>
      <line num="93" type="stmt" count="1"/>
      <line num="95" type="stmt" count="1"/>
      <line num="98" type="stmt" count="1"/>
      <line num="100" type="stmt" count="1"/>
      <line num="103" type="stmt" count="1"/>
      <metrics loc="105" ncloc="55" classes="0" methods="0" coveredmethods="0" conditionals="0" coveredconditionals="0" statements="26" coveredstatements="22" elements="26" coveredelements="22"/>
    </file>
    <metrics files="1" loc="105" ncloc="55" classes="0" methods="0" coveredmethods="0" conditionals="0" coveredconditionals="0" statements="26" coveredstatements="22" elements="26" coveredelements="22"/>
  </project>
</coverage>

Since the second one contains lines with supposedly no coverage (e.g. <line num="3" type="stmt" count="0"/>), my codecov result is 86 % instead of 100 % like in local.

The lines in questions are if (function_exits('some_function')) statements seen in the source file. They simply assert that the function does not exist before declaring it.
Here’s how it looks in codecov.

I have zero ideas why the XML reports are different. Both environments are running the same PHP version and dev. dependencies. The phpunit.dist.xml file is the same for both cases and it’s being respected, since otherwise the test would fail, considering the bootstrap file is only defined in the phpunit.dist.xml file.

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    I'm not 100% understanding what's going on, but I fixed it. I did two things:

    1. Using the codecov official reference article for PHP that is available here, I updated my workflow to match their's and ended up with the following:
    name: Update codecov

on:
  push:
    branches: [ "master" ]
  pull_request:
    branches: [ "master" ]

permissions:
  contents: read

env:
  LANG: "sl_SI.utf8"

jobs:
  codecov:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v3
        with:
          ref: ${{ github.head_ref }}

      - name: Set up system locale
        run: |
          sudo apt-get install -y locales
          sudo locale-gen ${{ env.LANG }}

      - name: Install composer and dependencies
        uses: php-actions/composer@v6
        with:
          php_extensions: gettext

      - name: PHPUnit Tests
        uses: php-actions/phpunit@v3
        env:
          XDEBUG_MODE: coverage
        with:
          configuration: phpunit.xml.dist
          php_extensions: gettext xdebug
          args: --coverage-clover ./coverage.xml

      - name: Upload to Codecov
        uses: codecov/codecov-action@v2
        with:
          files: ./coverage.xml
          verbose: true
    1. I moved the all the recommended settings from php.ini to phpunit.xml(.dist), since obviously they are not being used by the GH action, unless in phpunit.xml.
        <php>
        <ini name="display_errors" value="On"/>
        <ini name="display_startup_errors" value="On"/>
        <!--
            PHPUnit recommended PHP config
            https://phpunit.readthedocs.io/en/9.5/installation.html#recommended-php-configuration
        -->
        <ini name="memory_limit" value="-1"/>
        <ini name="error_reporting" value="-1"/>
        <ini name="log_errors_max_len" value="0"/>
        <ini name="zend.assertions" value="1"/>
        <ini name="assert.exception" value="1"/>
        <ini name="xdebug.show_exception_trace" value="0"/>
    </php>

    This fixed it and the Clover XML file generated remotely perfectly matches the local (dev) one. I was therefore able to achieve 100 % coverage :)


  2. 0

    Thanks to your clear use-case (the library under test is a single file) and also the well written question, it can be answered how the code coverage differed:

    • If the functions have already been defined before the code-coverage is gathered and also from a different file (not that one under test), the actual lines of the functions are not covered.

    • If the functions are undefined, they will be covered by the code-coverage, not only their if clauses.

    Albeit this is clear to say in an answer, we also may tend to ask then why that was the case, however your question shows too little information so that doing so would tend to speculation.

    However from what I know the difference is not explained by using a different code-coverage mechanism, e.g. xdebug vs. pcov, which is another common reason for different coverage, however on a much more detailed level. It is also not xdebug 2 vs. xdebug 3, as you’re using PHP 7.2 which only has xdebug 2 (and btw. the PHP version conflicts to what you merchandise the package for: >= 7.4 – obviously that is not a requirement but fake [not a moral judgement]).

    I’d recommend you double check the requirements in an automated test so that you don’t blindly run the whole-testsuite only to find out in coverage later on, that the code you expected to be covered was not at all (while the test suite reported differently, tending to be fake as well [not a moral judgement again, just saying]). This could be done by asserting the return value of the require_once call in bootstrap.php already, then running phpunit with php assertions active & throwing (which is recommended for running PHP for tests in PHP 7.0+).

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search