Question 38520 in Ubuntu Question posted in
The official Ubuntu documentation can be found here.

GCP Compute Engine full disk not resized file system on public image – Ubuntu

GCP VM with 99,8% full disk has not resized its file system after increasing the disk capacity in google console.

I have a small VM on GCP based on public image ubuntu-2004-focal-v20220419.
There was a 10GB disk with root partition and file system.
Some logs occupied 99,8% of the disk capacity.
I increased the disk size to 15GB (without removing any data) and restarted the VM.
Documentation says:

For VMs with public images, Compute Engine automatically resizes the root partition and file system after you increase the size of the boot disk and restart the VM.

however, I can see that file system was not resized

~$ sudo lsblk
sda       8:0    0    15G  0 disk 
├─sda1    8:1    0   9.9G  0 part /
├─sda14   8:14   0     4M  0 part 
└─sda15   8:15   0   106M  0 part /boot/efi

I am wondering why is it so? I expected that additional 5GBs will be merged with sda1 automatically based on above documentation. Could it be, that the disk is so full, so some GCP background processes are not able to resize the file system?

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    I have posted my solution on Server Fault community. I hope this will be helpful for someone. https://serverfault.com/a/1119504/450511


  2. 0

    I gave this response in order to assist other communities with this problem, and @John Hanley responded with a suitable response in the comment area.

    Important notes:

    • Before you modify the file system on your disk, create a snapshot.
    • Boot disks use MBR partitions, which are limited to 2 TB in size. Do not resize boot disks beyond 2 TB.
    • I recommend shutting down the system before modifying storage.List item

    Step 1 – Create a default instance in the Google Console for testing.

    The default disk size is 10 GB.

    Step 2 – Connect to the instance.

    Go to the Google Console -> Compute Engine. A list of VM instances is displayed. On the line with your instance, click “SSH”. You can also use the CLI: “gcloud compute ssh <instance_name>”.

    Execute the “df” command and record the output.

    df > df_before_resize.txt

Filesystem     1K-blocks    Used Available Use% Mounted on
udev              292220       0    292220   0% /dev
tmpfs              60672    3184     57488   6% /run
/dev/sda1       10253588 1645864   8067156  17% /
tmpfs             303352       0    303352   0% /dev/shm
tmpfs               5120       0      5120   0% /run/lock
tmpfs             303352       0    303352   0% /sys/fs/cgroup

    Step 3 – Shutdown the instance.

    Step 4 – Resize the disk to 16 GB.

    Go to the Google Console -> Compute Engine -> Disks. Select the disk for the instance. Click the edit button and resize the disk to 16 GB.

    Step 5 – Start the instance and connect with SSH.

    Step 6 – Execute the “df” command again.

    Filesystem     1K-blocks    Used Available Use% Mounted on
udev              292220       0    292220   0% /dev
tmpfs              60672    3176     57496   6% /run
/dev/sda1       15414196 1659504  13004496  12% /
tmpfs             303352       0    303352   0% /dev/shm
tmpfs               5120       0      5120   0% /run/lock
tmpfs             303352       0    303352   0% /sys/fs/cgroup

    Comparing the line for /dev/sda1 confirms that Debian 9 Stretch on Google Compute does automatically resize the root file system. Now I wanted to know how.

    Step 7 – Get the serial port console output.

    gcloud compute instances get-serial-port-output --zone us-east4-c instance-1 > console.log

    Documentation for gcloud compute instances get-serial-port-output.

    Step 8 – Analyze the Console Output.

    This produced about 1,000 lines. To reduce this output I used grep to search for output that is probably related to resizing the file system. Notice how I spelled “resiz”. I want the search to include “resize” and “resizing”.

    grep -e resiz -e expand console.log

    This produced the following:

    Oct 28 [    3.979172] EXT4-fs (sda1): resizing filesystem from 2620928 to 3931643 blocks
Oct 28 00:11:57 localhost expand-root.sh[458]: Growing partition /dev/sda1
Oct 28 00:11:57 localhost expand-root.sh[458]: CHANGED: partition=1 start=4096 old: size=20967424 end=20971520 new: size=31453151,end=31457247
Oct 28 00:11:57 localhost expand-root.sh[458]: Resizing ext4 filesystem on /dev/sda1
Oct 28 00:11:57 localhost expand-root.sh[458]: resize2fs 1.43.4 (31-Jan-2017)
Oct 28 00:11:57 localhost kernel: [    3.979172] EXT4-fs (sda1): resizing filesystem from 2620928 to 3931643 blocks
[    4.577767] EXT4-fs (sda1): resized filesystem to 3931643
Oct 28 00:11:57 instance-1 kernel: [    4.577767] EXT4-fs (sda1): resized filesystem to 3931643
Oct 28 00:11:58 instance-1 expand-root.sh[458]: Filesystem at /dev/sda1 is mounted on /; on-line resizing required
Oct 28 00:11:58 instance-1 expand-root.sh[458]: old_desc_blocks = 2, new_desc_blocks = 2
Oct 28 00:11:58 instance-1 expand-root.sh[458]: The filesystem on /dev/sda1 is now 3931643 (4k) blocks long.

    Step 9 – Figure out what command is used to resize the root file system.

    From the console output I can see the command “expand-root.sh”. Using the command “which expand-root.sh”, I found this command located at “/usr/bin/expand-root.sh”.

    #!/bin/bash
# Expands a partition and filesystem using growpart and an appropriate
# filesystem tool for live filesystem expansion. Takes three arguments:
# DEVICE, such as "/dev/sda"
# PARTITION, such as "1"
# FILESYSTEM, such as "ext4"
 
DEVICE="${1}"
PARTITION="${2}"
FILESYSTEM="${3}"
 
if [[ -z "${DEVICE}" || -z "${PARTITION}" || -z "${FILESYSTEM}" ]]; then
  echo "Requires: $0 DEVICE PARTITION FILESYSTEM"
  exit 1
fi
 
# Grow partition using growpart
if [[ -x /usr/bin/growpart ]]; then
  echo "Growing partition ${DEVICE}${PARTITION}"
  /usr/bin/growpart "${DEVICE}" "${PARTITION}"
else
  echo "/usr/bin/growpart was not found"
  exit 1
fi
 
echo "Resizing ${FILESYSTEM} filesystem on ${DEVICE}${PARTITION}"
case "${FILESYSTEM}" in
  xfs)  xfs_growfs / ;;
  ext2) resize2fs "${DEVICE}${PARTITION}" ;;
  ext3) resize2fs "${DEVICE}${PARTITION}" ;;
  ext4) resize2fs "${DEVICE}${PARTITION}" ;;
  *) echo "Unsupported filesystem,  unable to expand size." ;;
esac

    Debian 9 Stretch running on Google Cloud Compute Engine does resize the root volume automatically if you resize the root disk. Very nice.

    Google Cloud offers the feature to resize the disk while the VM instance is running. Can you resize the root disk and the root filesystem while the system is running? If so, how?

    Step 1 – Go to the Google Console -> Compute Engine -> Disks. Select the disk for the instance.

    Step 2 – Click the “CREATE SNAPSHOT” button to create a recovery snapshot.

    Step 3 – Click the edit button and resize the disk to 25 GB.

    Step 4 – Connect to the instance.

    Step 5 – Execute this command to see the disk size and partition layout:

    fdisk -l

    This command produces the following output:

    Disk /dev/sda: 25 GiB, 26843545600 bytes, 52428800 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x70df3223
 
Device     Boot Start      End  Sectors Size Id Type
/dev/sda1  *     4096 31457246 31453151  15G 83 Linux

    Notice that the disk size is reported as “25 Gib” but the partition size is still “15G”. This means that the disk drive was resized to 25 GB but the root partition (filesystem) stayed the same size. Let’s try to resize the root filesystem while the system is running.

    expand-root.sh /dev/sda 1 ext4

Growing partition /dev/sda1
CHANGED: partition=1 start=4096 old: size=31453151 end=31457247 new: size=52424671,end=52428767
Resizing ext4 filesystem on /dev/sda1
resize2fs 1.43.4 (31-Jan-2017)
Filesystem at /dev/sda1 is mounted on /; on-line resizing required
old_desc_blocks = 2, new_desc_blocks = 4
The filesystem on /dev/sda1 is now 6553083 (4k) blocks long.

    Automating the Resize

    Now let’s take this even further and completely automate resizing a VM disk and resizing the root filesystem from the command line or a script.

    STEP 1 – SNAPSHOT DISK.

    First, we will snapshot the disk just in case.

    gcloud compute disks snapshot instance-1 --snapshot-names instance-1-snapshot --zone us-east4-c --description="This is a test snapshot"

    Documentation for gcloud compute disks snapshot.

    STEP 2 – RESIZE DISK.

    This command will resize the VM instance disk. Modify for the size you require, disk name and zone. Notice the –quiet flag. This prevents the prompt to confirm the resize.

    gcloud compute disks resize instance-1 --zone us-east4-c --size 30GB --quiet

    Documentation for gcloud compute disks resize.

    STEP 3 – RESIZE ROOT FILESYSTEM.

    This command will use SSH to connect to the instance and execute the expand-root.sh program remotely.

    gcloud compute ssh instance-1 --zone us-east4-c --command "sudo expand-root.sh /dev/sda 1 ext4"

    Documentation for gcloud compute ssh.

    STEP 4 – VERIFY RESIZE.

    A final “df” command on the on VM instance shows that the new root filesystem is 30 GB (30,896,016 blocks).

    Filesystem     1K-blocks    Used Available Use% Mounted on
udev              292220       0    292220   0% /dev
tmpfs              60672    3180     57492   6% /run
/dev/sda1       30896016 1663296  27853564   6% /
tmpfs             303352       0    303352   0% /dev/shm
tmpfs               5120       0      5120   0% /run/lock
tmpfs             303352       0    303352   0% /sys/fs/cgroup

    This is the complete blog-link to @John Hanley’s documentation.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search