Upgrading Qubes to Fedora 24

I like Fedora, having worked with it since Fedora Core 1 and, even before then, with RedHat all the way back to v4.2, and with several Unix distributions going back to 1980 when I was playing around with the Xerox Alto and Star. Today I run Fedora 25 (F-25) on one of my systems. When I started investigating Qubes in 2016, my first clue this was going to be good was that their foundation was Fedora.

When initially installed, my Qubes v3.1 used Fedora 23, one version behind Fedora’s current version at that time. Fedora releases a new version about every six months with F-26 expected to come out June 2017. When F-25 came out in November 2016, I upgraded my Fedora system right away. Qubes had released v3.2 at the end of September 2016 still using F-23, which would potentially put Qubes two versions behind soon. Fedora typically obsoletes — no more updating — the version two behind its current version about one month after that current version releases. That meant F-23 would be unsupported about Jan 2017. But, in November 2016, Qubes announced its release of their F-24 template.

I been busy. Finally got around to downloading and installing the Qubes F-24 template. But, some quirks came up. Here’s what I did and how it went, including quirks.

Step 0: Backup!

Some work is done on dom0, the primary domain from which all the VMs derive. Before starting, better have a full backup. Depending on how complex your installation is, the backup could take a while. Start it and walk away.

When the backup is finished, close all Fedora 23 AppVMs except network & firewall.

Check available disk space:

[...@dom0 ~]$ df -h .
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/qubes_dom0-root  909G  115G  749G  14% /

I’ve got a terabyte drive on my Qubes system with plenty of space available. This will be important later.

Step 1: Duplicate

Clone the Fedora 23 template into Fedora 24 to prep the upgrade.

[...@dom0 ~]$ time qvm-clone fedora-23 fedora-24
--> Creating directory: /var/lib/qubes/vm-templates/fedora-24
--> Copying the private image:
/var/lib/qubes/vm-templates/fedora-23/private.img ==>
/var/lib/qubes/vm-templates/fedora-24/private.img
--> Copying the root image:
/var/lib/qubes/vm-templates/fedora-23/root.img ==>
/var/lib/qubes/vm-templates/fedora-24/root.img
--> Creating icon symlink: /var/lib/qubes/vm-templates/fedora-24/icon.png -> /usr/share/qubes/icons/template.png
--> Copying the template's appmenus templates dir:
/var/lib/qubes/vm-templates/fedora-23/apps.templates ==>
/var/lib/qubes/vm-templates/fedora-24/apps.templates
--> Copying the template's appmenus template icons dir:
/var/lib/qubes/vm-templates/fedora-23/apps.tempicons ==>
/var/lib/qubes/vm-templates/fedora-24/apps.tempicons
--> Copying whitelisted apps list: whitelisted-appmenus.list
--> Copying whitelisted apps list: vm-whitelisted-appmenus.list
--> Copying whitelisted apps list: netvm-whitelisted-appmenus.list
--> Converting Appmenu Templates...
--> Adding Apps to the Menu...
--> Commiting template updates... COW: /var/lib/qubes/vm-templates/fedora-24/root-cow.img...

real    3m37.285s
user    0m4.193s
sys     0m40.322s

Step 2: Prepare for Low Disk Space

If available disk space is low it might cause a complaint during installation. Create a 5G temporary cache image:

[...@dom0 ~]$ truncate -s 5GB /var/tmp/template-upgrade-cache.img

The truncate(1) command creates, in this case, a 5,000,000,000 byte file prepared for use as a virtual drive. (If you want powers of 1,024, use 5G instead of 5GB. *sigh*) The file is preallocated for the requested size (-s).

Creating and using it is advised on the Qubes upgrade site only if the upgrade shows no errors. However, the upgrade doesn’t depend on your drive’s available disk space but on the total space available in the newly cloned template for the dnf(8) download and installation. I advise to create this cache image file anyway. My experience leading to this recommendation is documented below.

Step 3: Launch gnome-terminal

This step starts the newly cloned F-24 template and launches the terminal cloned from the F-23 template. Remember that F-24 isn’t on the system yet.

[...@dom0 ~]$ time qvm-run -a fedora-24 gnome-terminal
Running command on VM: 'fedora-24'...
Starting the VM 'fedora-24'...
--> Creating volatile image: /var/lib/qubes/vm-templates/fedora-24/volatile.img...
--> Loading the VM (type = TemplateVM)...
--> Starting Qubes DB...
--> Setting Qubes DB info for the VM...
--> Updating firewall rules...
--> Starting the VM...
--> Starting the qrexec daemon...
Waiting for VM's qrexec agent..................connected
--> Starting Qubes GUId...
Connecting to VM's GUI agent: .connected
--> Sending monitor layout...
--> Waiting for qubes-session...
 
real    0m30.956s
user    0m0.182s
sys     0m0.301s

Step 4: Attach the Low Disk Space Cache Image

With the template clone launched, now is the time to use the temporary cache image previously created. This instruction works with block devices. It attaches a file as a block device to the given domain (-A domainname). Without this attachment, the cloned VM will not have enough available space for dnf to write its files. My typical VM has only 2G of space, but you can grow it in the VM Settings (gear with a blue triangle icon). The dnf download of F-24 is just over 2G, and that doesn’t count any extra space you may need. I made — and corrected — this mistake, as documented below. Don’t make my mistake. Attach this cache image:

[...@dom0 ~]$ qvm-block -A fedora-24 dom0:/var/tmp/template-upgrade-cache.img

Step 5: Mount the Cache Image

From this step, most of the work is done on the new F-24 template cloned from F-23. First, prepare /dev/xvdi, which has the attached cache image:

[user@fedora-24 ~]$ sudo mkfs.ext4 /dev/xvdi

The truncate command that preallocated the file actually created a sparse file. Thus, the total space allocated is not used until actual data gets that far in the file. Using mkfs.ext4(8) turns it into an ext4(5) filesystem, which is the current default filesystem used for Linux.

[user@fedora-24 ~]$ sudo mount /dev/xvdi /mnt/removable

This mounts the drive into the /mnt/removable subdirectory, which is typically present, but empty, in all VMs. This mountpoint is a handy place for special case mounts, like this, but be sure it isn’t already in use by something else. If any data is in the /mnt/removable subdirectory when nothing is mounted on it, that data will become hidden by the mount. Don’t fear. Any data hidden by a mount becomes revealed when the mountpoint is unmounted.

Step 6: Clear dnf Cache

Before upgrading with dnf, clear its cache:

[user@fedora-24 ~]$ time sudo dnf clean all
81 files removed

real    0m0.586s
user    0m0.301s
sys     0m0.103s

This eliminates the packages, cache, and metadata leftover from previous F-23 updates and reclaims space.

Step 7: Upgrade to Fedora 24

Finally. Run the upgrade using dnf. The recommendation to add the option, --setopt, for the cache is, in my opinion, a mandate unless you already have at least 10G of available space — not total space — in your F-24 template cloned from the F-23 template. To me, that means your total size as reported by df -h should be at least 20G to consider skipping the use of --setopt. I will review my pain about this below.

[user@fedora-24 ~]$ time sudo dnf --releasever=24 --setopt=cachedir=/mnt/removable distro-sync

This upgrade output is huge. Not important to list it all here because it shows the typical dnf repo checks and the downloading for the F-24 distro. It also brings on F-24 variations from other repos, including Qubes and, in my case, RPMFusion repos.

Here is the transaction summary for my installation:

Transaction Summary
===========================================================================
Install 111 Packages
Upgrade 2169 Packages

Total download size: 2.1 G

As expected, the download is large. Expect to wait a while.

Why You Should Use --setopt, AKA, Silly me: Do What I Learned

Not realizing from the Qubes advice exactly how much free space should be available, I did not initially use the --setopt option to cache into a 5G temporary mountpoint. After waiting over half an hour (33m51.965s), with 2,227 of 2,280 files downloaded successfully, I got a disk space error. The downloaded packages were saved, as usual, so I thought it would be better to expand the F-24 template instead of use the temporary cache.

My F-23 template had a private storage max of 2G and a system storage max of 10G. Therefore, the F-24 template cloned from it got the same. The storage max of 10G is what the system allows for expansion, but df showed only 2G. With 2.1G to download and no expansion, it missed.

Returning to the dom0 terminal, I ran truncate (step #2). This created the image file with 4.7G allocated (ls -oh) but holding 5,000,000,000 byte capacity (ls -o).

[...@dom0 ~]$ ls -oh /var/tmp/template-upgrade-cache.img
-rw-rw-r-- 1 lreznick 4.7G Mar 17 15:51 /var/tmp/template-upgrade-cache.img

[...@dom0 ~]$ ls -o /var/tmp/template-upgrade-cache.img
-rw-rw-r-- 1 lreznick 5000000000 Mar 17 15:51 /var/tmp/template-upgrade-cache.img

[...@dom0 ~]$ ls -sh /var/tmp/template-upgrade-cache.img
0 /var/tmp/template-upgrade-cache.img

Truncate uses SI units (1000-based) with the “GB” notation. Using “G” notation would give IEC (1024-based) unit allocation. As a sparse file, it takes no data space yet (ls -sh). The system knows about the allocated space, but the space doesn’t actually exist on the disk until used. All ls options, except the -s option, read the directory to see the space reserved, while -s looks at the blocks actually used.

Next, I ran qvm-block (step #4) on dom0. That took about 2 seconds. Returning to the F-24 template, I checked that the /dev/xvdi device was present:

[user@fedora-24 ~]$ ls -oh /dev/xvdi
brw-rw---- 1 root 202, 128 Mar 17 16:21 /dev/xvdi

I ran the mkfs.ext4 and mount commands (step #5):

[user@fedora-24 ~]$ time sudo mkfs.ext4 /dev/xvdi
mke2fs 1.42.13 (17-May-2015)
Discarding device blocks: done 
Creating filesystem with 1220703 4k blocks and 305216 inodes
Filesystem UUID: 2da64c58-98c0-4973-8db7-3af64fb29e90
Superblock backups stored on blocks: 
    32768, 98304, 163840, 229376, 294912, 819200, 884736

Allocating group tables: done 
Writing inode tables: done 
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done


real    0m2.206s
user    0m0.065s
sys     0m0.379s

[user@fedora-24 ~]$ sudo mount /dev/xvdi /mnt/removable/

[user@fedora-24 ~]$ df -h /dev/xvdi
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvdi       4.5G  9.4M  4.3G   1% /mnt/removable

With extra space ready for a cache directory, I reran dnf clean all because the on-disk cache is no longer the same as the /mnt/removable cache directory, so it won’t see the files already downloaded.

[user@fedora-24 ~]$ time sudo dnf clean all
2289 files removed

real    0m3.697s
user    0m1.193s
sys     0m0.561s

Then, reran dnf using --setopt:

[user@fedora-24 ~]$ time sudo dnf --releasever=24 --setopt=cachedir=/mnt/removable distro-sync

After another 31 minutes downloading the F-24 packages, it quit again, complaining about various packages needing over 400-some-odd MB each. The final summary when it quit with errors was:

Error Summary
-------------
Disk Requirements:
At least 512MB more space needed on the / filesystem.

According to df, my F-24 template root file system had only 1.7G available after dnf quit. Whatever was in use during the run was no longer in use after dnf quit. The template needed more space. It was allowed to grow to 10G at the time. I decided to double its allowance. To do this, I had to shutdown the F-24 template, increase the system allowed max, and try again.

Trying to change the system max allowance in the Qubes VM Manager’s VM Settings (gear icon with a blue triangle) from 10,240 MB to 20,480 MB, I got an error message:

VM start required to complete the operation, but not allowed. Either run the operation again allowing VM start this time, or run resize2fs in the VM manually.

This was a surprise. In Qubes v3.1 I was able to grow the space using the VM Manager. Qubes v3.2 wasn’t allowing it to grow. My 20,480 MB setting remained in the VM Settings, but the disk size report (df -h) still showed /dev/mapper/dmroot, the VM’s root (/) directory, to be 9.8G, correlating to the previously set 10,240 MB limit. So, while the settings max in the VM settings changed, the actual space allocated did not. Bug report #1875 shows this, but the last response was a year ago. Able to change these settings before using the VM manager, something else is wrong here.

Before manually resetting the size using resize2fs(8), I restarted the F-24 template’s gnome-terminal, reattached the template-upgrade-cache.img, remounted the /dev/xvdi to /mnt/removable, verified /mnt/removable was mounted, and reran dnf. It showed all the downloaded packages, checked itself internally, asked to continue. Approved its continue and eventually it gave the same error with a different request amount for more space.

BTW, I love the what the author of resize2fs put in its man page:

Note: when kilobytes is used above, I mean real, power-of-2 kilobytes, (i.e., 1024 bytes), which some politically correct folks insist should be the stupid-sounding “kibibytes”. The same holds true for megabytes, also sometimes known as “mebibytes”, or gigabytes, as the amazingly silly “gibibytes”. Makes you want to gibber, doesn’t it?

Before running resize2fs, I’d have to shut down the fedora-24 VM, check whether any loop devices still pointed to it, and manually resize the root.img file for it. This is what the loop devices attached to the F-24 VM look like before shutting the VM down:

[...@dom0 ~]$ sudo losetup -a
[...]
/dev/loop7: [64257]:54004896 (/var/lib/qubes/vm-templates/fedora-24/root.img)
/dev/loop8: [64257]:54004893 (/var/lib/qubes/vm-templates/fedora-24/root-cow.img)
/dev/loop9: [64257]:54004895 (/var/lib/qubes/vm-templates/fedora-24/private.img)
/dev/loop10: [64257]:54005173 (/var/lib/qubes/vm-templates/fedora-24/volatile.img)
[...]

[...@dom0 ~]$ ls -oh /var/lib/qubes/vm-templates/fedora-24/root.img
-rw-rw---- 1 lreznick 20G Mar 18 09:10 /var/lib/qubes/vm-templates/fedora-24/root.img

[...@dom0 ~]$ ls -sh /var/lib/qubes/vm-templates/fedora-24/root.img
9.9G /var/lib/qubes/vm-templates/fedora-24/root.img

The first ls shows that resizing done by changing in the VM Manager did change the size of the root.img sparse file. That’s why the VM Settings showed the changed size. That part of it worked. But the second ls shows that root.img used 9.9G of actual disk space. In other words, the sparse file allocation was at the desired 20G, but the actual usage was still at the 9.9G.

If the sparse file was ready for storage expansion into 20G, why did it not expand?

Restarted the fedora-24 VM to see if the increase would now happen, despite the error message. It checked the repositories for any new info, asked to download, and started the download again. Resizing the F-24 template must have flushed the download cache.

As usual, this took nearly half an hour and failed after 2062 of 2280 files were downloaded, this time with curl(1) errors:

[MIRROR] perl-Socket-2.024-1.fc24.x86_64.rpm: Curl error (23): Failed writing received data to disk/application for http://linux.mirrors.es.net/fedora/updates/24/x86_64/p/perl-Socket-2.024-1.fc24.x86_64.rpm [Failed writing body (1075 != 1452)]
[FAILED] perl-Socket-2.024-1.fc24.x86_64.rpm: Curl error (23): Failed writing received data to disk/application for http://linux.mirrors.es.net/fedora/updates/24/x86_64/p/perl-Socket-2.024-1.fc24.x86_64.rpm [Failed writing body (1075 != 1452)]
(2064-2065/2280): pe 82% [================- ] 1.3 MB/s | 1.7 GB 04:37 ETA
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: Error downloading packages:
Curl error (23): Failed writing received data to disk/application for http://linux.mirrors.es.net/fedora/updates/24/x86_64/p/perl-Socket-2.024-1.fc24.x86_64.rpm [Failed writing body (1075 != 1452)]

[user@fedora-24 ~]$ df -h /
Filesystem          Size  Used Avail Use% Mounted on
/dev/mapper/dmroot  9.8G  9.7G     0 100% /

The space didn’t expand, so ran resize2fs in the F-24 template.

[user@fedora-24 ~]$ sudo resize2fs -p /dev/mapper/dmroot 
resize2fs 1.42.13 (17-May-2015)
Filesystem at /dev/mapper/dmroot is mounted on /; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 2
The filesystem on /dev/mapper/dmroot is now 5242880 (4k) blocks long.

[user@fedora-24 ~]$ df -h /
Filesystem          Size  Used Avail Use% Mounted on
/dev/mapper/dmroot   20G  9.7G  9.0G  52% /

That made the size real. Reran dnf, which should pick up from its cache. This time dnf got all the way through the download and started transaction handling. My installation consisted of 4,467 file actions. It will take some time. Don’t just get a coffee. You may want a meal. And a good book.

On my system, the job finished after three hours.

Step 8: Shutdown the Updated Template

Shutdown the F-24 template, either using the Qubes VM Manager or:

[...@dom0 ~]$ qvm-shutdown fedora-24

Step 9: Remove the Cache Image

With the new fedora-24 template updated and shutdown, no need to umount the cache image because shutting down unmounts it. Just get rid of it:

[...@dom0 ~]$ rm /var/tmp/template-upgrade-cache.img

Step 10: Shrink VM’s Unused Space

Because files removed from a VM do not shrink the VM’s disk usage, run the following command to shrink the template:

[...@dom0 ~]$ time qvm-trim-template fedora-24
Disk usage before:
14356540 /var/lib/qubes/vm-templates/fedora-24/root.img
10462984 /var/lib/qubes/vm-templates/fedora-24/root-cow.img.old
24819524 total
Creating temporary VM...
--> Creating volatile image: /var/lib/qubes/appvms/trim-fedora-24/volatile.img...
--> Loading the VM (type = AppVM)...
--> Starting Qubes DB...
--> Setting Qubes DB info for the VM...
--> Updating firewall rules...
--> Starting the VM...
--> Starting the qrexec daemon...
Waiting for VM's qrexec agent.....................connected
Performing fstrim now...
fstrim done, cleaning up...
Disk usage after:
9321296 /var/lib/qubes/vm-templates/fedora-24/root.img

real    0m42.692s
user    0m0.202s
sys     0m1.519s

Usage numbers are in K (1,024) units, so to turn the root.img usage report into G, divide the before (14356540) and the after (9321296) by 1024^2. Thus, 13.7G was shrunk to 8.9G.

Step 11: Set the Template to Default

Using the Qubes VM Manager, select the Global Settings icon (looks like a blue ball on a gear). Make the new fedora-24 the default template and click the OK button. You don’t have to do this if you prefer some other template to be the default.

Step 12: Shutdown All Fedora-23 VMs

At this stage, every VM based on the now-defunct fedora-23 template must be shutdown before converting them to the new fedora-24 template. By now, the only ones running should be sys-net and sys-firewall. However, sys-whonix uses sys-net and sys-firewall, so even though it’s not based on the Fedora template, shut it down, too. Shut them all down:

[...@dom0 ~]$ qvm-shutdown --all

Note: A message, “libvertError: Requested operation is not valid: Domain is not running” may pop up, drawing attention to a probable bug in the Qubes Manager. Just click the OK button.

Step 13: Change All Fedora-23 VMs to Fedora-24

One at a time, for each VM based on fedora-23, select it and click its VM Settings icon (the gear with a blue triangle on the icon). Change its template setting to fedora-24 and click the OK button.

Step 14: Change DispVM to Fedora-24

Disposable VMs (DispVMs) require running a command in dom0 to set their template VM. If you set fedora-24 as the default template, use the --default-template option. Otherwise use the fedora-24 argument unless you prefer some other template.

[...@dom0 ~]$ time qvm-create-default-dvm --default-template
--> Using TemplateVM: fedora-24
--> Creating directory: /var/lib/qubes/appvms/fedora-24-dvm
--> Copying the template's private image: /var/lib/qubes/vm-templates/fedora-24/private.img
--> Creating volatile image: /var/lib/qubes/appvms/fedora-24-dvm/volatile.img...
--> Creating icon symlink: /var/lib/qubes/appvms/fedora-24-dvm/icon.png ->
/usr/share/icons/hicolor/128x128/devices/appvm-gray.png
--> Creating default whitelisted apps list:
/var/lib/qubes/appvms/fedora-24-dvm/whitelisted-appmenus.list
--> Starting NetVM sys-firewall...
--> Starting NetVM sys-net...
--> Creating volatile image: /var/lib/qubes/servicevms/sys-net/volatile.img...
--> Loading the VM (type = NetVM)...
--> Starting Qubes DB...
--> Setting Qubes DB info for the VM...
--> Updating firewall rules...
--> Starting the VM...
--> Starting the qrexec daemon...
Waiting for VM's qrexec agent.........connected
--> Starting Qubes GUId...
Connecting to VM's GUI agent: .connected
--> Sending monitor layout...
--> Waiting for qubes-session...
--> Creating volatile image: /var/lib/qubes/servicevms/sys-firewall/volatile.img...
--> Loading the VM (type = ProxyVM)...
--> Starting Qubes DB...
--> Setting Qubes DB info for the VM...
--> Updating firewall rules...
--> Starting the VM...
--> Starting the qrexec daemon...
Waiting for VM's qrexec agent........connected
--> Starting Qubes GUId...
Connecting to VM's GUI agent: .connected
--> Sending monitor layout...
--> Waiting for qubes-session...
--> Creating volatile image: /var/lib/qubes/appvms/fedora-24-dvm/volatile.img...
--> Loading the VM (type = AppVM)...
--> Starting Qubes DB...
--> Setting Qubes DB info for the VM...
--> Updating firewall rules...
--> Starting the VM...
--> Starting Qubes GUId...
Connecting to VM's GUI agent: .......connected
Waiting for DVM fedora-24-dvm ...
/qubes-used-mem
Disk detached successfully

DVM boot complete, memory used=293368. Saving image...

Domain fedora-24-dvm saved to /var/lib/qubes/appvms/fedora-24-dvm/dvm-savefile

DVM savefile created successfully.

real    1m18.676s
user    0m1.130s
sys     0m1.847s

Step 15: Remove the Fedora-23 Template

Unless you want to keep the fedora-23 template — remember that it was cloned to create fedora-24, so all your files that were in 23 are now in 24 — it’s time to remove the fedora-23 template from the system.

[...@dom0 ~]$ sudo dnf remove qubes-template-fedora-23

BEWARE: Be sure to type 23!

Step 16: Remove Fedora-23 From the Application Start Menu

At this stage, the F-23 template is gone, yet it’s still in your Start Menu. To remove it:

[...@dom0 ~]$ ls -oh ~/.local/share/applications/fedora-23-*
-rw------- 1 lreznick 389 Mar 17 11:06 /home/lreznick/.local/share/applications/fedora-23-fontmatrix.desktop
-rw------- 1 lreznick 341 Mar 17 11:06 /home/lreznick/.local/share/applications/fedora-23-gnome-control-center.desktop
-rw------- 1 lreznick 385 Mar 17 11:06 /home/lreznick/.local/share/applications/fedora-23-gnome-gkrellm.desktop
-rw------- 1 lreznick 390 Mar 17 11:06 /home/lreznick/.local/share/applications/fedora-23-gpk-update-viewer.desktop
-rw------- 1 lreznick 408 Mar 17 11:06 /home/lreznick/.local/share/applications/fedora-23-org.gnome.Software.desktop
-rw------- 1 lreznick 383 Mar 17 11:06 /home/lreznick/.local/share/applications/fedora-23-org.gnome.Terminal.desktop
-rw------- 1 lreznick 265 Mar 17 11:06 /home/lreznick/.local/share/applications/fedora-23-qubes-appmenu-select.desktop
-rw------- 1 lreznick 399 Oct 13 09:04 /home/lreznick/.local/share/applications/fedora-23-redshift-gtk.desktop
-rw------- 1 lreznick 405 Mar 17 11:06 /home/lreznick/.local/share/applications/fedora-23-system-config-printer.desktop

[...@dom0 ~]$ sudo rm ~/.local/share/applications/fedora-23-*

Step 17: Restart

All done! Either restart your VMs, or reboot your system. The F-24 template may want an upgrade if something came into the repo since your installation.

Leave a Comment