Duplicating Disks

In a previous article, I mentioned using dd(1) to turn a CD/DVD image into an ISO file. There are several reasons to create and use ISO images:

  1. The CD/DVD is getting old and making the ISO may be your last chance to get the data from it.
  2. A memory stick has a bunch of files on it, but it is old, maybe showing some file errors, and you can’t trust it any more.
  3. You downloaded an ISO image and want to put it on a stick or drive.
  4. You have an ISO file and want to get the files in it without putting it on a stick or drive.

Note: Here’s the tl;dr:

Take an ISO from a drive:

$ sudo dd if=DEVICENAME of=PATHNAME.iso

Put an ISO on a drive:

$ sudo dd if=PATHNAME.iso of=DEVICENAME

The DEVICENAME is the /dev/whatever name for the mounted drive you want to use, as df(1) shows in its left column. The PATHNAME refers to the full path of the ISO file you want to use.

Finding the Device

Because this process copies to or from a drive, whether an actual drive or a USB stick or SD card treated like a drive, devices are isolated from regular user access. Typically requiring privileged access, the sudo(8) command offering root privileges prefixes use of dd when accessing devices. When regular Linux system users refer to their files, they have ownership permissions to use those files. Permissions look like this:

[user@fedora-23-dvm ~]$ ls -oh
total 280K
drwxr-xr-x 2 user 4.0K Mar 13 07:49 Desktop
drwxr-xr-x 2 user 4.0K Mar 13 07:49 Documents
drwxr-xr-x 2 user 4.0K Mar 13 07:49 Downloads
drwxr-xr-x 2 user 4.0K Mar 13 07:49 Music
drwxr-xr-x 2 user 4.0K Mar 13 07:49 Pictures
drwxr-xr-x 2 user 4.0K Mar 13 07:49 Public
-rw-rw-r-- 1 user  79K Oct 13 11:38 rpmlist-qubesv31.txt
-rw-rw-r-- 1 user  32K Oct 16 07:19 rpmlist-qubesv32.txt
-rw-rw-r-- 1 user 135K Nov 25 12:17 rpmlist-sagerF24.txt
drwxr-xr-x 2 user 4.0K Mar 13 07:49 Templates
drwxr-xr-x 2 user 4.0K Mar 13 07:49 Videos

Above is a list of the current contents of a Disposable VM (DispVM) in Qubes. The VM is derived from Fedora 23. My user name is “user” and I’m logged in to my home directory (~). I put some rpmlist files into it for comparing long file listings to long directory listings. The directories are all empty.

Using the ls -oh option shows a variation on the long listing (-lh) that only gives the owner’s name, not the group membership. Most of the time, nobody cares about the group membership of a file, especially when the only user on the system is you. The important column is the one on the left composed of 10 characters. Reading from left to right:

  1. This character defines a “d” for a directory or a “-” for a regular file.
  2. The next three characters define the owner’s (user’s) permissions. These three show an “r” for read permission, then a “w” for write permission, and an “x” for execute permission. A “-” appears when the relevant permission is not given. Owners typically have “r” and “w” for their own files while “x” is only found on program files that can run and on directories that the owner can change into.
  3. The next three characters define the group’s permissions. Any member of the group able to use the file get these permissions. Group names are shown with a “-l” or “-g” option, not with the “-o” option, but they’re more often used on multiuser server systems. Notice that the group typically does not get write permission for directories, meaning other members of the group cannot add or remove files from such directories, both operations requiring the ability to write to the directory.
  4. The last three characters define everybody else’s (other’s) permissions. These are the permissions that all other users have, except the root user, who bypasses permissions. All others may have read-only permissions for files, if that, but directories allow reading (ls) and executing (cd), but not writing (create and destroy entries).

Other characters in these positions identify special files or permission facilities. On Linux, devices appear as files. Keep that in mind for just about everything on Unix and Linux: Everything is a file. This was a major insight in the original design of Unix that Linux and other Unix work-alikes inherited. When looking at the listing of device files in the /dev directory, you’re seeing a special view.

Plug a USB stick into the system and get a listing from fdisk(1):

$ sudo fdisk -l
[...]
Disk /dev/xvdi: 249.8 MiB, 261881856 bytes, 511488 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xc6df3976

Device Boot Start End Sectors Size Id Type
/dev/xvdi1 40 511487 511448 249.7M c W95 FAT32 (LBA)

The system attached the stick to /dev/xvdi and mounted the first and only partition on /dev/xvdi1. What permissions do those device files have?

$ ls -o /dev/xvdi*
brw-rw---- 1 root 202, 128 Mar 15 12:09 /dev/xvdi
brw-rw---- 1 root 202, 129 Mar 15 12:09 /dev/xvdi1

Both show a “b” in the first character, meaning a block device. Root has read/write permission to the whole device and to the first partition. Using the “-l” option instead would show the group name as “disk”, a pseudouser name for operating on the disk devices without being the root user. Notice also that instead of a length there are two numbers separated by a comma and space. These are the device numbers (major, minor) used by the kernel to identify the device driver. The important take-away is that only root can read or write directly to these devices, so all operations on these must use sudo.

After mounting the device, it looks like this:

$ df -h
Filesystem           Size Used Avail Use% Mounted on
[...]
/dev/xvdi1           250M  12M  238M   5% /run/media/user/Spare256

In other words, the first partition, /dev/xvdi1, mounted on /run/media/user/Spare256, uses the USB stick’s label name as the subdirectory name for the mountpoint.

Copying the Disk

Before creating the ISO file for this stick, be sure there is enough free space to store it. This particular stick is 250M (labeled 256 MB). I chose it because it’s a small sample, easy to work with. If yours is several G, better be sure of your available space.

$ df -h .
Filesystem          Size  Used Avail Use% Mounted on
/dev/mapper/dmroot  9.8G  7.8G  1.5G  85% /home

There is a period at the end of the command line. This command shows that the DispVM in use has only 1.5G remaining, but that’s big enough for this example. If it wasn’t I’d have to expand the VM, use another VM with more space, or move some files elsewhere to make space.

The general form of the command to copy the entire drive into an ISO file is:

$ sudo dd if=DEVICENAME of=PATHNAME.iso

The dd (data dump) program takes the input file (if) identified by the device name and writes it to the output file (of) identified by the pathname of the file using an ISO suffix. Other operations defined in its man page can be added to this command. Because a large, time consuming ISO file might result, a good dd argument to add is “status=progress” to show you its progress through the file.

$ time sudo dd if=/dev/xvdi1 of=testrun.iso status=progress
260297216 bytes (260 MB) copied, 33.355364 s, 7.8 MB/s
511448+0 records in
511448+0 records out
261861376 bytes (262 MB) copied, 33.5436 s, 7.8 MB/s

real    0m36.127s
user    0m1.505s
sys     0m9.526s

$ ls -o testrun.iso 
-rw-r--r-- 1 root 261861376 Mar 15 12:45 testrun.iso

The first progress line shows the bytes transferred, the time elapsed, and the transfer speed of the copy as it goes. When dd finishes, it shows how many records it processed. If you don’t give dd a block size (bs), it uses the sector size (512 bytes) as the default input block size (ibs) and output block size (obs). Thus, 511,448 records (blocks) transferred successfully. Status updates periodically, so the last report itshown was only 2MB before the end. The final result shows the total that correlates with the records in/out. Notice that the listing without the “-h” option shows the actual byte size for verification.

Proving the Copy

Before doing anything drastic, like tossing the original CD/DVD out now that you have a copy, check that the copy is good. Mount the ISO file as if it were a disk drive and compare it with the original.

First, find a directory that isn’t being used to mount it on. The /mnt subdirectory could be used if it’s not already in use somehow. Sometimes a subdirectory of /mnt is available.

$ sudo mount -o ro testrun.iso /mnt/removable/

$ df -h
Filesystem          Size  Used Avail Use% Mounted on
[...]
/dev/xvdi1          250M   12M  238M   5% /run/media/user/Spare256
/dev/loop0          250M   12M  238M   5% /mnt/removable

They’re both there.

$ ls -ohd /run/media/user/Spare256/ /mnt/removable/
drwxr-xr-x 3 root 16K Dec 31 1969 /mnt/removable/
drwxr-xr-x 3 user 16K Dec 31 1969 /run/media/user/Spare256/

The “ls -d” option says show the directory only, not the files in it. If nearly everything is text the diff(1) command will do a creditable job:

$ time diff -qr /mnt/removable/ /run/media/user/Spare256/

real   0m1.593s
user   0m0.011s
sys    0m0.074s

The “-q” (quiet) option reduces the report to whether differences were detected without considering what kinds of differences. The “-r” option recurses through the subdirectories to compare all the files in the directories named. No news is good news. With no report output, they’re the same. Of course, you could compare each file in binary using cmp(1) but it has no recursive option, so there would have to be a loop:

$ time for f in $(find /mnt/removable -type f)
do
  echo cmp $f /run/media/user/Spare256/$(basename $f)
done

The problem with this loop comes in whether you eliminated spaces in the filenames. But, using “diff -qr” is typically good enough.

Duplicating the Copy

Before writing the ISO file onto another USB stick or drive, make sure the destination drive has at least the capacity of the original. Get as close to the original’s size as possible without going under. A 40G ISO file will not fit on a 32G stick, but it will fit on a 64G stick. Remaining space will be unallocated.

One way to avoid unallocated space is to format the destination drive into partitions where one partition is the size you need, or slightly larger, and the remaining space is a distinct partition. The cfdisk(8) program can do this. When creating the first partition, give it the ISO file size reported by “ls -oh“. If cfdisk allocates too little, quit without saving and redo it slightly larger. Then, allocate the second partition with all remaining space.

After formatting the new partitions, although technically the partition to hold the ISO doesn’t need formatting, make sure the partitions are not mounted. Use “df -h” to see if they’re present. If so, use umount(8):

$ sudo umount /run/media/user/LABELNAME

or

$ sudo umount /dev/DEVICENAME

Use the LABELNAME from the rightmost column that “df -h” gave or the DEVICENAME that the leftmost column that “df -h” gave. After umount, check with fdisk:

$ sudo fdisk -l

When they’re missing from the df output and present in the fdisk list, you’re ready to write the ISO file:

$ sudo dd if=PATHNAME.iso of=PARTITIONNAME

Of course, you can add “status=progress” to dd to monitor the operation. This time the command uses the PARTITIONNAME instead of the DEVICENAME. That way the partition table doesn’t get overwritten. When the ISO file is not intended to create a bootable filesystem, such as when you extract a CD/DVD that was not intended to boot — typically an operation system distribution is bootable — there’s no data in the ISO that makes it look like a filesystem. Write such data onto an existing partition for which a filesystem exists, such as FAT or NTFS. Remember that the partition has a number at the end of the device name.

If you choose not to create a second partition for the remaining space, but to create the only partition and take all the space, then use dd to write the smaller ISO, you’ll end up with the partition shrunk down to the ISO’s size when mounted. The rest of the partition space will be unavailable. Better to claim the space in multiple partitions and just write to one with the correct size.

Mounting an ISO on Windows 10

Open the Windows Explorer file viewer and select the device or directory containing the ISO file. In the file viewer list, select the ISO file.

  1. Double-click on the ISO file. On some installations, Windows opens a Power2Go app to explore the files in the ISO. One option allows burning it to a disk. Another option allows mounting it as a virtual drive, making available the directories and files on it.
  2. Right-click on the ISO file. The menu may offer several ways to view the files including the Power2Go app if installed, but the “Open With” submenu offers opening with Windows Explorer. Opening this way mounts the ISO file on a virtual drive. Directories and files are now available. Checking the properties shows that Windows is treating it as a CD/DVD even though it’s just a file.

There is a version of dd available for Windows that can create ISO files. It works somewhat differently due to the Windows way of working with devices, so study the options and test carefully.

Also, MinGW has MSYS utilities for running GNU utilities on Windows. Utilities include bash, gawk, and grep that improve cmd.exe operations.

Leave a Comment