Sudo and Globbing

The question is how we can use the sudo utility to display a list of files in a directory to which we have absolutely no Unix filesystem privileges

Consider the following directory and files contained therein:

$ ls -l
total 4
drwxrwx---. 2 root root 4096 May 22 21:14 demo
$ su 
Password: XXXXXXXX 
# ls -l demo
total 0
-rw-r--r--. 1 root root 0 May 22 21:14 file1
-rw-r--r--. 1 root root 0 May 22 21:14 file2
-rw-r--r--. 1 root root 0 May 22 21:14 file3
# exit
exit


Note the directory permissions are 770 and the user and group owner is root.

Here is the output when I attempt to list the subdirectory contents:

$ whoami
fpm
$ ls -l demo
ls: cannot open directory demo: Permission denied
$ 


This is what I expect because I am not root nor a member of the root group. To enable me to view the directory contents, the directory permissions need to be at least 771 (drwxrwx–x) for a non-root user such as myself to list the directory.

So how can I work around this limitation? One workaround is by using sudo. Assuming I have root privledges via sudo either explicitly or as a member of the wheel group, here is what happens when I naively use sudo to attempt to list the subdirectory contents:

$ sudo ls -l demo/file1
-rw-r--r--. 1 root root 0 May 22 21:14 demo/file1
$ sudo ls -l demo/*
ls: cannot access demo/*: No such file or directory
$ 


I still cannot list the files in the demo subdirectory.

The above does not work because the shell attempts to glob demo/* before passing the result of globing to sudo which then passes it unchanged to ls. Since the shell is not running as root, it is not able to traverse the directory and globbing fails. Thus, ls receives the unexpanded globbing pattern demo/* in its argument list.

Note that ls and other Linux utilities do not perform globbing. The utilities expect a list of filenames as command line arguments.

The most straight forward way to get correct results is to invoke ls -l demo/* in a sub-shell. Quote the command string using either double quotes (“) or single quotes (‘).

$ sudo sh -c "ls -l demo/*"
-rw-r--r--. 1 root root 0 May 22 21:14 demo/file1
-rw-r--r--. 1 root root 0 May 22 21:14 demo/file2
-rw-r--r--. 1 root root 0 May 22 21:14 demo/file3
$ sudo sh -c 'ls -l demo/*'
-rw-r--r--. 1 root root 0 May 22 21:14 demo/file1
-rw-r--r--. 1 root root 0 May 22 21:14 demo/file2
-rw-r--r--. 1 root root 0 May 22 21:14 demo/file3
$ sudo -s ksh -c "ls -l demo/*"
-rw-r--r--. 1 root root 0 May 22 21:14 demo/file1
-rw-r--r--. 1 root root 0 May 22 21:14 demo/file2
-rw-r--r--. 1 root root 0 May 22 21:14 demo/file3
$ 


As you can see, there is no difference in output.

Note that there may be a difference if you use shell variables. Here is an example where there is a difference.

$ sudo sh -c 'echo $USER'
root
$ sudo sh -c "echo $USER"
fpm


There are other workarounds to this issue besides sudo. For example, I could be added to the root group but that has security implications. Another group of which I am a member could be given group ownership of the subdirectory. POSIX ACLs could be used to grant browsing and other permissions to the subdirectory.

Case Insensitive File Listings

In general, Linux filenames are case-sensitive. So how case you list the files in your current directory in a case-insensitive way?

Technically, you are looking to perform case-insensitive globbing.

Here are some ways:

# touch DEMO demo DeMo
# ls -l [Dd][Ee][Mm][Oo]
-rw-r--r--. 1 root root 0 Mar 24 11:39 demo
-rw-r--r--. 1 root root 0 Mar 24 11:39 DeMo
-rw-r--r--. 1 root root 0 Mar 24 11:39 DEMO

# ls -l | grep -i demo
-rw-r--r--. 1 root root 0 Mar 24 11:39 demo
-rw-r--r--. 1 root root 0 Mar 24 11:39 DeMo
-rw-r--r--. 1 root root 0 Mar 24 11:39 DEMO

# ls -l ~(i:demo)
-rw-r--r--. 1 root root 0 Mar 24 11:39 demo
-rw-r--r--. 1 root root 0 Mar 24 11:39 DeMo
-rw-r--r--. 1 root root 0 Mar 24 11:39 DEMO
# 


The first method works on all modern shells and is very efficient. The second method invokes another process in most shells, and the third method only works in the Korn shell.

The zsh shell has the CASE_GLOB and EXTENDED_GLOB options which perform a similar function. Refer to the zsh manpages for more details.

Another method is to use the find utility with the iname option:

find . -maxdepth 1 -iname 'demo' -ls
131632    0 -rw-r--r--   1 root     root            0 Mar 24 11:39 ./DeMo
131622    0 -rw-r--r--   1 root     root            0 Mar 24 11:39 ./demo
130963    0 -rw-r--r--   1 root     root            0 Mar 24 11:39 ./DEMO

Configuring IP Dynamic Port Ranges

As you are probably aware, IP port numbers are 16 bit unsigned integers in the range 0 to 65535. IANA (Internet Assigned Numbers Authority) manages these port numbers. See the IANA Service Name and Transport Protocol Port Number Registry for more information.

IANA specifies the range 49152 to 65535 be used for dynamic (AKA private) ports. From the above referenced document:

   Port numbers are assigned in various ways, based on three ranges: System
   Ports (0-1023), User Ports (1024-49151), and the Dynamic and/or Private
   Ports (49152-65535); the difference uses of these ranges is described in  
   RFC6335. 

   System Ports are assigned by IETF process for standards-track protocols, 
   as per RFC6335.  User Ports are assigned by IANA using the "IETF Review" 
   process, the "IESG Approval" process, or the "Expert Review" process, as per
   RFC6335.  Dynamic Ports are not assigned.

Contrary to the IANA, Linux kernels typically use the range 32768 to 61000. In 2007, the Linux kernel maintainers decided to use that range following a short discussion on the Linux kernel mailing list (See lkml.org).

Two kernel settings control the IP dynamic port ranges on a system, i.e. ip_local_port_range and ip_local_reserved_ports.

# pwd
/proc/sys/net/ipv4

# ls | grep port
ip_local_port_range
ip_local_reserved_ports

# cat ip_local_port_range 
32768	61000
# cat ip_local_reserved_ports 

#

The range specified in ip_local_port_range determines the port assigned to an application by default when the application itself does not specify a TCP or UDP port to use for an outgoing connection. The upper range value (port number) was reduced to 61000 because ports above that number are used for masquerading ports when masquerading is enabled. The lower range value was selected because it was felt that more ports were needed than suggested by IANA.

From the kernel documentation:

ip_local_port_range - 2 INTEGERS
     Defines the local port range that is used by TCP and UDP to
     choose the local port. The first number is the first, the
     second the last local port number. The default values are
     32768 and 61000 respectively.

The ip_local_port_range enables you to add exceptions to the range of ports listed in ip_local_port_range.

From the kernel documentation:

ip_local_reserved_ports - list of comma separated ranges
     Specify the ports which are reserved for known third-party
     applications. These ports will not be used by automatic port
     assignments (e.g. when calling connect() or bind() with port
     number 0). Explicit port allocation behavior is unchanged.

     The format used for both input and output is a comma separated
     list of ranges (e.g. "1,2-4,10-10" for ports 1, 2, 3, 4 and
     10). Writing to the file will clear all previously reserved
     ports and update the current list with the one given in the
     input.

     Note that ip_local_port_range and ip_local_reserved_ports
     settings are independent and both are considered by the kernel
     when determining which ports are available for automatic port
     assignments.

     You can reserve ports which are not in the current
     ip_local_port_range, e.g.:

         $ cat /proc/sys/net/ipv4/ip_local_port_range
          32000  61000
         $ cat /proc/sys/net/ipv4/ip_local_reserved_ports
          8080,9148
	
      although this is redundant. However such a setting is useful
      if later the port range is changed to a value that will
      include the reserved ports.

      Default: Empty

Access A VMware Virtual Disk from Linux

Recently I needed to recover some data off a VMware Workstation virtual disk using Linux. As you probably know, VMware Workstation (and VMware vSphere) represents a physical disk by a virtual disk whose backing store is one or more VMDK files.

VMDK (Virtual Machine Disk) is a documented file format, which uses the file extension .vmdk. It describes containers for virtual hard disk drives to be used in virtual machines like VMware Workstation or VirtualBox. Initially developed by VMware for its virtual appliance products, nowadays VMDK is an open format.

Fortunately, VMware provides a simple utility, vmware-mount, for mounting a VMware virtual disk on a Linux system. The utility depends on Fuse (File system in USEr Space) which is a loadable kernel module. Fuse allows regular (non-root) users to create and access their own file systems. The file system code runs in user space, while the Fuse module provides a bridge to the actual kernel mount interfaces. Fuse was merged into mainstream Linux in kernel version 2.6.14.

# lsmod | grep fuse
fuse    91371  7 

# modinfo --filename fuse
/lib/modules/3.17.8-300.fc21.x86_64/kernel/fs/fuse/fuse.ko.xz


Note that you may have to create loop devices (/dev/loopX) for Fuse to work correctly. Fedora 20 and 21, for example, do not have loop devices installed by default.

You can mount the VMDK as a loop device:

# ls -l
-rw-r--r--. 1 fpm fpm 7706112 Jan 28 09:14 UEFI-disk1.vmdk

# vmware-mount -p ./UEFI-disk1.vmdk 
Nr      Start       Size Type Id Sytem                   
-- ---------- ---------- ---- -- ------------------------
 1       2048    2095104 BIOS  B Win95 FAT32

# vmware-mount ./UEFI-disk1.vmdk /mnt
# df
Filesystem     1K-blocks      Used Available Use% Mounted on
....
/dev/loop0       1045488     28304   1017184   3% /mnt
# 

# cd /mnt
# ls -l
total 8
drwxr-xr-x. 6 root root 4096 Dec  1 05:04 efi
-rwxr-xr-x. 1 root root   10 Nov 29 19:24 startup.nsh

# find efi/tools
efi/tools
efi/tools/ftp.efi
efi/tools/hostname.efi
efi/tools/hexdump.efi
efi/tools/which.efi
efi/tools/python.efi
# 

# vmware-mount -d /mnt 
#

or you can mount a flat-file representation of the entire virtual disk:

# vmware-mount -f ./UEFI-disk1.vmdk /mnt 

# df
Filesystem     1K-blocks      Used Available Use% Mounted on
/dev/sda4      479468544  17989652 460335532   4% /
devtmpfs         8153896         0   8153896   0% /dev
tmpfs            8164292       260   8164032   1% /dev/shm
tmpfs            8164292      1004   8163288   1% /run
tmpfs            8164292         0   8164292   0% /sys/fs/cgroup
tmpfs            8164292       256   8164036   1% /tmp
/dev/sda4      479468544  17989652 460335532   4% /home
/dev/sdb1      480589524 383220116  72933748  85% /home/fpm
/dev/sda2         487652    131166    326790  29% /boot
/dev/sda1         204580    105944     98636  52% /boot/efi
tmpfs            1632860        20   1632840   1% /run/user/1000
/dev/sdc1         978692     35016    943676   4% /run/media/fpm/CENTON USB

# cd /mnt
# ls -al
total 1048576
-rw-r--r--. 1 fpm fpm 1073741824 Jan 30 21:47 flat

# vmware-mount -L
Disks with mounted partitions:
	/home/fpm/tmp/UEFI-disk1.vmdk	/mnt/flat

# fdisk -lu /mnt/flat

Disk /mnt/flat: 1 GiB, 1073741824 bytes, 2097152 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00067806

Device     Boot Start     End Sectors  Size Id Type
/mnt/flat1       2048 2097151 2095104 1023M  b W95 FAT32

# dd if=/mnt/flat bs=512 skip=2048 count=1 | xxd
0000000: eb58 906d 6b64 6f73 6673 0000 0208 2000  .X.mkdosfs.... .
0000010: 0200 0000 00f8 0000 3f00 ff00 0000 0000  ........?.......
0000020: 00f8 1f00 0008 0000 0000 0000 0200 0000  ................
0000030: 0100 0600 0000 0000 0000 0000 0000 0000  ................
0000040: 0000 2967 c83b 9120 2020 2020 2020 2020  ..)g.;.         
0000050: 2020 4641 5433 3220 2020 0e1f be77 7cac    FAT32   ...w|.
0000060: 22c0 740b 56b4 0ebb 0700 cd10 5eeb f032  ".t.V.......^..2
0000070: e4cd 16cd 19eb fe54 6869 7320 6973 206e  .......This is n
0000080: 6f74 2061 2062 6f6f 7461 626c 6520 6469  ot a bootable di
0000090: 736b 2e20 2050 6c65 6173 6520 696e 7365  sk.  Please inse
00000a0: 7274 2061 2062 6f6f 7461 626c 6520 666c  rt a bootable fl
00000b0: 6f70 7079 2061 6e64 0d0a 7072 6573 7320  oppy and..press 
00000c0: 616e 7920 6b65 7920 746f 2074 7279 2061  any key to try a
00000d0: 6761 696e 202e 2e2e 200d 0a00 0000 0000  gain ... .......
00000e0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000f0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000100: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000110: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000120: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000130: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000140: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000150: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000160: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000170: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000180: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000190: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00001a0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00001b0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00001c0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00001d0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00001e0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00001f0: 0000 0000 0000 0000 0000 0000 0000 55aa  ..............U.


Recognize what sector this is? Look at the 55 AA signature at the end of the sector. This designates an Master Boot Record (MBR). An MBR is located in the first sector of an MBR-partitioned disk.

Note that with VMware Workstation 10 you may get the error message VixDiskLib: Invalid configuration file parameter. Failed to read configuration file. You can safely ignore that message.

Best practice is to make a backup copy of your virtual disk before mounting it using the vmware-mount utility.

Shell String Concatenation

Many people are unaware that string concatenation using the += syntax is fully supported in modern versions of both the Bash and Korn shells

Here is a simple example which demonstrates the syntax:

str="Hello"
$ str+=" "
$ str+="World"
$ echo $str
Hello World
$ 

If you use this syntax with “numbers”, you end up with a string that looks like a number but is not a number.

$ a=2
$ a+=4
$ echo $a
$ 24