Repair Damaged RPM Database

If you encounter any of the following messages:

rpmdb: PANIC: fatal region error detected; run recovery
error: db3 error(22) from dbenv->open: Invalid argument
error: cannot open Packages index using db3 - Invalid argument (22)
error: cannot open Packages database in /var/lib/rpm
error: rpmdb: damaged header #1439 retrieved -- skipping. 


or a similar RPM-related error message, it is probable that your RPM database is damaged.

RPM uses a database to maintain the package dependency information. This database is normally located at /var/lib/rpm.

# cd /var/lib/rpm
# ls -al
total 86420
drwxr-xr-x.  2 root root     4096 Nov 18 21:09 .
drwxr-xr-x. 45 root root     4096 Aug 25  2013 ..
-rw-r--r--.  1 root root  5820416 Sep  8  2013 Basenames
-rw-r--r--.  1 root root    16384 Aug 25  2013 Conflictname
-rw-r--r--.  1 root root   237568 Nov 18 21:30 __db.001
-rw-r--r--.  1 root root    73728 Nov 18 21:30 __db.002
-rw-r--r--.  1 root root  1318912 Nov 18 21:30 __db.003
-rw-r--r--.  1 root root  2695168 Sep  8  2013 Dirnames
-rw-r--r--.  1 root root    28672 Sep  8  2013 Group
-rw-r--r--.  1 root root    28672 Sep  8  2013 Installtid
-rw-r--r--.  1 root root    73728 Sep  8  2013 Name
-rw-r--r--.  1 root root    36864 Aug 19  2013 Obsoletename
-rw-r--r--.  1 root root 76402688 Sep  8  2013 Packages
-rw-r--r--.  1 root root   921600 Sep  8  2013 Providename
-rw-r--r--.  1 root root   647168 Sep  8  2013 Requirename
-rw-r--r--.  1 root root        0 Aug 18  2013 .rpm.lock
-rw-r--r--.  1 root root   172032 Sep  8  2013 Sha1header
-rw-r--r--.  1 root root   102400 Sep  8  2013 Sigmd5
-rw-r--r--.  1 root root     8192 Aug 19  2013 Triggername


RPM uses Berkeley DB, either version 3 or 4, for its database engine. Each of the Berkeley DB subsystems within an environment is described by one or more shared memory regions. Regions contain the per-process and per-thread shared information, including mutexes, that comprise a Berkeley DB environment.

Any files created in the filesystem to back the regions are created in the environment home directory specified to the DB open call. These files are named __db.###, e.g. __db.001, __db.002 and so on. When region files are backed by the filesystem, as in RPM, one file per region is created.

By the way, statistics about these regions can be displayed using the db_stat utility:

# cd /var/lib/rpm
# db_stat -e
Thu Dec 10 23:01:10 2015	Local time
0x120897	Magic number
0	Panic value
5.2.36	Environment version
9	Btree version
9	Hash version
1	Lock version
18	Log version
4	Queue version
2	Sequence version
1	Txn version
Wed Nov 18 21:09:09 2015	Creation time
0xcfca7f84	Environment ID
2	Primary region allocation and reference count mutex [0/10269 0% !Own]
1	References
232KB	Current region size
808KB	Maximum region size
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Thread tracking information
7	Thread blocks allocated
8	Thread allocation threshold
37	Thread hash buckets
Thread status blocks:
	process/thread 1475/3077674688: active
	process/thread 1318/3078076096: out
	process/thread 1453/3077822144: out
	process/thread 1452/3078510272: out
	process/thread 1274/3078506176: out
	process/thread 1448/3078063808: out
	process/thread 1451/3078309568: out
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Environment file handle information
__db.001	file-handle.file name
0	file-handle.mutex [!Set]
0	file-handle.reference count
3	file-handle.file descriptor
0	file-handle.page number
0	file-handle.page size
0	file-handle.page offset
0	file-handle.seek count
0	file-handle.read count
0	file-handle.write count
DB_FH_OPENED	file-handle.flags
....

To fix a corrupted RPM database:

# rm -rf /var/lib/rpm/__db*
# db_verify /var/lib/rpm/Packages
# rpm --rebuilddb

I would also run the following commands to clear out YUMs caches and test the rebuilt RPM database:

# yum clean all
# rpm -qa 


Obviously to minimize risk, you should make a backup of all files in /var/lib/rpm/ before attempting to rebuild the RPM database.

What causes a RPM database corruption? It is typically due to an aborted database update process. Occasionally, it occurs when the RPM package itself is updated. When RPM accesses the Berkeley DB files, it makes temporary entries within the tables while it searches for data. If you abort RPM often, this issue will occur much sooner because locks were not cleared when a thread died in Berkeley DB library routines.

The db_verify utility is a Berkeley DB utility which verifies the structure of one or more files and the databases they contain. It should only be used on files that are not being modified by another thread of control. It is used here to ensure that the installed package headers are not corrupted.

Finally, rpm –rebuilddb is used to rebuild the database indices from the installed package headers.

What is /proc/config.gz?

Linux can store a gzip’ed copy of the kernel configuration file used to build the kernel in the kernel itself, and make it available to users via /proc/config.gz.

This file is not always present in a Linux distribution. It is only present when /proc support is enabled and CONFIG_IKCONFIG and CONFIG_IKCONFIG_PROC are set to Y in the kernel configuration file. Most distributions do not enable it by default; instead they include the kernel configuration file under /boot.

If in-kernel configuration support is built as a kernel module, then before you can use /proc/config.gz, the configs module needs to be loaded into the kernel using insmod or modprobe.

To quickly check a configuration setting, you could use zcat or zgrep:

# zcat /proc/config.gz | grep IKCONFIG
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y

To extract a copy of the kernel configuration file you could do something like the following:

# zcat /proc/config.gz > /boot/config-$(uname -r).new

Finally, if CONFIG_IKCONFIG is set but CONFIG_IKCONFIG_PROC was not set, the extract-ikconfig script can be used to extract the configuration file from a kernel image file.

Zero Padding Brace Expansions in the Korn Shell

Both bash and zsh shells support leading zeros in ranges:

$ echo {1..10}
1 2 3 4 5 6 7 8 9 10
$ echo {01..10}
01 02 03 04 05 06 07 08 09 10
$ echo {001..010}
001 002 003 004 005 006 007 008 009 010
$

From the bash manpage section on brace expansion:

Supplied integers may be prefixed with 0 to force each term to have the same 
width. When either x or y begins with a zero, the shell attempts to force all 
generated terms to contain the same number of digits, zero-padding where necessary

If you want a comma separated range of numbers, you can do something like:

$ echo {01..10} | tr ' ' ','
01,02,03,04,05,06,07,08,09,10

The Korn shell does not support the leading zero syntax in brace expansion. However it does support an equally powerful alternative syntax using %0d:

$ echo {1..10}
1 2 3 4 5 6 7 8 9 10
$ echo {01..10%02d}
01 02 03 04 05 06 07 08 09 10
$ echo {01..10%03d}
001 002 003 004 005 006 007 008 009 010

$ echo {01..10%02d} | tr ' ' ','  
01,02,03,04,05,06,07,08,09,10

Finally, you can also use this style of syntax to create a range of leading zero hexadecimals:

$ echo {7..18%02x}  
07 08 09 0a 0b 0c 0d 0e 0f 10 11 12

Intel Processor Power Management

Modern Intel processor power management is based on a combination of two techniques: changing the CPU frequency (P-states) and using idling power states (C-states). A third technique, used on older processors but rarely on modern processors, is throttling of the CPU (T-states).

P-states are voltage-frequency pairs that set the speed and power consumption of an Intel processor. When the operating voltage of the processor is lower, so is the power consumption.

C-states are idle power saving states, in contrast to P-states, which are execution power saving states. During a P-state, the processor is still executing instructions, whereas during a C-state (other than C0), the processor is idle, meaning that nothing is executing. Thus C-states are idle states and P-states are operational states. C-states and P-states are orthogonal, i.e. one can vary without affecting the other. C-states have been available for processors targeted at the mobile market for many years but have been only available for the server processors since Nehalem.

In the C0 state, a processor executes instructions. A processor running in any other C-state is somewhat idle. The higher the C-state number, the deeper the CPU sleep mode and the more power saved. The downside to C-States is the latency when reverting to C0. Some C-States have submodes with different power saving latency levels. C1 is always available but which other C-states and submodes are supported is processor-specific.

If you want the absolute minimum latency, set the kernel parameter idle to poll. This keeps the processor(s) at C0 even when idle. Note that keeping processors at C0 can interfere with hyperthreading, i.e. logical cores.

From the the current Linux Kernel Parameters document:

idle=		[X86]
			Format: idle=poll, idle=halt, idle=nomwait
			Poll forces a polling idle loop that can slightly
			improve the performance of waking up a idle CPU, but
			will use a lot of power and make the system run hot.
			Not recommended.
			idle=halt: Halt is forced to be used for CPU idle.
			In such case C2/C3 won't be used again.
			idle=nomwait: Disable mwait for CPU C-states

The higher the P-state, the lower the frequency and voltage at which a processor runs. Once again, the number of P-states is again processor-specific and implementations differ across the various processor families. However, P0 is always the highest-performance state. Higher P-state numbers represent slower processor speeds and lower power consumption. By the way, P-state technology is known as SpeedStep in Intel processors.

Linux supports a CPUfreq subsystem which can be used to modify P-States. If it is enabled on a particular Linux distribution, you will find the relevant files under /sys/devices/system/cpu/. There will be a separate subdirectory for each processor with a cpufreq subdirectory in each processor directory, holding a number of files that define the parameters for CPUfreq.

Finally, a useful tool is cpufreq-info which can be used to retrieve cpufreq-related kernel information.

On-disk File Timestamps

The Single Unix Specification, Base Definitions (XBD), Section 4.8 entitled “File Times Update” states

“An implementation may update timestamps that are marked for update immediately, or it may update such timestamps periodically.”

This means that, for example, that file read and write operations are free to set the appropriate flags in the in-memory structures and do the actual updating of the on-disk filesystem structures at a later time.

Assuming periodically means from time to time, it implies that a POSIX-compliant operating system is free to update it’s on-disk structures when it is convenient for the operating system to do so. This means that, should the operating system crash, the on-disk recorded times might not reflect the actual true last access times.

BSD and Solaris UFS are two filesystems that exhibit such behavior. The Solaris UFS file systems does not update access times unless the disk is accessed or a certain events occur. BSD only periodically updates it’s on-disk filesystem timestamps. On BSD, all

timestamps that are marked for update are updated when the file ceases to be open by any process or before a fstat(), fstatat(), fsync(), futimens(), lstat(), stat(), utime(), utimensat(), or utimes() is successfully performed on the file.

From a forensics viewpoint, this implies that you need to understand the underlying filesystem on-disk timestamp updating mechanism before you can reliably rely on an individual file timestamp.