Systemd is not the end of the world but someone needs to save us from it

This post has been months in the making. I believed I've reduced the rant-iness to a minimum level. It add nothing to the discussion elsewhere on the Internet. It is however, a burden off my chest.
I became aware of systemd as an init system in recent months as it gained traction in Fedora and it's competition with upstart. I didn't give it much of an interest as it primarily dealt with services that are more associated with the desktop: Plug and Play, power management etc. Imagined my surprise when I installed CentOS7 and it was the default init option. And going back wasn't an option.
The discussion around it's adoption has been intensive. This is a website that documents the fallacies in the arguments against for it and the follow-on discussion for a now-dead website advocating for boycotting it.
 My main beef with systemd is philosophical. Systemd's complexity and how it goes against the concept of "many simple programs doing basic jobs well working together" is not where Linux should be going. While many deride this backward-looking, that concept has served us well and has brought us this far. Yes, I do know that not everything that has brought us this far can carry us into the future but that point is evident only in hindsight. We should try new things but not at the cost of what works.  They should also be subject to discussion and mutual agreement. And finally there should be a transition between what is to what will become.

The opposite of this concept is "one big program doing many things complicated things", the best example being Windows. While some point out that systemd is a collection of applications, they are being developed together as a whole and made inter-dependent.
Another beef is the sense of (or lack thereof) of community around the development. The fan-boy-ism around it and the attitude of the developers (e.g the attitude towards corruption in binary logs) is alienating. Although it runs both ways, it is more healthy on the sysvinit side of things. I understand why there is a strong push-back from parts of the Linux community. This is the age-old difference between Linux users that are excited by solving problems and those that like to go home at the end of the day. Systemd is this complicated box / tool that can be used to solve complex problems in a complex way.  

Recover from a bad superblock

When things go really bad, you may not be able to recover a disk. In those times, think of salvaging the data, reformat and live to fight another day. Consider how valuable the data is versus the time spent on repairing something that is damaged and may not be salvagable. testdisk photorec ddrescue are the tools to think of when you come that decision
But I do enjoy a challenge and when a USB disk was brought to me with mounting problems, I just couldn't pass it up. It was an uncommon setup. The USB stick had two partitions, one with an ext3 filesystem and the other with FAT32. I decided to focus on the ext3 filesystem first.

FSCK (Photo credit: SFview)

To cut a long story short, my efforts to mount the disk met with screens full of error messages and cryptic clues as to what went wrong. Running fsck seemed to clean it first but it still would not mount the partition. Running fsck again would yield more and a different set of errors. My previous boss love to used the expression "time to decide: Fish or cut bait". It was one of those times.
This is probably the last ditch effect before you make that fateful decision. This is the line in the sand and the one you have to cross before deciding to put your effort in getting the data out and start all over again.
The recovery process involves rewriting the information about the partition. Specifically, reinitializing the superblock and group descriptors. However, reinitializing does not touch the data part of the partition. It does not touch things like the inodes and the blocks themselves. So by starting out with a 'fresh' set of information that is used to mount the disk, there is a possibility that the data may still be readable. After that, the data part gets checked and hopefully what you end up with is a filesystem that can be mounted properly.
The process can only be done when the partition is not mounted. If you have tried other ways, it most probably isn't. Mine wasn't, obviously.
So here's the process.
1. First, figure out the block size of the USB drive (in this case /dev/sdf1). I need that information to re-build the partition information. Run the command
dumpe2fs /dev/sdaf1 | grep 'block size' -i
Block size:               4096

2. Then format the superblocks. The command below won't format the whole partition, only the superblocks. It is critical that you use the correct block size gathered from the previous step
mke2fs -S -b 4096 -v /dev/sdf1

3. Now that the partition information is 'fresh', I checked the inodes to figure out what else could be wrong with the filesystem. Remember ext3 = ext2+journalling. So, ext2 tools still work
e2fsck -y -f -v -C 0 /dev/sdf1

4. Now that I've done with one element of the ext3 equation, it time to fix the journalling system or more specifically the journal data .
tune2fs -j /dev/sdf1

5. Re-attempt to mount the partition. If everything went well, you should be able to mount the partition and read the data.

After that, for hard disks, you have to determine whether the disk has reached it's threshold limits. Things like SMART properties will help you get that information.

Interested to know more: http://ubuntuforums.org/showthread.php?t=1681972&page=5&p=10434656#post10434656

Recover from a missing kernel : The Problem

You read right. A missing kernel. Although this sounds terminal, the fix was fortunately simple enough. If you are in a jam the solution is here. But the journey how it got to this is a cautionary tale of "a little knowledge is a dangerous thing'. This is a long post.

A novice sysadmin in a small company had problems with CentOS VMs on VMWare ESX version 3. I had set it up for them a few years ago and had been maintaining it for a while after that. I recommended to the management to send their sysadmin for training on VMWare administration, even if it wasn't for certification. They agreed on principle but never did anything about it. Don't get me wrong. The guy was smart. He shadowed my work and understood what I was doing and knew to ask questions when he didn't. Not formally trained but experienced in administrating services (e.g. Samba, printing), I think it was a normal progression for him to take on more work closer-related to installation and configuration.

The VMware config - Linux Kernel Dance
I had not heard from them for a while when I began getting calls about "network problems". A quick look and I figured out the VMs that was running their DHCP and DNS servers had frozen up (if you are wondering, the  Magic SysReq key a.k.a. Ctrl-Alt-SysReq BUSIER works in the VMI console). Apparently the VMs were running out of resources with the CPU hitting and sustaining 100% average utilization. There weren't many VMs on the server and being Linux, I knew I could cram more than they were running currently. A more closer look revealed that it was caused by vmware-tools not being loaded. It wasn't being loaded because the sysadmin had updated the kernel but not reconfigured vmware-tools. This was happening for some time despite the message during bootup warning him about it.
I call this the linux vmware-config dance. For reasons known to VMWare, Linux is a second-class citizen. Even though, VMware ESX and ESXi and their flagship product VSphere, run on a Linux kernel, Linux support comes second in everything. The all-powerful VMware Machine Interface (VMI) client is Windows only. Don't point to that pathetic web-based management system. On Linux, we could start and stop servers but console access is broken or at best, works sporadically. We can't even create a VM using it. It's better with VMware Server Free (previously ESX). The web interface provides full access but it requires an ssl 1.0 support which is insecure and requires manual parameter configuration in Firefox to work .
The Vmware-tools service provides the kernel optimized access to memory, disk and network access. If it's not running, the VM can't do things like share memory with other VMs. Basically, it'll run slower and eat up more memory. And apparently, run it long enough, some resource gets gobbled up bit by bit without being properly released. The kicker is that since re-configuring vmware-tools affects network access, you can't do it remotely via ssh. It must be done via the console either through the web interface (VMware Server Free only) or the VMI for the paid stuff.
VMWare requires that vmware-tools to be reconfigured every time there is a kernel update. Updated kernels need to be loaded first, so a reboot is required. If you use a RedHat or SuSe kernel, the related Vmware-tools modules will load ok. But if it doesn't it'll recompile the modules. So you will need gblic and at least kernel headers to recompile. Depending on the distro, you may need to load kernel sources to get the headers. It's also good to restart the server after reconfiguration and reloading of vmware-tools to test whether there are knock-on effects on other modules. So to recap: restart the machine to load new kernel, reconfig vmware-tools and restart to cleanly load and test vmware tools and other modules. Now times that with the number of VMs you have and look at spending a lot of time doing this.

Linux to the Rescue... Again

The least favorable job a Linux guy can get is... supporting newbie Windows users. While we live in a virus-free, relatively trojan-less environment, our Windows brethren are waist deep in shady toolbars, gotcha embedded web auto-downloads and the un-safe USB drives. It tickles me to no end when a web pop-up tries to convince me that I am looking at my files in windows explorer..on Linux. And while we may feel smug in the knowledge that our understanding of the underlying technology and Internet services allow us to take the necessary precautions, it is these skills that we are often employed to get Windows users back to being productive in the office.
While Linux-To-The-Rescue meant in the past safe partition resizing courtesy of parted (and later libparted-powered tools), all encompassing backups of partitionimage and harrowing hard disk ER with Photorec and Testdisk,  for a long time, virus and malware recovery work tools were limited to ClamAV, which itself it rather limited and really designed to detect viruses in e-mail.

Boot up screen - Notice the Memtest86+ memory tester

Well, now we have new hero on the block: AVG Rescue CD. I had thought about something along these lines some time ago. With the lawlessness of Windows of a few years ago, between Microsoft threatening to turn it's back on XP one more time (favoring Vista against user's wishes) and the overwhelming rejection of the business community to Vista, viruses and trojans seem to propagate at will; building botnets that continue to be reconstituted past when their mother ship has been detroyed. They were getting wilder too, being able to evade (literally) virus scans or rendering installed (but not updated) virus scanners impotent. I had a talk with friends at PandaSecurity about the viability of building a live cd around their command line scanner version that would mount windows partitions and scan them a few years ago. This is the best time to catch the trojans and virus, knife them when they are asleep. I had a problem with malware on my work PC's Windows partition. I ended up booting up from the Linux partition and trying to find the offending files using Clamav. This was before the age of writable NTFS (courtesy of ntfs-3g). So it was a cycle of scanning on Linux, copying down the location of the infected files, booting into Windows in SafeMode and deleting the file and back to booting on Linux and scanning again. Repeat until ClamAV found no more and then I would boot into Windows properly and run the updated Windows antivirus. It took a course of two days.

Main Menu

I heard nothing back from Panda. But a few months back, Panda also came out with a working LiveCD version that you can boot into and scan the PC with. I've used Panda's and AVG's and for now, I prefer AVG's LiveCD because it is light, works quickly to boot up, easy to update the pattern and program, has a character-based menu driven interface and is compatible with a lot of network hardware. It wasn't as compatible a few versions ago  (not recognizing some on-board NIC), but even then it worked on more computers than the Panda Security version. Panda's LiveCD's  GUI demands, which I think uses direct VESA / framebuffer rendering, makes it incompatible with a lot of PCs I use. AVG also bundles some tools commonly found on standard Linux recovery CDs like, PhotoRec and TestDisk. Both of these solutions read the disk in it's entirety and takes a long time to finish (read: hours). Users can't work on anything while it is running either (although one friend did marvel at what he could do with the Links text-based web browser on the AVG version that supports console switching). Users hate to wait but then again, it's their fault they run Windows. :)