Migrating to RHEL 6 — what surprised us

Red Hat Enterprise Linux 5 served us faithfully for four years. But with the end of extended support on the horizon and the ever-increasing demands of our Java applications on system resources, we decided at the turn of the year to upgrade to RHEL 6. What looked like a routine operation turned out to be an interesting challenge.

Why we migrated¶

RHEL 6 brought a new kernel 2.6.32 with significantly improved memory management, ext4 as the default filesystem and a completely reworked init system — Upstart instead of classic SysV init. For our production environment with dozens of services this meant rewriting most init scripts. The main motivation was support — RHEL 5 was approaching the end of full support and security patches were arriving more slowly. For our clients in the financial sector, current OS support is a condition of operation.

SELinux — for real this time¶

Let’s be honest — on RHEL 5 we disabled SELinux on most servers. It was easier than dealing with endless AVC denial messages. On RHEL 6 we decided to do it right and leave SELinux in enforcing mode. The first week was painful. Our custom Java applications running on Tomcat needed their own SELinux policies. The GlassFish server was writing logs to non-standard directories. Oracle Instant Client had problems accessing shared libraries. Each issue meant hours with audit2allow and manual policy tuning.

But the result is worth it. We now have servers where a compromised web application cannot read outside its own directory. For our clients in the banking sector, this is a fundamental security improvement.

Filesystem — ext4 and LVM¶

The transition from ext3 to ext4 went surprisingly smoothly. We particularly appreciated the faster fsck — on servers with terabyte filesystems this means a difference of hours during an unplanned restart. Extents instead of indirect block mapping sped up sequential I/O operations, which was most noticeable during backups. LVM snapshots on ext4 work more reliably and faster. Our backup script, which makes a consistent snapshot of the Oracle database, sped up by 40%.

Application compatibility¶

The biggest problem? Libraries. RHEL 6 switched to glibc 2.12 and some older binaries simply stopped working. Specifically, we were troubled by Oracle Database client 10g — it is not officially supported on RHEL 6. We had to upgrade to the 11g client, which meant changes to tnsnames.ora and testing all JDBC connections. Java 6 (OpenJDK) on RHEL 6 runs without issues, but be careful about differences in cryptographic providers.

Upstart vs. SysV init¶

Upstart is conceptually better than SysV init — event-driven, parallel service startup, automatic respawn. But rewriting twenty custom init scripts to Upstart configuration cost two days of work. Fortunately RHEL 6 also supports the old SysV scripts via a compatibility layer, so we migrated gradually. Tip: don’t migrate everything at once.

Networking and firewall¶

RHEL 6 still uses iptables, but the configuration tools are improved. NetworkManager is new as the default network manager. On servers we disable it immediately — we want static configuration in /etc/sysconfig/network-scripts, not a dynamic manager.

Lessons from the migration¶

Migrating an OS in production is never trivial. Plan at least twice as much time as you estimate. Test SELinux policies in advance on the staging environment. And above all — have a rollback plan.