AllowTcpForwarding no

If you disable SSH port forwarding, the logs will contain error messages when users try to use it. If it’s enabled, there will be no log entries. It’d be really great if I could have more fine grained control over what ports could be forwarded or if I could specify that port forwarding can only be done to localhost, but I’ve found no such setting so far. Maybe I can hack that in later. This did allow me to catch the user account that was doing it from the error messages:

Apr  9 12:12:16 <host>.cse.ohio-state.edu sshd[21527]: [ID 800047 auth.info] Accepted password for <user> from <remote IP> port 49571 ssh2

Apr  9 12:54:15 <host>.cse.ohio-state.edu sshd[21527]: [ID 800047 auth.info] Received request to connect to host <remote MTA> port 25, but the request was denied.

We checked in Splunk and found that the user in question logged in from numerous IPs in Europe. I was able to reproduce what I observed with the following SSH command:

ssh -N -L 1234:<remote MTA>:25 <CSE login server>

This forwards port 1234 to port 25 on the remote MTA though one of our login servers, all without actually logging in. This is why I couldn’t match it with a username. Luckily this only happened with one account. I disabled the account and turned SSH port forwarding back on because we actually have a couple of use cases where it’s necessary.

I wrote equivalent shell and Perl scripts to monitor for this situation. A shell script version that works on Linux follows:

#!/bin/bash

FROM='rowland'
RECIPIENTS='rowland'
PORTS='25 587'
EXCLUSIONS='127.0.0.1:25 127.0.0.1:587 164.107.115.49:25 164.107.115.49:587'

found=0
declare -a data
count=0

for n in $(netstat -n -p --inet |grep -v WAIT |sed '1,2d' |awk '{print $4 "," $5 "," $7}'); do
    s_port=$(echo $n |cut -f 1 -d ',' |cut -f 2 -d :)
    destination=$(echo $n |cut -f 2 -d ',')

    for e in $EXCLUSIONS; do
        if [ "$destination" = "$e" ]; then
            continue 2
        fi
    done

    d_port=$(echo $destination |cut -f 2 -d :)

    for p in $PORTS; do
        if [ "$d_port" = "$p" ]; then
            found=1
            data[$count]="$n"
            count=$((count + 1))
        fi
    done
done

if [ $found -eq 1 ]; then
    port_list=$(echo $PORTS |sed 's/ /,/g')
    mailx -s 'SMTP Port Connection Check Results' -r $FROM $RECIPIENTS <<EOF
SMTP Port Connection Check Results
==================================

Host:  $(hostname)
Date:  $(date)
Ports: $PORTS

netstat
-------

Excluded Destinations:

$(for e in $EXCLUSIONS; do echo $e; done)

Data:

$(for d in ${data[@]}; do echo $d |sed 's/,/   /g'; done)

lsof
----

$(/usr/sbin/lsof -i TCP:$port_list)

who
---

$(who)

ps
--

$(ps -ef)
EOF
fi

That runs fast on a new Linux server, but the Solaris version is pretty slow on our old hardware. I wrote a Perl version as well:

#!/usr/bin/perl

use warnings;
use strict;
use Sys::Hostname;

my($from, $recipients, @ports, @exclusions, $found, $destination, $dport);
my(@data, $hostname, $date, $ex_str, $data_str, $port_list, $lsof, $who, $ps);
my($email);

$from = 'rowland';
$recipients = 'rowland';
@ports = qw(25 587);
@exclusions = qw(127.0.0.1:25 127.0.0.1:587 164.107.115.49:25 164.107.115.49:587);

open(NETSTAT, "netstat -n -p --inet |grep -v WAIT |sed '1,2d' |") or die "can't open netstat";
$found = 0;

while (<NETSTAT>) {
    $destination = (split(/\s+/))[4];
    next if (grep { $_ eq $destination } @exclusions);
    $destination =~ /\:(\d+)$/;
    $dport = $1;
    chomp($dport);

    if (grep { $_ eq $dport } @ports) {
        $found = 1;
        chomp;
        push @data, $_;
    }
}

close(NETSTAT);

if ($found == 1) {
    $hostname = hostname;
    $date = localtime;
    $data_str = join("\n", @data);
    $ex_str = join("\n", @exclusions);
    $port_list = join(',', @ports);
    $lsof = `/usr/sbin/lsof -i TCP:$port_list`;
    $who = `who`;
    $ps = `ps -ef`;
    $email = <<"EOF";
SMTP Port Connection Check Results
==================================

Host:  $hostname
Date:  $date
Ports: @ports

netstat
-------

Excluded Destinations:

$ex_str

Data:

$data_str

lsof
----

$lsof

who
---

$who

ps
--

$ps
EOF
    open(MAILX, "|mailx -s 'SMTP Port Connection Check Results' -r $from $recipients") or die "can't open mailx";
    print MAILX $email;
    close(MAILX);
}

I’ve not written a Perl script in about forever, so that’s just quickly written. I write Ruby programs these days, but I needed something in Perl on Solaris. The Perl version is much faster on Solaris, but on Linux there isn’t a performance issue with either. There are only slight differences on Solaris, but I’ll leave that as an exercise for the reader.

Those scripts basically catch any connections to the desired ports that don’t fall into the exception cases. This would catch many things potentially, and since we need to allow SSH port forwarding, I have these running every minute in cron as root, which is important for the lsof command.

It’s now possible to create a /usr/local/etc/identd.user file that contains a UNIX username on one line alone. I don’t really check the contents. It will use whatever you put in that file as part of the response. I figured this was simple enough to avoid checking in order to get out a fix. If that exists before reinstalling identd, it should restart and load that data for the default user if it would otherwise respond that there is no corresponding user. This file can be created, deleted, or modified at any time. To get identd to notice, you can send a SIGUSR1 signal to it using kill:

[rowland@rowland ~]$ ps -e |grep identd |grep -v grep |awk '{print $1}'
5504
[rowland@rowland ~]$ sudo kill -USR1 5504
Password:

These details are now in the manual page. The source code is in my Git repositories and the install package is available here. This is the same location as before.

One part of the instructions that new users (perhaps all users) don’t seem to follow is not running multiple sessions and not letting them run for more than 24 hours. I had a shell script to kill old sessions in the past, but I lost it some time ago when I was cleaning my UNIX home directory. I wrote a new Ruby program to handle this instead:

#!/usr/local/bin/ruby

require 'ffi'
require 'ffi/tools/const_generator'

# The current short hostname.
HOSTNAME = `hostname -s`.chomp

# Proc process stat line position constants on Linux.
PID = 0
NAME = 1
STARTTIME = 21

# Proc stat line position constants on Linux.
BTIME = 6

# 24 hours ago from right now as a Time object.
DAY_AGO = Time.at(Time.now.tv_sec - 60 * 60 * 24)

# FFI interface to sysconf() to figure out the number of click ticks per
# second. This is needed to figure out the process start time since boot,
# which is given in jiffies (the number of clock ticks). This also gives
# an interface to getpwuid() to figure other user details that are needed.
module Unix
  extend FFI::Library

  class Passwd < FFI::Struct
    layout :pw_name,   :string,
           :pw_passwd, :string,
           :pw_uid,    :uint32,
           :pw_gid,    :uint32,
           :pw_gecos,  :string,
           :pw_dir,    :string,
           :pw_shell,  :string
  end

  ffi_lib FFI::Library::LIBC
  attach_function 'sysconf', [:int], :long
  attach_function 'getpwuid', [:uint32], :pointer
end

cg = FFI::ConstGenerator.new do |gen|
  gen.include 'unistd.h'
  gen.const('_SC_CLK_TCK')
end

CLOCK_TICKS = Unix.sysconf(cg['_SC_CLK_TCK'].to_i)

# Figure out the boot time.
btime = nil

File.open '/proc/stat' do |file|
  btime = file.each_line.find { |s| s =~ /^btime / }
  btime = btime.split[1].to_i if btime
end

unless btime
  abort "Can't determine system boot time. See above line #{__LINE__}."
end

VncServer = Struct.new(:name, :pid, :start_time, :uid, :username, :home, :gid,
                       :display)
vnc_servers = []

boot_time = Time.at(btime)
header = "VNC Cleanup #{Time.now}"
puts header
puts '=' * header.length
puts
puts "Hostname: #{HOSTNAME}"
puts "Boot Time: #{boot_time}"
puts "Clock Ticks/Second: #{CLOCK_TICKS}"
puts
puts "VNC Processes"
puts "-------------"
puts
puts "* = Xvnc processes over 24 hours old that will be killed."
puts
error = false

Dir.glob('/proc/[0-9]*/stat') do |ps|
  begin
    stat = IO.read(ps).split
    name = stat[NAME]

    if name == '(Xvnc)'
      pid = stat[PID].to_i
      uid = nil
      username = nil
      home = nil
      gid = nil

      # Grab the effective UID and other user details as well.
      File.open "/proc/#{pid}/status" do |file|
        uid = file.each_line.find { |s| s =~ /^Uid:/ }

        if uid
          uid = uid.split[2].to_i
          passwd = Unix.getpwuid(uid)
          passwd = Unix::Passwd.new(passwd)
          username = passwd[:pw_name]
          home = passwd[:pw_dir]
          gid = passwd[:pw_gid]
        end
      end

      # Grab the display number.
      display = IO.read("/proc/#{pid}/cmdline").split("\u0000")[1]
      start_time = Time.at(btime + stat[STARTTIME].to_i / CLOCK_TICKS)
      name = name[1..-2]
      name += '*' if start_time < DAY_AGO
      vnc_servers << VncServer.new(name, pid, start_time, uid, username, home,
                                   gid, display)
    end
  rescue
    error = true
    puts "Skipped #{ps} because process appears to have exited during run."
  end
end

puts if error
printf "%-10s%-10s%-10s%-30s%-10s%-10s\n", "Name", "Display", "PID",
       "Start Time", "UID", "Username"
printf "%-10s%-10s%-10s%-30s%-10s%-10s\n", "----", "-------", "---",
       "----------", "---", "--------"

vnc_servers.sort { |a, b| a.username <=> b.username }.each do |vnc|
  printf "%-10s%-10s%-10d%-30s%-10s%-10s\n", vnc.name, vnc.display, vnc.pid,
         vnc.start_time, vnc.uid, vnc.username
end

if Process.euid == 0
  puts
  puts "Sleeping for 5 seconds. Press ^C to cancel..."
  puts
  sleep 5

  vnc_servers.sort { |a, b| a.username <=> b.username }.each do |vnc|
    if vnc.start_time < DAY_AGO
      puts "Killing VNC server on display #{vnc.display} for #{vnc.username}:"
      puts
      puts "\tSwitching to user #{vnc.username}."
      # This is likely overkill, but it's fine. Don't forget to change GIDs
      # first while still root of course :-) The reverse in going back doesn't
      # matter.
      Process.gid = vnc.gid
      Process.egid = vnc.gid
      Process.uid = vnc.uid
      Process.euid = vnc.uid
      # This complains, but I don't know why. I do this manually to ensure
      # everything is cleaned up.
      #status = system("vncserver -kill #{vnc.display}")
      Process.kill :TERM, vnc.pid
      log_file = "#{vnc.home}/.vnc/#{HOSTNAME}#{vnc.display}.log"
      pid_file = "#{vnc.home}/.vnc/#{HOSTNAME}#{vnc.display}.pid"
      puts "\tUnlinking #{log_file}."
      File.unlink log_file
      puts "\tUnlinking #{pid_file}."
      File.unlink pid_file
      puts "\tSwitching to user root."
      Process.gid = 0
      Process.egid = 0
      Process.uid = 0
      Process.euid = 0
      puts
    end
  end
else
  puts
  puts "Run as root to kill VNC servers."
  puts
end

That uses Ruby 1.9.2p180, which is installed in /usr/local/bin on our systems. I’m sure I’ll upgrade that soon. It also uses FFI. It’s pretty elaborate for a VNC cleanup program, and it is probably overkill, but I feel it’s a little clearer than using ps output, though that’s completely reasonable. I wanted to use FFI though, and I need to write something that looks for processes that have too much resident memory. Another issue we have is with Eclipse processes requiring huge amounts of virtual memory, but using little resident memory. We really can’t limit RSS use on Linux, and trying to limit VSS kills Eclipse. So, I’ll use Ruby/FFI and /proc to figure out what to kill periodically, or at least just monitor the systems.

One thing I should note about the code above is that it is necessary to switch UIDs (probably not GIDs) to let root delete some files associated with the VNC server in the user’s ~/.vnc directory. We don’t allow root access over NFS for login servers. I probably don’t need to switch the real UID/GID though. I had problems executing the vncserver command as the user. It seemed to work for the most part, but it complained that “Xvnc seems to be deadlocked.” It didn’t clean up the user PID file either. I decided to deal with it manually. Again, this is pretty much overkill, but if it’s useful for someone else on Linux, there it is.

Before I get to the details, I previously figured out how to configure Thunderbird to be the default mail client for everyone. Firefox is the default web browser, but Thunderbird is not the default mail client. I’m including that detail in the solution along with some other default setting changes I’ve done. Additionally, I installed XEmacs by hand. I couldn’t find an RHN RPM, though there is one for GNU Emacs. Not to start an XEmacs/GNU Emacs war, but I don’t see why anyone would prefer GNU Emacs really. However, to be fair, I’ve not used either for a long time. We have a specific need for XEmacs as it was chosen over GNU Emacs more than a decade ago. Personally I use TextMate and MacVim (gvim when on Unix of course). I don’t know why Red Hat requires me to use the RHEL Workstation Optional channel for much of what I need either. Why make it harder for no reason? Anyway, my first task was to create a launcher for XEmacs:

#!/usr/bin/env xdg-open

[Desktop Entry]
Version=1.0
Type=Application
Terminal=false
Icon[en_US]=/usr/share/icons/gnome/scalable/apps/accessories-text-editor.svg
Name[en_US]=XEmacs
Exec=xemacs
Comment[en_US]=XEmacs editor
Name=XEmacs
Comment=XEmacs editor
Icon=/usr/share/icons/gnome/scalable/apps/accessories-text-editor.svg

I created this by hand and pulled it out of my own home directory from ~/.gnome2/panel2.d/default/launchers/xemacs.desktop. I saved it in /usr/share/applications, which had the added benefit of adding it to the Applications –> Other menu automatically (and immediately no less) thus killing two birds with one stone, though I would never do that. Birds are nice. Luckily I found a nice scalable icon that fits the bill as well. I didn’t want to grab yet another file and add it to the system, and I wanted something scalable.

Next I had to figure out how to add this to the Gnome Panel. In case you’re interested, the Gnome Administration Guide is here, but it appears to be a little old. There is a lot of information about how to do this as a user, but not so much on how to set this as a default for all users. I’m not really interested in trying to use Sabayon profiles. That’s overkill for me. I just want to set default GConf values to make it work. Again, I resorted to doing this by hand and looking at the difference between find output to get an idea of where to make the changes in the /etc/gconf/gconf.xml.defaults/%gconf-tree.xml file. I was able to figure out that the changes should be additional directories under /apps/panel/objects. Note that there is an /apps/panel/default_setup/objects path as well, but I didn’t need to touch that. I added two new directories and set the same attributes under each when compared to what I did for my own user account. I logged out and in again, but it didn’t work. I messed with this forever, and I messed up some of the paths by accident when trying the default_setup path instead, so I needed to clean things up before trying what I thought might actually work. It’s always best to start with a clean slate. I used the –recursive-unset gconftool-2 option, but that left the directories. I edited the %gconf-tree.xml by hand to get it back into a pristine state and then tried the solution that worked. I did this on a virtual machine, which I suggest for any changes like this. Gnome is “touchy” IMO, and I don’t want to mess up production login servers. I should also note that settings changed in the %gconf-tree.xml file using the gconftool-2 command might not immediately be reflected in gconf-editor, probably because gconfd-2 is running from the Gnome Display Manager (or something like that is my theory).

When I started making default GConf changes at the beginning of this Linux migration, I tried editing the %gconf-tree.xml file manually. I quickly found that adding new packages and running updates messed up my changes. I don’t know if that’s normal, and I have some personal issues with RHN updates at times (though the benefits of a package manager outweigh these issues – that’s what I keep telling myself), so I started using gconftool-2 to make these changes directly to the %gconf-tree.xml file. My solution to this problem was to create a /usr/local/sbin/gconf-defaults script to set the proper default values:

#!/bin/bash

gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --type list --list-type=string --set /apps/panel/applets/clock/prefs/cities '[<location name="" city="Columbus" timezone="America/New_York" latitude="39.994999" longitude="-82.876389" code="KCMH" current="true"/>]'

gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --type string --set /desktop/gnome/url-handlers/mailto/command 'thunderbird %s'

gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --type bool --set /desktop/gnome/url-handlers/mailto/enabled true

gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --type bool --set /desktop/gnome/url-handlers/mailto/needs_terminal false

gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --type bool --set /apps/gnome-screensaver/idle_activation_enabled false

# Terminal launcher.
gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --type string --set /apps/panel/objects/terminal_launcher/menu_path 'applications:/'
gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --type string --set /apps/panel/objects/terminal_launcher/launcher_location '/usr/share/applications/gnome-terminal.desktop'
gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --type string --set /apps/panel/objects/terminal_launcher/bonobo_iid ''
gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --type string --set /apps/panel/objects/terminal_launcher/custom_icon ''
gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --type bool --set /apps/panel/objects/terminal_launcher/locked false
gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --type bool --set /apps/panel/objects/terminal_launcher/panel_right_stick false
gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --type string --set /apps/panel/objects/terminal_launcher/object_type 'launcher-object'
gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --type bool --set /apps/panel/objects/terminal_launcher/use_custom_icon false
gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --type string --set /apps/panel/objects/terminal_launcher/tooltip ''
gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --type string --set /apps/panel/objects/terminal_launcher/toplevel_id 'top_panel'
gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --type string --set /apps/panel/objects/terminal_launcher/action_type 'lock'
gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --type bool --set /apps/panel/objects/terminal_launcher/use_menu_path false
gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --type int --set /apps/panel/objects/terminal_launcher/position 3
gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --type string --set /apps/panel/objects/terminal_launcher/attached_toplevel_id ''

# XEmacs launcher
gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --type string --set /apps/panel/objects/xemacs_launcher/menu_path 'applications:/'
gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --type string --set /apps/panel/objects/xemacs_launcher/launcher_location '/usr/share/applications/xemacs.desktop'
gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --type string --set /apps/panel/objects/xemacs_launcher/bonobo_iid ''
gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --type string --set /apps/panel/objects/xemacs_launcher/custom_icon ''
gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --type bool --set /apps/panel/objects/xemacs_launcher/locked false
gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --type bool --set /apps/panel/objects/xemacs_launcher/panel_right_stick false
gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --type string --set /apps/panel/objects/xemacs_launcher/object_type 'launcher-object'
gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --type bool --set /apps/panel/objects/xemacs_launcher/use_custom_icon false
gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --type string --set /apps/panel/objects/xemacs_launcher/tooltip ''
gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --type string --set /apps/panel/objects/xemacs_launcher/toplevel_id 'top_panel'
gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --type string --set /apps/panel/objects/xemacs_launcher/action_type 'lock'
gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --type bool --set /apps/panel/objects/xemacs_launcher/use_menu_path false
gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --type int --set /apps/panel/objects/xemacs_launcher/position 4
gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --type string --set /apps/panel/objects/xemacs_launcher/attached_toplevel_id ''

# Add both to the list of objects.
gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --type list --list-type=string --set /apps/panel/general/object_id_list '[menu_bar,web_launcher,email_launcher,terminal_launcher,xemacs_launcher]'

You can see the settings I used to add the two launchers. The key types and values came from doing the same thing by hand. I won’t detail all of those. This is simply about how to accomplish adding the launchers in the first place. I will note the position values of 3 and 4 though. Those were much higher values than the launchers that existed when doing this by hand, but it was clear they could be incremented as integers. I picked those values to make sure they appeared next to the other launchers in the order I desired. The terminal launcher is for an existing launcher, but the XEmacs launcher is for my own xemacs.desktop launcher file. The settings I used will place the two launchers (terminal first, XEmacs second) on the left hand side next to the web browser, mail, and Gnote launchers (technically, Gnote seems to be an applet BTW). Each new launcher gets is own directory under /apps/panel/objects, with attributes within that directory, and the final key to getting it to work was adding those object directory names to the /apps/panel/general/object_id_list list in the last gconftool-2 command. The gconf-defaults command is run in /etc/rc.local every time the machine is rebooted.

There are some additional bonuses in that script as well. I changed the cities for the clock applet to Columbus, Ohio. It defaulted to Boston. I also set the mail handler to Thunderbird so that clicking on the mail launcher would start what we support as a mail client. Lastly, I disabled the screensaver. Our users connect from PCs running X-Win32, and they already have a screensaver. The default was something like 1 minute too. There’s no reason to annoy our users with silly screensavers when they already have one enforced through Group Policy on their PCs.

I had created a /usr/local/sbin/gconf-defaults-show command to see what the settings where for what I had set at any time as well. I added additional gconftool-2 commands to show the launcher related settings:

#!/bin/bash

echo
echo "/apps/panel/applets/clock/prefs/cities --> Columbus, etc.."
gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --get /apps/panel/applets/clock/prefs/cities
echo

echo "/desktop/gnome/url-handlers/mailto/command --> thunderbird %s"
gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --get /desktop/gnome/url-handlers/mailto/command
echo

echo "/desktop/gnome/url-handlers/mailto/enabled --> true"
gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --get /desktop/gnome/url-handlers/mailto/enabled
echo

echo "/desktop/gnome/url-handlers/mailto/needs_terminal --> false"
gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --get /desktop/gnome/url-handlers/mailto/needs_terminal
echo

echo "/apps/gnome-screensaver/idle_activation_enabled --> false"
gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --get /apps/gnome-screensaver/idle_activation_enabled
echo

echo "/apps/panel/objects/terminal_launcher --> Terminal launcher settings..."
gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults -a /apps/panel/objects/terminal_launcher
echo

echo "/apps/panel/objects/xemacs_launcher --> XEmacs launcher settings..."
gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults -a /apps/panel/objects/xemacs_launcher
echo

echo "/apps/panel/general/object_id_list --> [menu_bar,web_launcher,email_launcher,terminal_launcher,xemacs_launcher]"
gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --get /apps/panel/general/object_id_list
echo

The values I set for the new launchers can be seen easily with that command:

[rowland@beta ~]$ sudo gconf-defaults-show

/apps/panel/applets/clock/prefs/cities --> Columbus, etc..
[<location name="" city="Columbus" timezone="America/New_York" latitude="39.994999" longitude="-82.876389" code="KCMH" current="true"/>]

/desktop/gnome/url-handlers/mailto/command --> thunderbird %s
thunderbird %s

/desktop/gnome/url-handlers/mailto/enabled --> true
true

/desktop/gnome/url-handlers/mailto/needs_terminal --> false
false

/apps/gnome-screensaver/idle_activation_enabled --> false
false

/apps/panel/objects/terminal_launcher --> Terminal launcher settings...
 attached_toplevel_id =
 position = 3
 use_menu_path = false
 action_type = lock
 toplevel_id = top_panel
 tooltip =
 use_custom_icon = false
 object_type = launcher-object
 panel_right_stick = false
 locked = false
 custom_icon =
 bonobo_iid =
 launcher_location = /usr/share/applications/gnome-terminal.desktop
 menu_path = applications:/

/apps/panel/objects/xemacs_launcher --> XEmacs launcher settings...
 attached_toplevel_id =
 position = 4
 use_menu_path = false
 action_type = lock
 toplevel_id = top_panel
 tooltip =
 use_custom_icon = false
 object_type = launcher-object
 panel_right_stick = false
 locked = false
 custom_icon =
 bonobo_iid =
 launcher_location = /usr/share/applications/xemacs.desktop
 menu_path = applications:/

/apps/panel/general/object_id_list --> [menu_bar,web_launcher,email_launcher,terminal_launcher,xemacs_launcher]
[menu_bar,web_launcher,email_launcher,terminal_launcher,xemacs_launcher]

Note that I used xml:readwrite:/etc/gconf/gconf.xml.defaults for the config source in the gconf-defaults-show command, even though I only need to read. Changing that to read or readonly doesn’t remove the warning that it’s impossible to write or save changes, even though I am just getting values, so I ran it with sudo.

This seems overly difficult to me. It kind of demonstrates the difficulty in using Linux as a desktop to some degree, at least from the perspective of customizing the desktop for end users. Don’t get me wrong, I’ve been a Linux fan since the beginning, but I use Mac OS X as my desktop. It should be a little easier to set GConf defaults IMO. You can use the gconf-editor, but I seem to have problems with settings getting blown away. Figuring out what to set with gconftool-2 basically requires using the gconf-editor program though. Rolling Sabayon profiles is not the solution to this specific problem either. I think that’s obvious, and it would be serious overkill. I’m not an expert on customizing Gnome, and this is the first time I’ve had to do this seriously. That being the case, this is a viable solution to the problems of setting GConf defaults and also adding launchers to the Gnome Panel.

[rowland@rowland-mba Bonnie-64]$ ./Bonnie -s 8192
File './Bonnie.399', size: 8589934592
Writing with putc()...done
Rewriting...done
Writing intelligently...done
Reading with getc()...done
Reading intelligently...done
Seeker 1...Seeker 2...Seeker 3...start 'em...done...done...done...
              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    GB M/sec %CPU M/sec %CPU M/sec %CPU M/sec %CPU M/sec %CPU  /sec %CPU
            8  80.4 86.5 198.8 18.8  98.2 11.6  92.2 82.8 225.2 12.4  7028 31.2

Everything seems fast. I decided that I would keep my non-SSD 27″ iMac (it’s awesome otherwise) and treat this as a “real” laptop. My early-2008 MacBook Pro works fine, but it’s very bulky. It was a great desktop replacement, but I decided about a year ago to get an iMac. I was tired of carrying around the MacBook Pro, and this machine is actually faster all around.

I’ve dealt well with the lack of a DVD drive. I don’t think I really need one, and as far as other peripherals, I am sure the Thunderbolt connector will come in handy at some point. I don’t know about this whole 1440×900 resolution though, but it is a 13″ LCD, so I can mostly live with it. The LCD is not quite as good as my old MacBook Pro, but it’s still very good. I’m happy so far. That’s saying something. I’m picky about computing equipment.

NoMethodError

undefined method `init' for Mysql:Class

I saw this when I was considering moving all of this to Mac OS X Lion Server too, but since I went with Rackspace instead for now, I didn’t bother solving it immediately. But now I have to solve it in order to use the latest MySQL. I did some searching, and on Unix you can use LD_LIBRARY_PATH to solve this. On Mac OS X the equivalent is DYLD_LIBRARY_PATH, and setting that to /usr/local/mysql/lib does solve the problem. I don’t like that solution though. You can do some real magic with Mach-O binary files on Mac OS X, so I decided to wield some.

The problem turns out to be how the mysql_api.bundle file is linked. This is shown with the otool command:

otool -L ~/.rvm/gems/ruby-1.9.2-p180/gems/mysql-2.8.1/lib/mysql_api.bundle
/Users/rowland/.rvm/gems/ruby-1.9.2-p180/gems/mysql-2.8.1/lib/mysql_api.bundle:
	/Users/rowland/.rvm/rubies/ruby-1.9.2-p180/lib/libruby.1.9.1.dylib (compatibility version 1.9.1, current version 1.9.1)
	libmysqlclient.18.dylib (compatibility version 18.0.0, current version 18.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.0.0)
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)

The libmysqlclient.18.dylib file was linked, but there is no path to it. I tried adding an rpath using the install_name_tool command like this:

install_name_tool -add_rpath /usr/local/mysql/lib ~/.rvm/gems/ruby-1.9.2-p180/gems/mysql-2.8.1/lib/mysql_api.bundle

and that does add an LC_RPATH command to the end:

otool -l ~/.rvm/gems/ruby-1.9.2-p180/gems/mysql-2.8.1/lib/mysql_api.bundle
<snip>
Load command 13
          cmd LC_RPATH
      cmdsize 40
         path /usr/local/mysql/lib (offset 12)

But that does not solve the problem. My theory is because it comes after the library itself in the file “instructions” shown with otool. I solved this another way using install_name_tool to change the actual library linking directly:

install_name_tool -change libmysqlclient.18.dylib /usr/local/mysql/lib/libmysqlclient.18.dylib ~/.rvm/gems/ruby-1.9.2-p180/gems/mysql-2.8.1/lib/mysql_api.bundle

Now otool shows the full path to the libmysqlclient.18.dylib library and everything works fine:

otool -L ~/.rvm/gems/ruby-1.9.2-p180/gems/mysql-2.8.1/lib/mysql_api.bundle/Users/rowland/.rvm/gems/ruby-1.9.2-p180/gems/mysql-2.8.1/lib/mysql_api.bundle:
	/Users/rowland/.rvm/rubies/ruby-1.9.2-p180/lib/libruby.1.9.1.dylib (compatibility version 1.9.1, current version 1.9.1)
	/usr/local/mysql/lib/libmysqlclient.18.dylib (compatibility version 18.0.0, current version 18.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.0.0)
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)

Note that the path to the mysql_api.bundle file for the system Ruby on Mac OS X Lion is /Library/Ruby/Gems/1.8/gems/mysql-2.8.1/lib/mysql_api.bundle. I love that you can do stuff like this in Mac OS X. I’ve done it before in Xcode with my BroMonitor project in setting @executable_path.

I found that my Rackspace Cloud Server performed much better with disk I/O than my Slicehost VPS though they were otherwise identical. A simple Bonnie test shows this, though there are better tests. The first is my Slicehost VPS:

[root@test:~/Bonnie-64]# ./Bonnie -s 2048
File './Bonnie.11378', size: 2147483648
Writing with putc()...done
Rewriting...done
Writing intelligently...done
Reading with getc()...done
Reading intelligently...done
Seeker 1...Seeker 2...Seeker 3...start 'em...done...done...done...
              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    GB M/sec %CPU M/sec %CPU M/sec %CPU M/sec %CPU M/sec %CPU  /sec %CPU
            2  39.5 96.3 100.2 29.9  37.8 10.1  33.6 64.1  70.7  9.7   289  1.7

The second is my Rackspace Cloud Server:

root@web:~/Bonnie-64# ./Bonnie -s 2048
File './Bonnie.10400', size: 2147483648
Writing with putc()...done
Rewriting...done
Writing intelligently...done
Reading with getc()...done
Reading intelligently...done
Seeker 1...Seeker 2...Seeker 3...start 'em...done...done...done...
              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    GB M/sec %CPU M/sec %CPU M/sec %CPU M/sec %CPU M/sec %CPU  /sec %CPU
            2  60.0 100.0 284.2 71.5  86.5 17.4  64.4 89.9 249.8 24.6   208  0.9

I’m very happy with the move so far.

Note the improvised support for the heavy 27″ iMac and the Tachikoma (eveyrone should have a Tachikoma). This should be interesting. My feet hurt already, but I can be dedicated to an idea. Now, if I only had a solution for work.

When logging in there is an initial delay of around 15 to 20 seconds due to the fact that the system cannot really talk to the Active Directory servers. This is tolerable though. The prompt to synchronize the home directory does appear if you have the Mobile Account Preferences configured that way, but you can cancel it. After that it took even longer for the Dock and desktop to appear, and this was the biggest problem. I solved this by making a copy of the ActiveDirectory.plist file:

sudo cp /Library/Preferences/DirectoryService/ActiveDirectory.plist /Library/Preferences/DirectoryService/ActiveDirectory.plist.orig

and then changing the value of the “AD Mount Home As Share Point” key to false (this is around line 31 or so):

<key>AD Mount Home As Share Point</key>
<false/>

A number of those settings near the top come from the GUI configuration. This setting does not, and it does not change the values for the UNC home directory path or SMB protocol I mentioned previously, but this does keep the system from mounting the home directory as a share on the Dock and desktop. This avoids the biggest part of the log in delay. Once that’s edited, you can restart the DirectoryService process by simply killing it:

sudo killall DirectoryService

The launchd process will happily restart it. It is still possible to connect to a home directory share in finder and possibly put something on the Dock for that connection, but this keeps the system from trying to mount a share it cannot when off network. It would be nice if the system had the ability to define network locations in such a way as to not attempt Active Directory authentication or home drive mapping when off of the production network, but I’ve not found that kind of setting yet. I’m still looking though. We’ll see how Mac OS X Lion deals with this. At least FileVault should be nicer there since it will do full disk encryption, but I have some questions about how that will work of course.

One of our staff members was testing my RHEL 6 build and noticed that moving the mouse on menu items was very slow. It took way too long for the menus to drop down in Firefox for example. Switching between virtual desktops was agony as well. I had not noticed this when doing a remote X11 query from my Mac, though that has its own problems that I will touch on later. I confirmed that this also happened on my Windows PC, though for some reason I had not noticed it before. This might have been the result of a software update, but it’s likely I just had not noticed since I work from my Mac most of the time. Since I had recalled the previous issue, I had a theory that the Linux server was trying to send something over TCP to the Windows PC. I fired up tcpdump and captured some packets, but I then also recalled how hard it was to dig this specific situation out of all of the traffic X11 generates. I moved my Windows PC into a special Active Directory OU that has the Windows firewall disabled in Group Policy. This made it very clear what traffic was being rejected:

No.     Time        Source                Destination           Protocol Info
    251 2.314295    164.107.112.68        164.107.120.111       TCP      59337 > 4713 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 SACK_PERM=1 TSV=1031870027 TSER=0

...

No.     Time        Source                Destination           Protocol Info
    252 2.314412    164.107.120.111       164.107.112.68        TCP      4713 > 59337 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0

Getting that reset when trying to connect to port 4713 on the Windows PC is what makes things work “normally” (not slow in other words). The Windows firewall only drops packets as far as I know, and that’s what makes it slow. This only affected logging out in RHEL 5, but in RHEL 6 it affects much more apparently. My first solution was to add the following iptables firewall rule on the Linux server to cause the same effect as a Windows PC with no firewall:

# This keeps PulseAudio from contacting the remote client. With the our PC
# client firewalls, the attempt is dropped instead of rejected, so it ends
# up slowing everything down in the UI. This will immediately reject with
# a TCP reset and make things "normal" again.
-A OUTPUT -p tcp --dport 4713 -j REJECT --reject-with tcp-reset

As you can see from the comment, this is because of PulseAudio. I ripped most of PulseAudio out, but I couldn’t remove the libraries due to RPM dependencies. I like this more than the Windows firewall being open on that port because it keeps the traffic from going out on the network in the first place. I then remembered how I ended up solving this problem on RHEL 5 without using a firewall rule by adding the following to /etc/X11/xinit/xinitrc-common:

# Direct Enlightenment Sound Daemon traffic to the bit bucket (port 9) of
# the localhost. Nothing is listening there, so this will cause ESD traffic
# to immediately fail. The firewall can also be used, but this seems to
# introduce a 3 second delay (even when tcp-reset is used). We picked port
# 9 because that was not used and it is reserved, so no regular users would
# end up listening there. This fixes the logout hang from XDMCP sessions.
#
#                                               --rowland
export ESPEAKER="127.0.0.1:9"

I did not have the delay I mentioned in the comments on RHEL 6, but doing something like the above is preferable to the firewall rule in my opinion. It turns out you can do the same thing with PulseAudio by setting the PULSE_SERVER environment variable or adding the following to /etc/pulse/client.conf:

default-server = 127.0.0.0:9

That’s the solution I went with instead of the firewall rule, almost like last time with RHEL 5. If you have slow X11 queries to your RHEL 6 server, this might be the problem.

As far as X11 queries from a Mac goes, I had major issues logging in through GDM. I could type my username, but when I went to type my password the keyboard mappings were totally messed up from that point forward. Nothing I did made a difference. Apparently this has to do with the X Keyboard Extension. There is a -kb command line option to disable that, but it causes the X server to crash on a Mac. I had no luck in getting this to work with the newer version of XQuartz either. The only way I could do an X11 query was to use the nested X server like so:

Xnest -kb -fp tcp/<hostname>:7100 -query <hostname>

The “<hostname>” value should be replaced with the actual hostname of course. That actually works with the -kb option, and it only crashes every once in a while :-) So, it’s not perfect, but there are few Mac users in our environment, and they already have Unix with a much better user interface anyway. X11 – fun times indeed.