Unflushed Buffers

This release adds a simple event log for the log viewer, and tools for user management (add, modify, delete), as well as cleaning up some of the HTML.

I have held back adding features for annotating, bookmarking and sign-off until the next version.

Visit the project page or SVN repository.

While adding a login/logout feature to the logview1 project, I found the documentation a little sketchy. Fortunately Python is very easy to figure out, so I wrote a basic mechanism. You can find the code here in the SVN repository.

So when you try the main handler, http://blahblah/basedir/sessions_login.py/query it will redirect to a login screen until you are authenticated. Apart from that it’s a simple hit counter. It checks login name and password against a database table and sets session information.

It’s only been one day since I published the site on the web, but today I added some more of the features on the TODO list, so the script is more useful for searching logs.

For now I’m calling it Logview1… it’s a viewer, that views logs, and it’s number one!

Visit the project page, where I’ve posted the source and some screenshots.

As an exercise in learning Python, I decided a good simple practical application would be a tool to monitor system logs of various systems at once logging to an SQL database. This has already been quite helpful as I debug my OpenLDAP migration and taking a closer look at Solaris.

Python is a deliciously simple language and I had no trouble learning very quickly to rapidly develop a simple database driven application. My first app was quite useful to me, and I intend to add more features. A few years ago a friend and myself built a similar application in PHP but it has been lost over time.

I’ve released it on this website with its own project page, and posted an early alpha release if anyone cares to take a look.

Solaris got me re-interested in LDAP since Sun likes directory services, but I’m not using Sun’s LDAP server. Reading up on the documentation I was told this sets it all up:

ldapclient manual -v -a defaultsearchbase=dc=scriptforge,dc=org -a domainname=scriptforge.org IP_ADDRESS_OF_LDAP_SERVER

This will take a minute or so to run as services are stopped, changes made and services restarted. The output shows what it is doing and what files it is changing. Take a look at some of these config changes. One thing I found is that this process changes hostname lookup from DNS to LDAP which is what I didn’t want. This was changed by editing /etc/nsswitch.conf hosts line as follows:

hosts: dns files

Now we can get Solaris to query the LDAP passwd tree with:

getent passwd USERNAME

getent group GROUPNAME

Even finger works, but login does not (try SSH into the box with an LDAP username, it won’t work). This is because PAM hasn’t been configured to authenticate by LDAP. Edit /etc/pam.conf and there are different services listed (login, rlogin, ppp, other, passwd, and some kerberos ones) We need to change auth required pam_unix_auth.so.1 lines to auth sufficient pam_unix_auth.so.1 and then under the pam_dial_auth.so.1 lines we add this line:

login auth required pam_ldap.so.1

Add these lines for the different services as you see them in the file (ie change the login above to rlogin, rsh, ppp, other, passwd as needed).

STUCK! Damn this didn’t work! Anyone? Oh well may go get the samba integration going…

Setting up automounting

{not yet written}

Automounts and LDAP

{not yet written}

Now we have the directory running on a server and some accounts migrated, time to use it. There is a caveat before you start on Ubuntu. I advise setting a root password so you can log in as root rather than sudo, as I lost the ability to do so after I removed local accounts and my directory login was not in the sudoers file. When it’s running and you have your accounts set right you can go back to using sudo.

In the list of users and groups in your directory, perhaps you should change the uids and gids so that they are unique across your systems - for example start at 5000 instead of 1000.

Install the following package:

apt-get install ldap-auth-client

This will install the packages necessary to switch to LDAP authentication. Dpkg will ask you questions as to which LDAP server you will authenticate to, and ask for admin credentials so root can change passwords on the server. These responses are stored in /etc/ldap.conf but I reckon it’s easier to let debconf manage this config. If you need to change your answers, run this:

dpkg-reconfigure ldap-auth-config

Now to switch PAM (pluggable authentication modules) over to LDAP. This is done by:

auth-client-config -p lac_ldap -a

Now if you take a look in the /etc/pam.d/common-* files you will see entries like this:

auth sufficient pam_ldap.so

And if you try your login now (eg by ssh to the server) you will probably be able to login using LDAP authentication. Next step is to make sure all these LDAP users get their home directory from the server. This is where automount comes into play. Up to now I’ve just exported the entire /home on the file server by NFS, but that shows everyone’s home directory on servers that may not need it.

Setting up automounting

Need to install the autofs package (will install nfs-common for mounting NFS filesystems, if needed):

apt-get install autofs nfs-common

Then we edit the /etc/auto.master and add the line:

/home /etc/auto.home –timeout=90

Then create the file /etc/auto.home with the line:

* –fstype=nfs4,rw,proto=tcp,port=2049 NFS_SERVER:/home

As you can see, I’m using NFSv4, so on the server I have /export/home in the pseudo-filesystem bound to my real filesystem /home. On both client and server the file /etc/default/nfs-common needs to have NEED_IDMAPD=yes and the file /etc/idmapd.conf needs to have the same Domain = line.

Best to restart everything.

SERVER: /etc/init.d/nfs-common restart
SERVER: /etc/init.d/nfs-kernel-server restart
CLIENT: /etc/init.d/nfs-common restart
CLIENT: /etc/init.d/autofs start

Now try to ssh userinldap@localhost and you should be able to log in, and autofs will mount the home directory.

Automounts and LDAP

On the OpenLDAP server and Linux clients, both of which are Ubuntu, install the autofs-ldap package:

apt-get install autofs-ldap

On the LDAP server, edit /etc/ldap/slapd.conf and add this line after the core.schema and cosine.schema lines:

include /etc/ldap/schema/autofs.schema

Restart the directory with /etc/init.d/slapd restart. Now we need to add automount information into the LDAP tree. Here’s a good LDIF file to start:

dn: ou=auto.master, dc=scriptforge,dc=org
ou: auto.master
objectClass: top
objectClass: automountMap

dn: cn=faulteh,ou=auto.home, dc=scriptforge,dc=org
objectClass: automount
automountInformation: -fstype=nfs4,rw,proto=tcp,port=2049,hard,intr serenity:/home/faulteh
cn: faulteh

dn: cn=/home,ou=auto.master, dc=scriptforge,dc=org
objectClass: automount
automountInformation: ldap:serenity:ou=auto.home,dc=scriptforge,dc=org –timeout 90
cn: /home

dn: ou=auto.home, dc=scriptforge,dc=org
ou: auto.home
objectClass: top
objectClass: organizationalUnit

Open the directory viewer in a web browser and see what this LDIF has done. Now we need the client to get this information instead of using the /etc/auto.* files.

Edit the file /etc/nsswitch.conf to add the line:

automount: ldap

Edit the file /etc/ldap/ldap.conf to change the BASE and URI to suit your server. Restarting the autofs with /etc/init.d/autofs restart and logging in with ssh everything should be working.

On to the Solaris clients

OK apparently we have an issue on a lot of *nix systems like Linux and BSD that use the time_t structure which is like 32 bits to count time since 1970. Similar to Y2K, except when 2038 comes around it’s going to be cool 1970 again rather than a crappy 1901 desparation we faced back in 2000.

However, time will run out before 2038. Apparently there’s this bloody huge rock called Apophis that’s due to do a very close flyby of the earth in 2029. Probably won’t hit us, but the gravitational shift of flying that close will slingshot it back around and NASA estimated a 1 in 45000 chance it will hit in 2036. Woop di do you say, we’ll still have to fix this Y2K38 unix problem.

Nuh uh! Some 13 year old German kid doing a school project took a look at NASA’s numbers, and decided that if one of the satellites orbiting earth struck Apophis during it’s 2029 flyby, even though a smaller object, but travelling at 3km/s would cause the big rock to change trajectory. He concluded the brains at NASA got it wrong and there’s really a 1 in 450 chance we need a new planet. Woah!

So forget about the time_t issue, we need to put Bruce Willis on ice now so we can blow this space pebble up in time to then fix a possible time warp to the ’70s.

Installation (Ubuntu)

Forget about doing all the dirty work, others have done it for you. I’m using Ubuntu Linux (gutsy), and there are packages to help. I’m installing the directory on the same server as my main file server (NFS and Samba), but it can be moved to a standalone system if I need to scale up. This server also runs apache2 and php5, and since I will use a web app to admin the directory these are required as well. Installing the packages below will also install apache+php if needed:

apt-get install ldapscripts ldap-utils migrationtools slapd phpldapadmin

Answer the questions it gives you as best you can - you can always reconfigure with

dpkg-reconfigure slapd

Now you’re pretty much done. The questions dpkg asks set up a superuser account on the directory in the form of

cn=admin,dc=scriptforge,dc=org

Now before we go further, take a look at /etc/ldap/slapd.conf and get an idea of what options you can set here and what is set in the directory.

Next take a look and make sure slapd and apache are running with a bit of ps ax action. Apache is for the phpldapadmin package to help you manage the directory.

Open a browser to http://servername.or.ip.address/phpldapadmin and login with the superuser account above (modified for your own domain) and password you set during the dpkg config process. Take a look around at an empty directory.
Migration of the Local Accounts

See the migrationtools package installed earlier - these provide a series of scripts to get you started with the migration. This way you hardly deal with those silly LDIF files and minimum fuss on the command line. They take stuff from your /etc and put it in the directory (like /etc/passwd, /etc/group).

These scripts are in /usr/share/migrationtools so cd to that directory and get ready to run some commands. First you need to edit migrate_common.ph and look for $DEFAULT_MAIL_DOMAIN and $DEFAULT_BASE which you will need to change.

Then we run the script

./migrate_all_online.sh

It will ask you questions as to what your server name / IP address is, and user/pass (called credentials) to log in with. It also asks about if you want some sort of DUAConfigProfile created - choose no here.

Running the migration will take mere seconds but may come up with errors. I found it did this because dpkg or the package itself was created with some default tree and it was trying to recreate nodes that already existed. If this is the case, go to your browser and that phpldapadmin web app and delete each entry in the top of the tree, except the one about the admin, and then restart the migration script again but with an empty tree. I also had a problem migrating /etc/aliases but since this isn’t my mail server I simply deleted that file as it’s not used on the server, deleted the tree nodes so it was empty and tried again.

After the migration is complete, look through the tree in the browser and see how the migration tools organised your data.

You should see all your user accounts in ou=People, groups are in ou=Group, and so on. Wow, almost painless. There are some command line tools for browsing the tree, ldapsearch being one of those. Give it a try and search the directory.

So we have a directory server, and some accounts in there. It’s probably not secure, or organised, and ALL your groups and accounts are in there. Not every server needs bin, sys, daemon, news, mail users/groups but this is where having a list of what you want migrated comes in handy. You can start pruning the nodes you don’t want to replicate across the network.

Next we try to get a Linux desktop client to start authenticating to the directory…

This is only a small network with a handful of users, most of them remote. But the network is a large variety of systems and services which up to now has been been managed haphazardly, with user accounts created locally on each service. Time for a change. I’d managed an NIS deployment years back but it is fraught with security holes. Since I’ve been hacking away at Solaris for a week, now is the time to give LDAP another thought.

Previously I’ve thought LDAP was too complex for it’s own good. Especially when you can create simple web based authentication schemes with a couple of SQL tables, the LDAP protocol and it’s abstract object based nature with it’s trees and schemas. Especially when /etc/passwd and /etc/groups have been my friend for so many years. Don’t get me wrong, I still think it’s a bit too pretentious for it’s own good but this time I took a step by step approach to understanding it through practical application, and multiple iterations.

I didn’t get it right the first time, and it’s still not finished. But instead of trying to get LDAP to be a silver bullet, load up on smaller caliber weapons and keep firing until you tame or kill the beast.

This is a multi-part series that goes over the iterations I’ve made and/or plan to make.

• Part One - Make the TODO list
• Part Two - Deploy OpenLDAP and migrate local accounts (Ubuntu Linux)
• Part Three - Get Linux systems migrated to LDAP (Ubuntu Linux)
• Part Four - Migrate Solaris systems
• Part Five - Migrate some web services
• Part Six - Bitchslap some security into LDAP and your network
• Part Seven - Throw it out and start again
• Part Eight - Migrate more services (what else can we do?)

Any lark who stumbles across Part Seven may gasp but this network is primarily R&D not production. Anyone planning an LDAP deployment needs to document some things and test your theories in a lab/research environment before finalising deployment to production scenarios.

TODO List

Here is what I wanted to achieve both short and long term out of this migration.

1. A single login and group structure for Linux servers and desktops
2. Authenticate other Unix based systems (Solaris and BSD) to the directory
3. Automount home directories via NFS
4. Replicate a subset of the directory to a DMZ directory for the servers on that network (also for remote/extranets)
5. Harden up the LDAP deployment with some extra security layers (eg add SSL)
6. Move some web services to LDAP authentication mechanisms
7. Do the same for Email accounts
8. Convert Samba file sharing / windows login to directory
9. See what directory services can do to enhance mobile devices (roaming laptops, PDAs and mobiles, VoIP phones)
10. What other services can the directory enhance (DNS?)

So far I’m stuck in the middle of this process but plan to document how I tackled each item on the list.

Before starting, I made a list of the user accounts, groups and systems that were in the initial list to be migrated. This is not a definitive list and how you migrate each will change by the time you figure out the best way to structure the directory. Get it working, then get it working right, and you will come up with some novel ways to handle you data as you proceed.

You will need it when you realise, that you SUCK.

Took a break and tried my hand at Frets on Fire .. an open source version of those Guitar Hero games. Instead of the fake guitar thing, you play with your keyboard, sideways.

Yes, I SUCK. I don’t play a lot of games but it was a fun diversion. You can even get more songs from fan sites such as FretsonFire.net