))
Theory Cluster: theoc
How to perform better?
First problem: Kerberos and AFS
- Update on
krenewandTMUX
DESY employs Kerberos as primary authentication and validation method for its single sign-on system, which in addition uses the Andrew file-system (AFS) for providing home directories.1
The details of Kerberos are by itself interesting, however, they are out of scope for this article and I will only cover the terminology in a very vague way.2
The round-up is that the user authenticate against a single (or more) Key Distribution Center (KDC), where the Authentication Server (AS) authenticates using the user's password and grants the user a ticket, which is then stored locally, i.e. in /tmp/krb5cc_<uid>_<random>.
The granted Kerberos tickets allows us to authenticate against the login manager (using PAM) or directly with the SSH server, if the server and client both support the GSS-API.
In addition, the Kerberos ticket is also used to access the AFS, where our home directory resides, by obtaining a AFS token.
As Kerberos promises security in an insecure environment, the Kerberos ticket implements some restrictions to its application duration: It has a limited lifetime and a limited renewal lifetime.
After its lifetime is up, the ticket is automatically invalid.3
However, if the cumulative runtime of our lease is less than the renewal lifetime, we can renew our ticket grant and get a new ticket.
We renew our (current) Kerberos ticket either manually with kinit -R, or with krenew as background daemon.
The latter allows us, however, to automate the renewal process up to the renewal lifetime of the ticket grant.
As AFS is tightly connected to Kerberos, i.e. when we access AFS, we need a AFS token, which is typically granted using our Kerberos ticket when we log in, we also need to renew the AFS token.
For example, as soon as our Kerberos ticket's lifetime is up, the token is also invalid as it requires a valid Kerberos ticket.
Therefore, after a ticket renewal, we need to obtain a new AFS token with aklog.
But, before I give you the example on krenew, let me point out:
klist- gives an overview over the existing (user) tickets
tokens- gives an overview over the AFS (user) tokens
Be sure to regularly check your tickets and tokens, in case, something goes wrong - especially before your computation jobs crash.
For the examples, we need to install the kstart package, without privilege escalation, hence, we rely on a manual installation.
After the installation, we can go straight to the example section of the man-page of krenew:
krenew -a -K 10 -t &
krenew wakes up after 10 minutes (-K 10) to check if the ticket (cache) would need a renewal.
However, we always want to ensure that the ticket and token are valid, thus, each time krenew wakes up, it should renew ticket and token, which we accomplish with -a.
The option -t specifies to run aklog.
In principle, to run jobs without the need to be constantly logged in, we would start a TMUX session and then run krenew as our first program in the background.
Our ticket and token would then be renewed up to (currently) X days - check X for yourself, I do not want to expose this number to the public.
But, that is not sufficient, after logging out the current ticket cache will be deleted.
However, krenew relies on this specific ticket cache as we do not let krenew run a program.
We can circumvent this issue by creating a separate ticket cache for TMUX.
export KRB5CCNAME="FILE:/tmp/krb5cc_$(id -u)_${RANDOM}" kinit && aklog
Unfortunately, we need to do this for each new pane or window, alternatively, we can set in each pane or window the KRB5CCNAME variable to our new ticket.
A more minimal approach is to use nohup.
Normally, nohup would just straightly fail Kerberos and AFS after logging out - the Kerberos ticket and the AFS token are destroyed when logging out.
However, we can sandwich krenew into nohup:4
krenew -b -a -K 10 -t -- nohup <your command>
krenew duplicates the ticket cache,5 and keeps ticket and token alive (each wake-up renews them) and nohup keeps your program running.
Remark: If your computation takes longer than the default renewal lifetime, you can initialize a new ticket with a longer renewal lifetime (up to the maximum set by the KDC administrator) before running krenew:
kinit -r<X>d && aklog
But, in case your computation really takes some time, you should consider using a batch-system based cluster.
Installation of krenew
IMPORTANT NOTE: Do not just trust download links and my build process, especially when it comes to security software. Always question it and check the integrity of the links and of the downloads.
krenew is part of the software package kstart written by Russ Allbery.6
Before, we start with the download and build process, let me remind you, how to retrieve a public key from a public key server, which we then use to verify the software tarball with associated signature file.
- Run
gpg --keyserver keyring.debian.org --recv-keys 7D80315C5736DE75to download the public key of"Russ Allbery <[email protected]>". - Verify the content of the key, i.e. the output of
gpg.
Next, we download the source code directly from the author's webpage, verify it and handle the build process:
aria2c http://archives.eyrie.org/software/kerberos/kstart-4.2.tar.xz \ http://archives.eyrie.org/software/kerberos/kstart-4.2.tar.xz.asc gpg --verify kstart-4.2.tar.xz.asc kstart-4.2.tar.xz tar xvf kstart-4.2.tar.xz pushd kstart-4.2 ./configure --prefix=$HOME/local --enable-setpag ## --enable-setpag required make make check make install
Footnotes:
Personal opinion: If there would be an alternative to AFS… squint at NFSv4. Of course, the answer is not that simple and there are reasons for DESY not to (directly?) hop onto the NFSv4 train.
I strongly recommend the video on Kerberos by the Computerphile producers.
Side note by former Kerberos-Administrator: Keep the clocks of your servers and clients updated to the same time, else the different clocks mess up Kerberos.
When krenew runs a command it make a private copy of the ticket cache for the program.