I pretty exhaustively described the step-by-step instructions necessary to get Grid Engine 2011.11 to build from source on CentOS 6.2 or RHEL 6.2. This method builds almost all of SGE without any of the “disable just about everything” shortcuts.

Short link:
http://biote.am/6y

Long link:
http://bioteam.net/2012/01/building-open-grid-scheduler-on-centos-rhel-6-2/

Feedback welcome.

My main concern is that I want to keep the OS up to date. We are
currently on RHEL 5.3 and at very least I want to update it to the
latest in the 5 stream. So if I can update the OS without worry of
breaking the current Grid Engine install, I will be happy.
RHEL 5.7 is mostly binary compatible with 5.3, the missing things are
the bug fixes.

SGE does not use secret system calls or hooks to get information from
the kernel. SGE parses the /proc filesystem to fetch most of the
low-level system information on Linux (on AIX it is a different story,
but GE 2011.11 fixed this problem as well on AIX), and if your SGE
6.2u5 installation is working on your machines, then it will (or may
be I should say “should”) continue to work after the OS upgrade.

A few things that can break SGE: Linux 3.0 (the kernel version string
difference breaks an SGE shell script), AMD Opteron 6100 series
(Magny-Cours) or 6200 series (Bulldozer/Interlagos …) – if you use
core binding, or memory expansion.

Note that all those issues are fixed in later versions of SGE, but if
you are not doing any of the above upgrades, then SGE 6.2u5 should run
happily.

And there is no special variable for the project?

Aha, you are referring to the project which was assigned already to this job at submission time. Sorry, got it in the wrong way.

There is no direct variable, but you can check:

$ PRJ=$(grep "^acct_project=" $SGE_JOB_SPOOL_DIR/config)


$ echo ${PRJ#*=}

inside the epilog.

I defined an exclusive tag in complex resources for users that can request “-l excl=1 ”
Such users, lock free slots of hosts that have added excl=true to consumable resources.

1) In this case, how can I get accounting for all of slots that a user cause to lock on hosts?

it’s not recorded in the accounting file where the job ran, i.e. a “pe_hostfile” like record. As a result also the blocked slots are not recorded. What you can try if you need it: use a global (or queue) prolog, which runs under e.g. the sgeadmin account by prefixing it with the name (see `man queue_conf`, resp. `man sge_conf`) and copy the “pe_hostfile” with the information of the distribution to a central place. Whether the exclusive access was requested, is recorded already in the accounting file (like all -l resource requests).

With this information, you can compute the real blocked cores then.

– Reuti

Is there a way to assign node to multiple queues but not make the
node over subscribed? Are there any tips or reference site with the
information?



As Reuti said, one option is an RQS (resource quota set). I stole this
one from somewhere (my cluster has hosts with differing numbers of CPU
cores):

{
    name         limit_slots_to_cores_rqs
    description  Prevents core oversubscription across queues.
    enabled      TRUE
    limit        hosts {*} to slots=$num_proc
}

Getting share tree usage from the command line?

Short answer is that a “sge_share_mon” binary actually exists. I plan on checking this out, it would be nice if it had an XML output option as well.

• EnginFrame
• sge4vfx

We’ve purchased a license for Gaussian with support for parallelism
via Linda. A quick google doesn’t show up any tight integrations for
Linda/SGE.
i)Does anyone have a working tight integration config for Linda? Or
even a Gaussian specific one?

it was several times on the list. You can use the one for G03-D.01:

http://markmail.org/message/sgg3dhngpx75i45u

The name you specify in linda8.2/opteron-linux/bin/linda_rsh to replace the full path is just a name. It could even be foobar and you create a link called foobar by start_proc_args in $TMPDIR. The final communication method is the one defined in SGE’s qrsh_startup/qrsh_daemon. It’s just rsh to make it short.

– Reuti

The move to wordpress will stop the ruby and RAILS-related memory leaks that were causing this site to fall over all the time. I’m keeping a careful eye on the error logs so I can find and catch any permalink, RSS or XML related errors. Hopefully with enough work I can put in rewrite or redirect rules so that any link that worked on the “old” gridengine.info will continue to work. If you notice anything odd or want to highlight something that is broken please feel free to drop me a line at ‘dag@sonsorol.org’.

The entire point of this post is to show a modern, free (non Oracle) version of Grid Engine running on a 64bit Mac OS X system running OS X 10.6.7…

The screencaps are from “Grid Engine 8.0.0alpha” taken from the Univa-controlled fork hosted over at github.

This is a side effect of my efforts to get closer to the SGE source and get back into the swing of being able to (relatively) easily build binaries from the source tree when needed.

Eventually my personal “courtesy binaries” for Grid Engine will all end up here:

http://biote.am/sge

Realistically speaking I think BioTeam will try to maintain binaries for 32-bit and 64-bit versions of Linux as well as 64-bit versions for the shrinking world of Mac OS X based cluster systems running Grid Engine.

Running Grid Engine on a Mac is pretty much a losing proposition in 2011 now that Apple has clearly made the switch to becoming a consumer electronics company. As an enterprise operating system, server or compute node it’s a choice now only for legacy systems and perhaps engineering nerds like myself who use Mac laptops as workstations and like to have our “tools” running locally…

This particular OS X build actually went pretty smoothly, I managed to get all components to compile including the X11 ‘qmon’ GUI and the Java-based GUI installer. Had to build with “-no-secure” though which removes the SSL-based “CSP Secure Mode” installation option. I’ll keep trying to get the integrated SSL stuff to work but it’s not a dealbreaker — in many years I think I’ve only found one or two sites that actually operate SGE with certificate-enabled CSP mode.

OS X binaries for Grid Engine have not been uploaded to the
http://biote.am/sge site yet as I want to test on a different Mac then the build host just to make sure nothing will bomb out due to missing dependencies.

All the usual problems of running SGE on 10.6+ based systems apply though. I had to use the method described in my 2010 blog post to actually get the daemons running.