Building from source on Mandriva 2006
Andre has posted a basic overview showing Grid Engine built from source code on a Mandriva 2006 system running under VMware.
Building from source on Mandriva 2006
Andre has posted a basic overview showing Grid Engine built from source code on a Mandriva 2006 system running under VMware.
Courtesy binaries and RedHat Enterprise Linux 5 (RHEL5)
RHEL5 is out and Göran recently passed this heads-up warning to the developers mailing list:
It might be worth mentioning that this won’t work any more for people using the i386 variant of RHEL5. The pre-compiled i386 qmon uses libXm.so.2 (Openmotif 2.1) which is no longer available in RHEL5.
It is not a problem for x86_64 RHEL5. The pre-compiled x86_64 qmon uses libXm.so.3 (Openmotif 2.2) which is still available in RHEL5 as a compatibility package. (Openmotif 2.3 is the default.)
For i386 arch users of RHEL it looks like getting the ‘qmon’ GUI to work may require building from source or installing a third-party OpenMotif 2.1 compatible RPM.
Building grid engine from freebsd ports
As this thread points out, the aimk build script for Grid Engine 6.0u7 included in the freebsd ports tree has trouble building working versions of the berkeleydb spooling code.
A patch has been created but may not be in the ports tree yet. This URL may be a good place to check periodically for updates: http://www.freebsd.org/cgi/cvsweb.cgi/ports/sysutils/sge/.
RSS feed for recent gridengine source code changes
Feedback wanted on a new experiment I've been playing around with ...
If you make use of an RSS feed reader or aggregator and are interested enough in Grid Engine to want to keep an eye on codebase activity then please check out this newly created (and still in testing) RSS feed:
http://gridengine.info/feeds/gridengine-codebase-activity.rdf
The feed is generated via the cvs2rss.pl script which in turn, relies on cvs2cl.pl.
By default, cvs2rss.pl is only able to generate HTML links to websites using the ViewCVS CGI front end. However, since the code was so well written it was a trivial exercise for me to write a new CVS2RSS::Formatter::CollabNet perl module that is capable of deep linking into the collabnet-powered gridengine.sunsource.net CVS browsing tools.
The final customization was copied by from great hack done by newly minted Ph.D. and bioperl guru Jason Staijch. The original version of cvs2rss.pl creates HTML links that go directly to the revised file mentioned in the CVS commit log. Jason ended up hacking the cvs2rss Formatter:: code so that the URL (by default) points to a page that renders a "colored diff" comparison between the current file revision and the previous version. This is a great, direct and easy way to directly see *exactly* what was changed or added.
Here is an example showing the "default" behavior of cvs2rss.pl versus the "compare changes URL" behavior:
- Version 1.123.2.5 of "sge_event_master.c", default cvs2rss.pl URL
- Version 1.123.2.5 of "sge_event_master.c", the comparative URL
Nice hack, huh?
The RSS feed works best for single-file commits as it will generate links that point directly at the above mentioned "colored diff" URL hosted at the gridengine.sunsource.net website. For cases where *many* file changes are committed at the same time, the best we can do is link generically to the gridengine.sunsource.net CVS browsing page.
Please check this out and let me know if it is useful. Currently I'm generating the RSS manually but it should be easy to automate if people actually find it worthwhile.
Immediate Accounting Data Flushing 1
One of the short-comings of Grid Engine 6.0 (as compared to 5.3) is that a gap exists between when a job finishes and when the job shows up in the accounting tool. The result is that for a few seconds after a job ends, it is as though the job never existed. Many people have complained about it, and now it is fixed. With the next release of Grid Engine, that gap is closable.
A little background. One of the big changes going from 5.3 to 6.0 was making the qmaster multi-threaded. In 5.3, the qmaster was one big loop. With 6.0, the qmaster now runs in about 14 different threads. Among those threads is a timed event thread. The timed event thread is used to do things on a periodic basis. Many tasks that were handled in the main loop in 5.3 wandered into the timed event thread in 6.0. One of the things that moved over was the writing of accounting and reporting data to disk. In 5.3 the data was written as soon as it was available. In 6.0, to improve performance, the data is buffered before being written. The buffer period is controled by the flush_time parameter of the reporting_params in the global host configuration. The minimum value for this setting is 1 second.
The problem comes from the fact that two separate buffers are being written at the same time. One is the accounting information buffer, which is what qacct uses to find historical job data. The other is the reporting information buffer, which is used by ARCo to create a utility computing database. Because the reporting buffer produces massive amounts of information, in a normal system setting flush_time to any small value results in an overloaded qmaster. However, since the buffers are only flushed every flush_time seconds, there’s a gap between when the accounting information for a finished job enters the buffer and when it is written to the accounting file.
To fix this problem, we split the flush event into two: one for accounting and one for reporting. Now, it’s possible to set the accounting_flush_time, which controls the accounting flush interval, separately from the flush_time, which controls the reporting flush interval. In addition, if the accounting_flush_time is set to “00:00:00”, accounting information will not be buffered at all. It is instead written directly to the accounting file. (OK. In reality, it’s still written to the buffer, but the buffer is immediately flushed.) To maintain backward compatability, the accounting_flush_time parameter is optional. If is it not set, the accounting flush interval will be set to the flush_time, i.e. the accounting and reporting buffers will be flushed at the same time, just like with the original 6.0 behavior.
To get this fix, you either have to download the latest version of the maintrunk or s2 branch of the Grid Engine source, or you have to wait for the next release. You then must set the accounting_flush_time parameter in the reporting_params of the global host config. (See qconf -mconf global.)
Grid Engine source via CVS
The following URL is supposed to show people how to get SGE sourcecode but since the page itself is a master CollabNet template (thus it can not be edited or changed by Grid Engine project members or developers) it actually is not all that correct:
http://gridengine.sunsource.net/servlets/ProjectSource
Want to get SGE source via CVS? The main trick is to ignore the instructions about CVS user password ‘guest’. The fact is, there is no password for guest access. Just hit the enter key when prompted for a password.
Source tarballs for 5.x and 6.x
If you just want to download Grid Engine sourcecode and don’t need access to the most bleeding-edge stuff in the cvshead then this location may be better (look for the files with “TAG-src” in the filename):
http://gridengine.sunsource.net/servlets/ProjectDocumentList
XML Feeds