gridengine.info : Category external-tools-apps, everything about external-tools-apps

MarkMail: Mine the grid engine maillist archives

dag@sonsorol.org (chris) — Wed, 17 Sep 2008 07:48:31 -0400

MarkMail has just imorted all of the Grid Engine mailing lists from http://gridengine.sunsource.net into their archive, search, index and database system. Initial results are pretty impressive based on a few minutes of searching and experimentation -- seems like a great way to search the mailing lists for answers and info.

Click on the image above and you'll be take to a search on the term 'rqs'. Leave a comment with your impressions if you are so inclined.

MarkMail: Mine the grid engine maillist archives

dag@sonsorol.org (chris) — Wed, 17 Sep 2008 07:48:31 -0400

Click on the image above and you'll be take to a search on the term 'rqs'. Leave a comment with your impressions if you are so inclined.

Screenshots of enhanced Olesen FLEXlm tools in action

dag@sonsorol.org (chris) — Thu, 06 Mar 2008 09:21:00 -0500

In a follow-up post to Mark's recent announcement we've gotten our hands on some screenshots from Mark showing his tools in use. The screenshots show the results of using XSLT transformations to turn Grid Engine XML data into XHTML form suitable for web pages. The benefit includes web-based visibility into current resource (and software license!) usage. This is exactly the approach that I tried out with the xml-qstat project. Mark is pretty familiar with that effort and will be merging his improvements and enhancements into xml-qstat's SVN repository. Speaking personally as a "scratch an itch" programmer with no real software engineering skill or talent I'm pretty excited to have a real coder take a look at xml-qstat. Related to that I already owe a debt to Petr Jung from Sun who contributed the Java based CommandGenerator code that finally allows xml-qstat to be a 100% Java/Cocoon web application that does not require external perl daemons to cache XML state data.

Before the screen captures, I'd like to ask a favor of people who read this blog. I filed bug Issue #2335 back in July of 2007 and it has not received much love (or even a targeted milestone date for a fix). The bug is a simple one -- "qstat -f -xml" no longer reports load average data which (a) makes xml-qstat a whole lot less useful and (b) breaks the SGE developer philosophy of ensuring that command output returns the same information regardless of output format. Until that bug is fixed it does make sense for xml-qstat to have it's long overdue "1.0" release. If you have a user account over on http://gridengine.sunsource.net I'd appreciate it if you can cast one of your "votes" for Issue 2335. Thanks!

And now the screenshots (edited to mask out personal/company information). Click on each image for a larger version.

qhost overview

Click on through for the rest of the pictures ...

qstat full view (a)

qstat full view (b)

qstat queue summary

qstat resource summary

qstat view

Advanced Reservation plugin for Grid Engine

dag@sonsorol.org (chris) — Wed, 25 Oct 2006 17:54:22 -0400

Yoshio Tanaka posts the following:

... We are pleased to announce that advance-reservation plugin module
called PluS version 1.0.0 RC 1 is now available for download at the
PluS home page at:
  http://www.g-lambda.net/plus/ .

PluS (Plug-in Advance Reservation Manager for Torque and Grid Engine)
adds an advance-reservation function to Torque and Grid Engine.
For SGE, one of the following operations will be performed based on
the startup option.

(1) SGE queue base version
  - The SGE schedule is not replaced, and the reservation function is
    realized simply by managing the reservation queues.

(2) SGE self scheduling version
  - The original SGE scheduler is replaced by the PluS SGE scheduler
    which realizes the reservation management function and the job
    scheduling function.

...

The package is released under the Apache 2 License. It appears that the system has mainly been developed and tested on the following configuration: Linux 2.6.x, Intel x86, glibc 2.3.3, SGE 6.0u8

The HTML version of the PluS Manual is online here:
http://www.g-lambda.net/plus/wp-content/uploads/2006/10/manual.html.

The http://www.g-lambda.net/plus/ site contains a link to a PDF from a IEEE conference paper covering the system in more technical detail.

OpenDSP announcement: Web services access to SGE via DRMAA

dag@sonsorol.org (chris) — Thu, 11 May 2006 10:25:02 -0400

Interesting announcement on the mailing lists today regarding http://sourceforge.net/projects/opendsp.

From the website:

"OpenDSP is an open architecture implementation of SOAP Web Service multi-user access and policy-based job control using DRMAA (Distributed Resource Management Application API) routines supported by various distributed resource management systems."

For version v0.1 RC1 the feature list is pretty interesting. Hopefully I'll have time to experiment with this a bit.

sorting qstat output

dag@sonsorol.org (chris) — Sun, 12 Feb 2006 16:07:42 -0500

A user recently asked the mailing list for suggestions on sorting the full output of qstat by job start time.

Reuti replied back with a link to his most excellent script, a bash script called "status" that makes heavy use of awk under the hood. The script works with both SGE 5.3 and 6.x versions of qstat.

The script is hosted on the download section of the SGE project website:
http://gridengine.sunsource.net/servlets/ProjectDocumentView?documentID=8&showInfo=true

After downloading the script, usage is trivial. To sort output by job start time one would do:

 ./status -s time -a
Running jobs:
job-ID  # name                      owner      start time          running in
-----------------------------------------------------------------------------
   561  1 Job7458                   www        01/08/2006 18:59:05 all.q      (stalled)
   653  1 A11510113941883           www        02/08/2006 09:13:58 all.q      
   657  1 A11541113941889           www        02/08/2006 09:14:54 all.q      

Waiting jobs:
job-ID  # name                      owner      submit time        
------------------------------------------------------------------
   562  1 Job7458.cleanup           www        01/08/2006 17:38:14 (hold)
   654  1 btpymol                   www        02/08/2006 09:13:59 (Error)
   654  1 btpymol                   www        02/08/2006 09:13:59 (Error)
   655  1 merge                     www        02/08/2006 09:13:59 (hold)
   656  1 cleanup                   www        02/08/2006 09:13:59 (hold)
   658  1 btrasmol                  www        02/08/2006 09:14:55 (Error)
   658  1 btrasmol                  www        02/08/2006 09:14:55 (Error)
   658  1 btrasmol                  www        02/08/2006 09:14:55 (Error)
   659  1 merge                     www        02/08/2006 09:14:55 (hold)
   660  1 cleanup                   www        02/08/2006 09:14:55 (hold)
   407  1 impossibleJob             www        11/28/2005 09:58:42

FLEXlm license integration - Olesen method

dag@sonsorol.org (chris) — Wed, 16 Nov 2005 19:43:27 -0500

In a post to the Grid Engine users list, Mark Olesen proposed a novel approach to FlexLM license integration on Grid Engine managed clusters.

He offered to provide code that implemented his ideas with the caveat that he would be completely unable to provide support or end-user assistance. Since I had already written one of the FlexLM integration HOWTO's and have a personal interest in application integration issues I volunteered to try to support Mark's code if it meant that he would be able to release it to the general user community.

I've been playing with versions of his work for several weeks now and it is very cool, very useful and very well written. Within the last week, Mark has finalized his code and selected a Creative Commons license. Over the last two evenings I've been working on a GridWiki entry documenting the basic installation, configuration and installation.

The first draft (along with a downloadable tarball!) is now ready for public review at this URL: http://gridengine.info/wiki/index.php/Olesen-FLEXlm-Integration.

Obviously, Mark Olesen needs to be acknowledged for proposing his ideas and backing them up with well-written code. I'd also like to specifically thank the fine people at Schrödinger for providing access to some of their FLEXlm licensed software. Schrödinger's valuable contribution made it possible for me to test, evaluate and write about FLEXlm on Grid Engine managed systems. Thanks!

"unlimited" stack limit on Solaris

rayrayson@gmail.com (Rayson) — Mon, 10 Oct 2005 00:01:00 -0400

SGE by default sets the job’s stack limit to “unlimited”, but a lot of Solaris users find that their applications no longer work under SGE.

This blog entry explains the issue in detail

GridSAM - Grid Job Submission and Monitoring Web Service

dag@sonsorol.org (chris) — Fri, 30 Sep 2005 15:00:00 -0400

From http://gridsam.sourceforge.net/1.0/:

”The aim of GridSAM is to provide a Web Service for submitting and monitoring jobs managed by a variety of Distributed Resource Managers (DRM).”

Lots of GridBuzzwords^TM. Their code depends on OMII stuff and the developers are aligned with the GGF. A good first pass overview may be the User Guide or the Use Cases document.

(Via CB-SIG newsletter)

Parsing Grid Engine XML output into Python datastructures

dag@sonsorol.org (chris) — Tue, 20 Sep 2005 12:07:00 -0400

In this mailing list thread both Beth Meyer and Sebastian Stark report finding similar-but-different methods that easily allow them to import XML status information into Python data structures.

Take advantage of Grid Engine XML status reporting

dag@sonsorol.org (chris) — Tue, 20 Sep 2005 12:01:00 -0400

In my mind, one of the most under-appreciated and under-used features of the Grid Engine 6 series is the ability of the ”qstat” program to output terse or very detailed status information in XML form. For those of us used to parsing human-readable qstat output using perl regular expressions and other painful methods this is very welcome news. Grid Engine apparently also communicates internally now using XML-formatted messages but lets save that topic for a future post.

I wrote about early efforts at trying to do something useful and usable with Grid Engine XML for a Bio-IT World magazine column. A snippet of the article is reproduced here:

Where raw Grid Engine XML status data are concerned, XPATH is the technology that allows one to cut through the large volume of data to make targeted queries. Queries such as “Give me information about all pending grid jobs” would be represented as an XPATH search string: “//job_list[@state=‘pending’]”.

The use of XPATH addresses one problem: “How do I wade through lots of XML and pick out the bits that I’m actually interested in?” This is only a partial solution, as one still has to do something interesting (or at least visually pleasing) with the selected XML data. This is where another W3C recommendation comes into play: XSLT 1.0.

The full article can be read online here: http://www.bio-itworld.com/columns/inside-the-box/insidethebox0705

xml-qstat

One of my personal development projects lives at http://xml-qstat.bioteam.net. It represents an attempt to build rich Grid Engine monitoring tools that make use of XML status information. The software is released under a creative commons license and written in Perl. The actual XML handling is done by libxml2 and libxslt libraries from www.xmlsoft.org.