Tag: Accounting

Reconciling exclusive host access with the SGE accounting file

Posted by chrisdag – January 11, 2012

In this mailing list thread there is an interesting discussion about how to correctly account for all of the Job Slots that are blocked off or made inaccessible by users submitting “exclusive” jobs that take over an entire execution host. Stuff like this becomes more and more important in the era of commodity boxes with core counts climbing beyond 48 …

I defined an exclusive tag in complex resources for users that can request “-l excl=1 ”
Such users, lock free slots of hosts that have added excl=true to consumable resources.

1) In this case, how can I get accounting for all of slots that a user cause to lock on hosts?

it’s not recorded in the accounting file where the job ran, i.e. a “pe_hostfile” like record. As a result also the blocked slots are not recorded. What you can try if you need it: use a global (or queue) prolog, which runs under e.g. the sgeadmin account by prefixing it with the name (see `man queue_conf`, resp. `man sge_conf`) and copy the “pe_hostfile” with the information of the distribution to a central place. Whether the exclusive access was requested, is recorded already in the accounting file (like all -l resource requests).

With this information, you can compute the real blocked cores then.

– Reuti

New community contributed utility scripts

Posted by chrisdag – May 18, 2009

Thanks to Chris Bingham, for posting a few of his SGE utility scripts to the gridengine.info wiki and SGE users mailing list.

Scripts and comments from Chris:

  • User_Job_Stats: This script calculates either total or average usage for each user over a given time period (week, month or year to date, or a number of days).
    It also tots up these figures for the cluster globally, and calculates what percentage of the cluster’s potential capacity was actually used.
  • qtime: This one calculates the minimum, maximum and average times jobs had to wait to be executed over the given period.
    I wrote it so people could get a sense of how long they could expect to wait for their jobs to start when the submitted them. These figures were also used as a measure of cluster performance in the SLA for one system I worked on.
  • Queue_Job_Count: Simply determines what queues exist on the cluster, then counts up how many jobs each has processed over the given time period.
  • Queue_Change_State: A very short script that enables or disables all queue instances on the node its
    run on. I’ve found it useful for quickly knocking out the queues on nodes that are going
    down for maintenance.

Scripts and comments are available on the Wiki Page or as attachments to the original list posting.

New user contributed accounting script

Posted by chrisdag – May 15, 2007

A new “pull statistics from the SGE accounting log file” script has been posted to the SGE community. Olivier Blondel took Joe Landman’s “usage.pl” script and modified it to suit his own needs. The script can be found embedded inline with Olivier’s post to the users mailing list.

It’s official: Project Hedeby and ARCo join the SGE codebase

Posted by chrisdag – December 13, 2006

Sun has formally announced the additions promised at SC’06, the full announcement is available online here:

Of the two, ARCo is the more established layered product. This is the SQL driven accounting and reporting tool that was previously only available in the commercial version of N1 Grid Engine from Sun. ARCo uses Java to parse the SGE accounting logs for inclusion into an SQL back-end database. In addition to the metrics found in the accounting logs, ARCo has hooks for calculating useful “derived” metrics that are not explicitly stored in the accounting files.

When I first used ARCo (early on in its very first release version) one of the main weaknesses was the front end web based reporting console – for anything but the most basic reports, a user was expected to paste raw SQL queries into a web form. Sun’s act of putting ARCo into the open source codebase should hopefully kickstart an idea that has been floating around for a while — some sort of community wiki page or repository of user-generated ARCo queries and report templates. ARCo users are encouraged to send these sorts of tips and tricks to the users mailing list.

“Project Hedeby” aka the “Grid Engine Service Domain Management module” also mentioned at SC’06 is at an earlier stage in it’s development. The nontechnical description is as follows:

Project Hedeby provides access to a new technology which allows to dynamically manage resources across so called Service Domains. Service Domains can be envisioned as autonomous Grids controlled by a resource manager including but not limited to Grid Engine. Hedeby will adjust the allocation of resources to individual service domains in order to meet Service Level Objectives. Reallocating a host resource to another service domain may include re-provisioning of the underlying virtual or actual operating system stack.

In his interview with GRIDtoday, Fritz provides the following description:

“…* provides policy and demand-based re-allocation of arbitrary resources across service domains. Service domains are totally autonomous Grids which are controlled by a workload management facility, such as Grid Engine, but also by arbitrary other service infrastructures like application servers or web servers…”

Thanks to Andy for pointing out that the project codename, “Hedeby” refers to a Viking trade town from the 8th-11th century.

Simple perl reporting tool for SGE accounting data

Posted by chrisdag – October 11, 2006

Joe at Scalable Informatics is offering up a “quick -n- simple” reporting script for Grid Engine accounting and usage data.

Usage examples: 

[landman@minicc ~]$ ./usage.pl
Total usage: (in units of second(s))
        wallclock  :       46733.000 second(s)
        user time  :        1600.000 second(s) [3.42%]
        system time:          17.000 second(s) [0.04%]
        cpu time   :       70379.000 second(s) [150.60%]

user            wallclock       user time       system time     cpu time
       memory          percent of total time
landman         46733.000       1600.000        17.000
70379.000       0.000           100.000

New SGE accounting log analysis script committed

Posted by chrisdag – May 26, 2006

Andreas has checked in a Ruby script that does grid engine accounting file analysis. His email announcement has the details and a basic usage summary.

The script can be obtained from CVS or via a direct download: http://gridengine.sunsource.net/files/documents/7/82/analyze.rb.gz

Immediate Accounting Data Flushing

Posted by chrisdag – November 4, 2005

One of the short-comings of Grid Engine 6.0 (as compared to 5.3) is that a gap exists between when a job finishes and when the job shows up in the accounting tool. The result is that for a few seconds after a job ends, it is as though the job never existed. Many people have complained about it, and now it is fixed. With the next release of Grid Engine, that gap is closable.

A little background. One of the big changes going from 5.3 to 6.0 was making the qmaster multi-threaded. In 5.3, the qmaster was one big loop. With 6.0, the qmaster now runs in about 14 different threads. Among those threads is a timed event thread. The timed event thread is used to do things on a periodic basis. Many tasks that were handled in the main loop in 5.3 wandered into the timed event thread in 6.0. One of the things that moved over was the writing of accounting and reporting data to disk. In 5.3 the data was written as soon as it was available. In 6.0, to improve performance, the data is buffered before being written. The buffer period is controled by the flush_time parameter of the reporting_params in the global host configuration. The minimum value for this setting is 1 second.

The problem comes from the fact that two separate buffers are being written at the same time. One is the accounting information buffer, which is what qacct uses to find historical job data. The other is the reporting information buffer, which is used by ARCo to create a utility computing database. Because the reporting buffer produces massive amounts of information, in a normal system setting flush_time to any small value results in an overloaded qmaster. However, since the buffers are only flushed every flush_time seconds, there’s a gap between when the accounting information for a finished job enters the buffer and when it is written to the accounting file.

To fix this problem, we split the flush event into two: one for accounting and one for reporting. Now, it’s possible to set the accounting_flush_time, which controls the accounting flush interval, separately from the flush_time, which controls the reporting flush interval. In addition, if the accounting_flush_time is set to “00:00:00”, accounting information will not be buffered at all. It is instead written directly to the accounting file. (OK. In reality, it’s still written to the buffer, but the buffer is immediately flushed.) To maintain backward compatability, the accounting_flush_time parameter is optional. If is it not set, the accounting flush interval will be set to the flush_time, i.e. the accounting and reporting buffers will be flushed at the same time, just like with the original 6.0 behavior.

To get this fix, you either have to download the latest version of the maintrunk or s2 branch of the Grid Engine source, or you have to wait for the next release. You then must set the accounting_flush_time parameter in the reporting_params of the global host config. (See qconf -mconf global.)