1.5.2 bugfix release

Fixed a problem with gqdel, and disabled deep resubmission with direct stage out to GridFTP servers.

Deep resubmission can be triggered if there is a problem in the final stage out; and if that happens you end up with an uncertain state on the remote server. So deep resubmission is disabled, to make the job fast fail.

Having a couple of problems accessing the SVN reporsitary right now; so whilst I have more feature written, they're being held up with procedural issues.

Repo contains the new rpm, and tarball is available.

Not dead! And 1.5.1

I've not updated since the Uppsala conference; mostly due to unexpectedly high levels of work. As the title subtly hints at, I'm not dead yet (although in spite of a few things).

Much more relevantly, there's a 1.5.1 release containing some modest restructuring. This is primarily to make it fit in better with the Python conventions; so that I could use the more powerful packaging it offers.

In particular, there's an RPM repository set up, at http://www.scotgrid.ac.uk/gqsub/repo which contains RPM for gqsub. There's also a .tar.gz bundle that works much like the previous versions. If you're a system administrator, you probably want the RPM; or if you're and end user, the tar.gz bundle can be unzipped into your own directory and used from there. There is no functionality difference between the two.

Beyond that, work continues on the various features, albeit with some slippage.

Array jobs and orders arrays

The current multiple job interface - the array job semantics - was pulled from SGE/Torque, without a lot of reflection.

This is not necessarily a bad thing, given that it matched existing interfaces, and provided functionality that was needed by users. However it offers only a single model of sub-job interactions - the case of trivially parallel.

Within the home context of qsub, this is not unreasonable. If there are other dependency models, they could be handled separately, with low overhead. For example, an intial calculation needed by subsequent jobs would be run first, a final gather step run afterword, etc. In general, if a user had complex job chaining needs, they could always get qsub installed on the cluster, and have jobs launch jobs. That's not available from Grid contexts, and the latency on jobs makes doing initial/final jobs more painful.

So, I'm thinking about doing four specific modes of operation for array jobs. Firstly, the current and what will be the default - bag of tasks (unordered). The next two are slight changes on that - first-first and last-last; guaranteeing that the indicated task will be run to completion before any other job starts, or the final job won't be started until all the other tasks have completed. This would typically be used with the environment variables, so the same script will have all the job control built in, and select the behaviour at run time. Clearly, first-first and last-last could both be used at the same time.

The final sequencing mode would be strict-ordering. Each task in perfromed in order, and all tasks will complete before the next job is started. This allows any task to refer to the results of previous tasks. It's obvious that this also means first-first and last-last, as side effect.

I can compile these sequencing modes down to the WMS's DAG jobs - as they are clearly a subset of the Directed Acyclic Graph of tasks. My gut feeling is that these modes will give the most common sequencing types that are used, without having to handle complex specifications - although I think I need to see if I can dig out some data on that.

At the moment, I'm tagging these job ordering features as unscheduled - so many ideas, working on them ordered by user feedback. If you'd use this ordered sub tasks, let me know, and I'll bump it up the list!

EGEE User Forum

Arrived in Uppsala, Sweden for the EGEE User Forum. There's a poster about the data handling side of things, and I'll be demoing gqsub from the UKI stand at various points. Feel free to grab me to talk about gqsub, and other things if you're around!

Given that I've just got all the ideas that came from the last User Forum implemented, it's time to fill that buffer back up again...

CREAM Engine

As Stuart mentioned, I'm putting a CREAM engine together. A basic first draft has been checked in, and I'm in the process of writing a set of tests specifically for CREAM rather than amending the original test set (so I can make sure I haven't broken gLite submission, as well as checking the CREAM one works).

1.5.0 release, and further developments

That's the 1.5.0 release up on the gqsub page.

This handles data staging to and from Storage elements. It's a little ... direct ... in a few cases, so if you are staging multiple files and they have multiple replica's, then it doesn't guarantee to use the optimal replica. This is, however, a fairly minor issue - if you have to handle that much data then we're quite a bit beyond something that could run on a local cluster. Nevertheless, I will work on that part, and tighten it up. As it stands, it's comparable with other user tools that weren't written for a particular VO.

In parallel with that, Morag is working on a second submission engine on the back end - this one for direct submission to CREAM. It's not very different, which makes it an excellent 2nd target, and a solid step on the way to backend independence. In particular, direct submission to CREAM requires a GridFTP server, or something similar, in order to get the data back, so requirements for backends need to be handled. She's already cleaned up some of the job handling code; so look for direct CREAM submission (no WMS needed) in around 1.6 - 1.7 timeframe.

Now, on to more data handling code ...

Data transfers - now with workyness

As of the current SVN tree, data handling from SE's is supported. (It would have been yesterday, but there was a blip in the SVN severs).

The general method of specification is:

#GQSUB -W stagein=jobName@sourceName

or

#GQSUB -W stageout=jobName@targetName

The syntax is derived from the PBS/Torque/SGE model for specifying stagein/out files, with a few extensions.

The jobName is the name the file should have at the time of execution on the worker node. The sourceName and targetName are the specification of where to get the file from / put it to. If the jobName is ommitted, then the file arrives with the same name as it had remotely.

The valid ways of requesting a file are:

simpleFileName

local/path/to/file

A file local to the submission machine

host:file

host:path/to/file

An SCP destination, which the submission machine can access.

gsiftp://host/path/file

GridFTP URL

srm://host/path/to/file

SRM Storage Element

lfn:/path/to/file

Logical File Name (via LFC)

For SRM and LFC modes, these are converted at submission time into GridFTP urls. This is fine if the data is only at one place, but I'll change that later to automatically make it pull from the nearest.

Unfortunatly, determining the topologically nearest SE is ... tricky. Most of the usual tools (ping times, packet pairing for bandwidth assesment, etc) are unreliable, as ICMP is not guarenteed. So I think I'll have to get down and dirty with TCP/IP and HTTPS, and use that to make the estimates. Fun ... in an ironic sort of way.

Still, it's there, it works, and it does the job.

I'm going to do some more testing and stressing, before I wrap up 1.5 release - probably next week for the release.

Oh, look - our cluster is quiet right now. Time to find out what breaks when I throw throusands of jobs at it...