That's the 1.5.0 release up on the gqsub page.
This handles data staging to and from Storage elements. It's a little ... direct ... in a few cases, so if you are staging multiple files and they have multiple replica's, then it doesn't guarantee to use the optimal replica. This is, however, a fairly minor issue - if you have to handle that much data then we're quite a bit beyond something that could run on a local cluster. Nevertheless, I will work on that part, and tighten it up. As it stands, it's comparable with other user tools that weren't written for a particular VO.
In parallel with that, Morag is working on a second submission engine on the back end - this one for direct submission to CREAM. It's not very different, which makes it an excellent 2nd target, and a solid step on the way to backend independence. In particular, direct submission to CREAM requires a GridFTP server, or something similar, in order to get the data back, so requirements for backends need to be handled. She's already cleaned up some of the job handling code; so look for direct CREAM submission (no WMS needed) in around 1.6 - 1.7 timeframe.
Now, on to more data handling code ...
Showing posts with label datamanagement. Show all posts
Showing posts with label datamanagement. Show all posts
Thursday, 18 March 2010
Thursday, 4 March 2010
Data transfers - now with workyness
As of the current SVN tree, data handling from SE's is supported. (It would have been yesterday, but there was a blip in the SVN severs).
The general method of specification is:
#GQSUB -W stagein=jobName@sourceName
or
#GQSUB -W stageout=jobName@targetName
The syntax is derived from the PBS/Torque/SGE model for specifying stagein/out files, with a few extensions.
The jobName is the name the file should have at the time of execution on the worker node. The sourceName and targetName are the specification of where to get the file from / put it to. If the jobName is ommitted, then the file arrives with the same name as it had remotely.
The valid ways of requesting a file are:
For SRM and LFC modes, these are converted at submission time into GridFTP urls. This is fine if the data is only at one place, but I'll change that later to automatically make it pull from the nearest.
Unfortunatly, determining the topologically nearest SE is ... tricky. Most of the usual tools (ping times, packet pairing for bandwidth assesment, etc) are unreliable, as ICMP is not guarenteed. So I think I'll have to get down and dirty with TCP/IP and HTTPS, and use that to make the estimates. Fun ... in an ironic sort of way.
Still, it's there, it works, and it does the job.
I'm going to do some more testing and stressing, before I wrap up 1.5 release - probably next week for the release.
Oh, look - our cluster is quiet right now. Time to find out what breaks when I throw throusands of jobs at it...
The general method of specification is:
#GQSUB -W stagein=jobName@sourceName
or
#GQSUB -W stageout=jobName@targetName
The syntax is derived from the PBS/Torque/SGE model for specifying stagein/out files, with a few extensions.
The jobName is the name the file should have at the time of execution on the worker node. The sourceName and targetName are the specification of where to get the file from / put it to. If the jobName is ommitted, then the file arrives with the same name as it had remotely.
The valid ways of requesting a file are:
- simpleFileName
- local/path/to/file
- A file local to the submission machine
- host:file
- host:path/to/file
- An SCP destination, which the submission machine can access.
- gsiftp://host/path/file
- GridFTP URL
- srm://host/path/to/file
- SRM Storage Element
- lfn:/path/to/file
- Logical File Name (via LFC)
For SRM and LFC modes, these are converted at submission time into GridFTP urls. This is fine if the data is only at one place, but I'll change that later to automatically make it pull from the nearest.
Unfortunatly, determining the topologically nearest SE is ... tricky. Most of the usual tools (ping times, packet pairing for bandwidth assesment, etc) are unreliable, as ICMP is not guarenteed. So I think I'll have to get down and dirty with TCP/IP and HTTPS, and use that to make the estimates. Fun ... in an ironic sort of way.
Still, it's there, it works, and it does the job.
I'm going to do some more testing and stressing, before I wrap up 1.5 release - probably next week for the release.
Oh, look - our cluster is quiet right now. Time to find out what breaks when I throw throusands of jobs at it...
Thursday, 25 February 2010
Buckets and sieves
Deep into data management at the moment - hence buckets of data.
One of the more annoying aspects of working with computers is the leaky abstraction; and that's where the sieve comes in.
A leaky abstraction is a particularly cruel thing - it appears to promise things that don't hold true. For example, if something acts like a list that you can add things to, you expect to be able to add things to it. If it didn't allow things that started with the word 'aardvark', most people wouldn't notice it. The moment an entomologist gets involved, and the abstraction breaks apart.
The particular problem I've been having is that when staging in files, there's an established syntax for naming files, which allows them to be different names on the worker node than on the source filesystem. So what should happen if you want several files with the same source filename, but distinct names on the worker node?
Well, at the moment, you get a right mess. Files are staged in to a local file with the same filename as the remote. Then, if you need it, gqsub will rename them. If there's two files with the same remote name, then the last specified one clobbers the others. That's a problem that the gLite middleware has had for a while (mostly as it doesn't support renaming files at all), but it's a leak - it breaks the mental model we're trying to support here.
In addition, what happens if the user wants 2 copies of the same file, under different names, on the worker node? It's perhaps a rare case, but I can think of a few cases (mostly where the file is mutated by the work, and it's the differences that are important, not the absolute data). In this case, we have to consider where do we do the duplication, and how. I plumped for doing on the worker node - most filesystems will be able to do copy-on-write, and the network transfer is likely to be more expensive than the local copy.
Anyway, this is mostly about the issues, as a way of thinking about the resolutions.
Ultimatly, I'd like the abstractions presented to be as leak free as possible, so that people can reason about what will happen without having to make reference to the underlying system.
One of the more annoying aspects of working with computers is the leaky abstraction; and that's where the sieve comes in.
A leaky abstraction is a particularly cruel thing - it appears to promise things that don't hold true. For example, if something acts like a list that you can add things to, you expect to be able to add things to it. If it didn't allow things that started with the word 'aardvark', most people wouldn't notice it. The moment an entomologist gets involved, and the abstraction breaks apart.
The particular problem I've been having is that when staging in files, there's an established syntax for naming files, which allows them to be different names on the worker node than on the source filesystem. So what should happen if you want several files with the same source filename, but distinct names on the worker node?
Well, at the moment, you get a right mess. Files are staged in to a local file with the same filename as the remote. Then, if you need it, gqsub will rename them. If there's two files with the same remote name, then the last specified one clobbers the others. That's a problem that the gLite middleware has had for a while (mostly as it doesn't support renaming files at all), but it's a leak - it breaks the mental model we're trying to support here.
In addition, what happens if the user wants 2 copies of the same file, under different names, on the worker node? It's perhaps a rare case, but I can think of a few cases (mostly where the file is mutated by the work, and it's the differences that are important, not the absolute data). In this case, we have to consider where do we do the duplication, and how. I plumped for doing on the worker node - most filesystems will be able to do copy-on-write, and the network transfer is likely to be more expensive than the local copy.
Anyway, this is mostly about the issues, as a way of thinking about the resolutions.
Ultimatly, I'd like the abstractions presented to be as leak free as possible, so that people can reason about what will happen without having to make reference to the underlying system.
Monday, 15 February 2010
Data, data everywhere ...
and not a drop to download.
(I know it doesn't scan right - work with me here...)
Currently working on the data management aspects; aiming to make data as easy to work with as compute. And as efficient as possible...
There's a certain amount of mis-match with the various data tools in gLite / LCG. It's as if one group went with round wheels and the other build roads of inverted catenaries. Both fine ideas, but not when used together...
Within the Job Description Language, we can specify hard requirements on data files that we want to be 'close' to [0]. However, we can't get the job wrapper to handle staging in those files for us - so the job has to do that itself.
The primary tools [1] for handling data is the lcg-utils; which are pretty good. They work nearly transparently with gsiftp (single file) urls, srm (storage element) urls, and lfn (logical file name) uri's - so as you move through the hierachy, the tools are very similar. They're also auto load balancing, so picking a random replica each time for cases when the same data is in multiple locations.
Alas, these two don't play well - if we use the JDL DataRequirements, we'll be on an worker node 'close' to a Storage Element that has the data; but using lcg-utils naively, we'll pull the data from somewhere random. That's not good, given we already know we're close to one.
So the plan is, in the short term, do it that way, in all it's simple to write, inefficent glory. Once I have a working infrastructure for data staging, then I'll refine it. There's interesting trade-off's in the load on the Logical File Catalogue, versus runtime on the worker node, against reliability. But the first stab will one that 'works', in the sense of gets the job and the data in one place.
[0] Although if you specify a set of files that are not all present in a single Storage Element, then nowhere matches. Not the most helpful ...
[1] Read: most usable ones. You can do the same thing in more complicated ways, if you really want to...
(I know it doesn't scan right - work with me here...)
Currently working on the data management aspects; aiming to make data as easy to work with as compute. And as efficient as possible...
There's a certain amount of mis-match with the various data tools in gLite / LCG. It's as if one group went with round wheels and the other build roads of inverted catenaries. Both fine ideas, but not when used together...
Within the Job Description Language, we can specify hard requirements on data files that we want to be 'close' to [0]. However, we can't get the job wrapper to handle staging in those files for us - so the job has to do that itself.
The primary tools [1] for handling data is the lcg-utils; which are pretty good. They work nearly transparently with gsiftp (single file) urls, srm (storage element) urls, and lfn (logical file name) uri's - so as you move through the hierachy, the tools are very similar. They're also auto load balancing, so picking a random replica each time for cases when the same data is in multiple locations.
Alas, these two don't play well - if we use the JDL DataRequirements, we'll be on an worker node 'close' to a Storage Element that has the data; but using lcg-utils naively, we'll pull the data from somewhere random. That's not good, given we already know we're close to one.
So the plan is, in the short term, do it that way, in all it's simple to write, inefficent glory. Once I have a working infrastructure for data staging, then I'll refine it. There's interesting trade-off's in the load on the Logical File Catalogue, versus runtime on the worker node, against reliability. But the first stab will one that 'works', in the sense of gets the job and the data in one place.
[0] Although if you specify a set of files that are not all present in a single Storage Element, then nowhere matches. Not the most helpful ...
[1] Read: most usable ones. You can do the same thing in more complicated ways, if you really want to...
Subscribe to:
Posts (Atom)