[8.0] Interpret outputPath with LFN: prefix as an absolute one#8603
[8.0] Interpret outputPath with LFN: prefix as an absolute one#8603atsareg wants to merge 5 commits into
Conversation
|
Thank you. All seems fine to me. |
fstagni
left a comment
fstagni
left a comment
There was a problem hiding this comment.
@aldbr has added a lot of tests for JobWrapper in https://github.com/DIRACGrid/DIRAC/blob/integration/src/DIRAC/WorkloadManagementSystem/JobWrapper/test/Test_JobWrapper.py.
Can you see if (some of) those can be used here? I am always somewhat scared of changing the JobWrapper.
| lfnPath = os.path.dirname(lfn) | ||
| lfnLocal = os.path.basename(lfn) | ||
| globbedLfnList += [os.path.join(lfnPath, gLfn) for gLfn in List.uniqueElements(getGlobbedFiles(lfnLocal))] | ||
| if globbedLfnList != lfnList and globbedLfnList: |
There was a problem hiding this comment.
Slightly more precise:
| if globbedLfnList != lfnList and globbedLfnList: | |
| if globbedLfnList and globbedLfnList != lfnList: |
| # Check whether the list of LFNs has globbable patterns | ||
| globbedLfnList = [] | ||
| for lfn in lfnList: | ||
| lfnPath = os.path.dirname(lfn) | ||
| lfnLocal = os.path.basename(lfn) | ||
| globbedLfnList += [os.path.join(lfnPath, gLfn) for gLfn in List.uniqueElements(getGlobbedFiles(lfnLocal))] | ||
| if globbedLfnList != lfnList and globbedLfnList: | ||
| self.log.info("Found a pattern in the output data LFN list, LFNs to upload are:", ", ".join(globbedLfnList)) | ||
| lfnList = globbedLfnList | ||
|
|
||
| # Check whether the list of outputData has a globbable pattern |
There was a problem hiding this comment.
There is a code duplication between these inserted lines and those that follow, care to refactor for simplicity?
| # If output path is given with the LFN: prefix, take it as an absolute path | ||
| elif outputPath.startswith("LFN:"): | ||
| outputPath = outputPath[4:] | ||
| basePath = "" |
There was a problem hiding this comment.
Shouldn't this block be before the previous 2 lines? For the case when outputPath == "LFN:/some/where/some/thing.xyz"
Yes, there are many tests to JobWrapper added to the integration/9.0 . In 8.0 there are just few. Will see if I can add something here |
| - file names; in this case files with the specified names will be looked for in the job directory and uploaded | ||
| to a location specified by the OutputPath; | ||
| - file names with wild cards; same after the file names expansion; | ||
| - file names in a form "LFN:/vo/full/destination/path/file.name"; in this case the file will be uploaded |
There was a problem hiding this comment.
"file names"? "LFN:/vo/full/destination/path/file.name" is a a full "file path" (directory path / filename). It's better to be precise to avoid confusion.
And in the previous 2 lines, did you mean "filename", "file path", or both?
| - file names with wild cards; same after the file names expansion; | ||
| - file names in a form "LFN:/vo/full/destination/path/file.name"; in this case the file will be uploaded | ||
| to the specified LFN path without taking into account the OutputPath. Note that file.name here can be also | ||
| specified with wild cards. |
There was a problem hiding this comment.
Give an example of a wild card.
| for example "/lhcb/user/a/atsareg/1234/1234567", where 1234567 is the job ID; | ||
| - if given as path starting with "/", it will be appended to the user's home | ||
| directory; | ||
| - if given as "LFN:/output/path", it will be taken as an absolute path for |
There was a problem hiding this comment.
What happens If, while submitting a user job I request OutputPath = LFN:/atlas/I_should_not_be_allowed/here/?
3 participants
BEGINRELEASENOTES
*WorkloadManagement
NEW: JobWrapper - interpret outputPath with LFN: prefix as an absolute one
NEW: JobWrapper - allow wild cards in output LFNs
For examples look into release.notes
ENDRELEASENOTES