Add wiki page about "creating your own `sfm` script"

[matthdsm]

Hi,

Readme says there should be a wiki page about creating your own sfm scripts, but the wiki is empty.
Can this be added? We're looking into creating our own sfm for a more horizontal spread of the load.

Thanks
M

[caherzee]

Right. It may take a bit of time for me to document this.

You may already want to take a look at the following go file that implements the sfm command:
https://github.com/ExaScience/elprep/blob/master/cmd/sfm.go

We used to give example python scripts for implementing the sfm command in the original elPrep implementation written in Common Lisp. It is maybe useful to take a look at those scripts, but the elPrep workflow has become more complex than it used to be: https://github.com/ExaScience/cl-elprep Cf. the elprep_sfm.py and other .py files.

I will inform you when I have documentation.

Thanks,
Charlotte

[matthdsm]

Thanks,

I'm just worried about missing a crucial piece while rolling our own sfm. The purpose here is to write al small nextflow workflow that enables us to spread the chucks over multiple nodes, negating the RAM hog effect on a single node.

If there's no special gotcha's wrt to intermediate files, I think we'll manage!

[matthdsm]

Small follow up question. I saw in the code the spread file from split is ignored sometimes. Could you elaborate in which cases that might be needed/useful?