kslurm is a new tool being developed to make job submission on the ComputeCanada clusters easier. It is intended to be a more functional, robust version of Neuroglia-helpers and will eventually replace it.
A full description and install instructions can be found on the repository:
https://github.com/pvandyken/kslurm
Features
Currently, the project has implemented three basic commands for job submission:
- kbatch: for batch submission jobs (no immediate output)
- krun: for interactive submission
- kjupyter: for Jupyter sessions
The biggest advantage over previous submit commands is their use of a regex-base argument parser. With these commands, you can request resources using an intuitive syntax:
krun 4 3:00 15G gpu
This command will request an interactive session with 4 cores, for 3hr, using 15GB of memory, and a gpu.
krun can also be run with a command:
krun 00:30 python my_program.py
This command will request a half hour session with the default 1 core and 4G of memory. The job will be scheduled using srun (meaning it runs almost immediately), and the output will be displayed on the terminal.
Unlike sbatch, kbatch does not require a submissions script. You can directly submit a command:
kbatch 2-00:00 snakemake --profile slurm
This will request a 2 day job running snakemake.
All commands support the job templates offered by neuroglia-helpers, using the familiar syntax:
kbatch -j Fat snakemake --profile slurm
Resources specified in the job template can be overriden by providing the appropriate argument:
kbatch -j Fat 1G snakemake --profile slurm
This will request a “Fat” job, including 32 cores and a 1-day runtime, but with only 1GB of memory.
Support and contributions
The project is still in beta, so if you come across any issues, I would greatly appreciate bug reports on the github repo. Please report bugs there, rather than in this forum, to keep things organized. I would also appreciate any suggestions to improve the experience. The design was based around my own typical usage of SLURM, so I’d love to hear useful it is for others’ workflows. Finally, if you’d like to contribute code, I will happily take pull requests. A number of issues to tackle are already listed on the github repo.