Use pixi to manage development conda environments#3044
Use pixi to manage development conda environments#3044bouweandela wants to merge 24 commits intomainfrom
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #3044 +/- ##
=======================================
Coverage 96.15% 96.15%
=======================================
Files 270 270
Lines 15805 15805
=======================================
Hits 15198 15198
Misses 607 607 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
| os.environ["ESMFMKFILE"] = f"{rtd_conda_prefix}/lib/esmf.mk" | ||
| os.environ["PROJ_DATA"] = f"{rtd_conda_prefix}/share/proj" | ||
| os.environ["PROJ_NETWORK"] = "OFF" | ||
|
|
There was a problem hiding this comment.
I see the docs build nicely, so these can indeed be removed, but why are we removing them now - is it something to do with pixi?
There was a problem hiding this comment.
Yes, as explained in the comment above, the conda environment used by the previous way of installing the environment on readthedocs was not activated. Pixi does not have this problem, so this code can be removed.
There was a problem hiding this comment.
I'd keep it somewhere somehow, just in case. I think the whole purge of conda-forge way of installing is a bit premature at this point in time, even if we have pixi in for end users- which is nice, I can already think of a few cases where pixi won't work, so I'd argue we still keep the old conda stashed somewhere, in case we have to use it. How, I need to think of that
|
hey @bouweandela a few points:
(base) valeriu@valeriu-PORTEGE-Z30-C:~/ESMValCore$ pixi --version
pixi 0.67.0
(base) valeriu@valeriu-PORTEGE-Z30-C:~/ESMValCore$ pixi shell
WARN Skipped running the post-link scripts because `run-post-link-scripts` = `false`
- bin/.librsvg-pre-unlink.sh
To enable them, run:
pixi config set --local run-post-link-scripts insecure
More info:
https://pixi.sh/latest/reference/pixi_configuration/#run-post-link-scripts
WARNING: Did not detect successful shell initialization within 3 second(s).
Please check on https://pixi.sh/latest/advanced/pixi_shell/#issues-with-pixi-shell for more tips.
er passphrase for /home/valeriu/.ssh/id_rsa_jasmin:
Identity added: /home/valeriu/.ssh/id_rsa_jasmin (/home/valeriu/.ssh/id_rsa_jasmin)
Agent pid 963160
Enter passphrase for /home/valeriu/.ssh/id_rsa_cloud:
valeriu@valeriu-PORTEGE-Z30-C:~/ESMValCore$ pixi config set --local run-post-link-scripts insecure
✅ Updated config at /home/valeriu/ESMValCore/.pixi/config.toml
valeriu@valeriu-PORTEGE-Z30-C:~/ESMValCore$ git status
On branch pixi
Your branch is up to date with 'origin/pixi'.
Untracked files:
(use "git add <file>..." to include in what will be committed)
.pixi/
nothing added to commit but untracked files present (use "git add" to track)
valeriu@valeriu-PORTEGE-Z30-C:~/ESMValCore$ pixi shell
WARNING: Did not detect successful shell initialization within 3 second(s).
Please check on https://pixi.sh/latest/advanced/pixi_shell/#issues-with-pixi-shell for more tips.
er passphrase for /home/valeriu/.ssh/id_rsa_jasmin:
Agent pid ...Why is it putting |
ah! It works now: You gotta tell folks to not have anything that waits for a password in their |
|
Thanks for reviewing V!
I suspect you may have been on a different branch than the one from this pull request the first time you ran that command, because
As the name suggests, the command |
I put it in |
|
note (am not sure how important it is) that a that don't show up in the pixi env: |
|
It was already there, now you've added it a second time. Maybe it is because you ran |
nope that was showing up as ungit before - leave it like that - twice is better 😁 |
There should be no need to do that |
my point is that we need a bit of doc with this what you told me, or to tell folks not to have any shell init with passwords in their |
it absolutely should be a need to do that when one wants to test if the package still builds well after changes to eg |
|
After changes to pyproject.toml, one can test that things work as expected by e.g. running To create a module for an HPC, you will probably want to do the same as in the Dockerfile: clone the repo in the location where the module software should be, checkout the released version, create a pixi environment, and put the activation script (entrypoint without the |
Not as straightforward as that: for one, Docker is not allowed on an HPC, two, Singularity comes with all manners of permission issues, and three -most important - a user will never have access to the pixi shell that an admin creates. forget about working with shells there, it's all paths to various bin dirs |
I meant: "follow the same steps", not "use the container".. The steps are here: Lines 9 to 12 in e0507ec |
Added in 94f10e0 |
just like Carroll Shelby told his chief mechanic that said she can come apart: one way to find out 😁 https://www.youtube.com/watch?v=5OfjrMWjrhs |
|
One thing I think you should mention is that people should not have multiple ESMValXXX dirs - and they have - lots of eg |
in this vein, how the heck does one create new named environments without duplicating the project ie the repo? Say I need an environment |
Pixi avoids the problem by using hardlinks, so it only keeps one copy of a file, even if it's used in multiple places: https://pixi.prefix.dev/latest/workspace/environment/#de-duplication |
Edit pyproject.toml, add the extra environment (plus any needed features) there and run |
ah! Found the solution - in pyproject.toml |
|
Ow man, you gonna get so many users (not technical ones, though) so confused with all these new procedures - they are still confused about using conda/mamba - pixi takes this one notch up the level of confusability. Pixi is a lot more under the hood technical than conda - I don't think our ESMValTool users will get used to it easily if they want to do the slightest bit of development, more than |
|
we should think along these lines when we explain Pixi to our users (prob best to structure the docs too, or the UET in their Tutorial) - lifted from ChatGPT that makes a very solid point: pixi is a lot more technical than conda, and I think non-technical users can struggle with it more than with conda, what do you think? You’re not wrong—but it’s a bit more nuanced than “Pixi is harder.” 🧠 The core difference
That shift alone makes Pixi feel more “technical,” especially at first. 👍 Where Conda is easier for non-technical users
👉 For beginners or casual users, Conda is usually easier to start with. 🤔 Where Pixi feels harder
👉 That does raise the barrier for non-technical users. ⚖️ But Pixi isn’t harder for no reasonPixi is optimizing for things Conda struggles with:
So:
🎯 My honest take
💡 Why Pixi feels more technicalIt’s not just complexity—it’s where the complexity lives:
That’s great for engineers, but unfamiliar to beginners. 🧭 Practical compromiseA lot of people end up doing:
|
|
Learning something new is always hard, and learning a new technical skill is more annoying if it's not your main interest but rather something you want to get over with as quickly as possible with a minimal time investment. So yes, this will require some initial time investment and that may be frustrating to some, but I believe it will save our contributors time in the long run. No more waiting for solving environments will be an immediate time gain, and the much better reproducibility is very helpful when doing science. While the observations above by ChatGPT above may be true in general, they are not for our project. Our contributors, at least the ones that want to change the dependencies, already need to understand two or more different configuration files: pyproject.toml for the Python dependencies and (multiple in the case of ESMValTool) environment.yml files for the conda-forge dependencies. Pixi simplifies that by making it possible to do all the configuration in a single file (pyproject.toml) with a single syntax. One thing that we could do to make the transition easier is to do it at the upcoming workshop. Like that, we will be able to help out anyone who is there, either in person or online, immediately. |
valeriupredoi
left a comment
valeriupredoi
left a comment
There was a problem hiding this comment.
I agree with you, bud!
One thing that we could do to make the transition easier is to do it at the ESMValGroup/Community#273. Like that, we will be able to help out anyone who is there, either in person or online, immediately.
I think that's a good idea too! At any rate, very many thanks to you for doing this work, bud - excuse my constant nagging during the review process, I am always trying to find holes in the plot so we can fix them when we review, rather than when we have users barking at us 🐶
2 participants
Description
Switch to pixi for managing the development conda environments.
See ESMValGroup/ESMValTool#4407 for an extensive description and ESMValGroup/ESMValTool#3997 for further background.
Links to documentation:
Before you get started
Checklist
It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.
To help with the number pull requests: