Error retrieving training data

[munechika-koyo]

Even though imas-core related features have not been officially confirmed yet, I would like to just note this issue.

Summary

I encountered the error when I used the imas.training module:

>>> import imas.training
>>> entry = imas.training.get_training_db_entry()

Error message

18:01:17 WARNING  On-disk data is stored in DD 3.39.0 which has a different major version than the requested DD version (4.0.0). IMAS-Python will convert the data automatically, but this does not cover allchanges. See https://imas-python.readthedocs.io/en/latest//multi-dd.html#conversion-of-idss-between-dd-versions @db_entry.py:606
---------------------------------------------------------------------------
DataEntryException                        Traceback (most recent call last)
Cell In[17], line 1
----> 1 entry = imas.training.get_training_db_entry()

File ~/Documents/cherab/imas/.pixi/envs/default/lib/python3.13/site-packages/imas/training.py:39, in get_training_db_entry()
     37 def get_training_db_entry() -> imas.DBEntry:
     38     """Open and return an ``imas.DBEntry`` pointing to the training data."""
---> 39     return _initialize_training_db(imas.DBEntry)

File ~/Documents/cherab/imas/.pixi/envs/default/lib/python3.13/site-packages/imas/training.py:30, in _initialize_training_db(DBEntry_cls)
     28 output_entry.create()
     29 for ids_name in ["core_profiles", "equilibrium"]:
---> 30     ids = entry.get(ids_name)
     31     with patch.dict("os.environ", {"IMAS_AL_DISABLE_VALIDATE": "1"}):
     32         output_entry.put(ids)

File ~/Documents/cherab/imas/.pixi/envs/default/lib/python3.13/site-packages/imas/db_entry.py:354, in DBEntry.get(self, ids_name, occurrence, lazy, autoconvert, ignore_unknown_dd_version, destination)
    303 def get(
    304     self,
    305     ids_name: str,
   (...)    311     destination: Optional[IDSToplevel] = None,
    312 ) -> IDSToplevel:
    313     """Read the contents of an IDS into memory.
    314 
    315     This method fetches an IDS in its entirety, with all time slices it may contain.
   (...)    352             core_profiles = imas_entry.get("core_profiles")
    353     """  # noqa
--> 354     return self._get(
    355         ids_name,
    356         occurrence,
    357         None,
    358         destination,
    359         lazy,
    360         autoconvert,
    361         ignore_unknown_dd_version,
    362     )

File ~/Documents/cherab/imas/.pixi/envs/default/lib/python3.13/site-packages/imas/db_entry.py:566, in DBEntry._get(self, ids_name, occurrence, parameters, destination, lazy, autoconvert, ignore_unknown_dd_version)
    563     raise IDSNameError(ids_name, self._ids_factory)
    565 # Note: this will raise an exception when the ids/occurrence is not filled:
--> 566 dd_version = self._dbe_impl.read_dd_version(ids_name, occurrence)
    568 # DD version sanity checks:
    569 if not dd_version:
    570     # No DD version stored in the IDS, load as if it was stored with
    571     # self.dd_version

File ~/Documents/cherab/imas/.pixi/envs/default/lib/python3.13/site-packages/imas/backends/imas_core/db_entry_al.py:326, in ALDBEntryImpl.read_dd_version(self, ids_name, occurrence)
    323     dd_version = read_ctx.read_data(dd_version_path, "", CHAR_DATA, 1)
    325 if time_mode not in IDS_TIME_MODES:
--> 326     raise DataEntryException(
    327         f"IDS {ids_name!r}, occurrence {occurrence} is empty."
    328     )
    329 return dd_version

DataEntryException: IDS 'equilibrium', occurrence 0 is empty.

Here, I used the imas-core personally compiled.

Detailed investigation

After checking the contents of imas.training, I just tried to run the following code:

import imas
from importlib.resources import files

assets_path = files(imas) / "assets/"
entry = imas.DBEntry(f"imas:ascii?path={assets_path}", "r")
for ids_name in ["core_profiles", "equilibrium"]:
	ids = entry.get(ids_name)

Then, I received nearly the same error message as above.

Also, When I executed the following code,

import imas
from importlib.resources import files

assets_path = files(imas) / "assets/"
for ids_name in ["core_profiles", "equilibrium"]:
	entry = imas.DBEntry(f"imas:ascii?path={assets_path}", "r")
	ids = entry.get(ids_name)

then I didn't see any error.

When I checked the value of time_mode at

IMAS-Python/imas/backends/imas_core/db_entry_al.py

Line 325 in 523bea2

if time_mode not in IDS_TIME_MODES:

it returned -999999999, which is probably identical to imas.ids_defs.EMPTY_INT.

I presume that the error might be related to al-core or imas-core.

Note

• This inspection took place on the local PC.
• I couldn't find any error on ITER SDCC with IMAS-Python module loading.

[maarten-ic]

Hi @munechika-koyo,

I presume that the error might be related to al-core or imas-core.

I expect so as well, given that your last code sample (which creates a new DBEntry for the equilibrium IDS) does work.

Which version of imas-python and imas-core do you use, and how did you compile imas-core?

import imas
import imas_core

print(f"{imas.__version__=}, {imas_core.version_tuple=}, {imas_core._al_lowlevel.get_al_version()=}")

[munechika-koyo]

Hi @maarten-ic,
The code you suggested resulted in the following:

imas.__version__='2.0.1', imas_core.version_tuple=(5, 4, 2), imas_core._al_lowlevel.get_al_version()='5.4.2'

I compiled imas-core using the rattler-build conda build tool.
The files required for rattler-build are attached below.

recipe.yaml
remove_numpy_constraint.patch
variants.yaml
build_core.sh
build_py.sh

[maarten-ic]

Hi @munechika-koyo,

Thanks for your response! I have no experience with rattler-build (or using conda environments in general), so cannot judge if the issue could be with the way you build imas-core...

If you'd like to further debug this issue, it could be worth running this in a debugger (such as gdb) and try to figure out why 🔒AsciiBackend::readData returns an EMPTY_INT for ids_properties/homogeneous_time in the equilibrium IDS in the first case, but not in the second one.
Debugging this could be a rabbit hole, and may not be worth diving into :)

For the record, the code works fine when I use a clean venv on my laptop and install both imas-python (from PyPI) and imas-core (from the ITER git):

$ python -m venv venv
$ . venv/bin/activate
$ pip install imas-python git+ssh://git@git.iter.org/imas/al-core.git
[...]
$ python
[...]
>>> import imas.training
>>> imas.training.get_training_db_entry()
10:55:19 WARNING  On-disk data is stored in DD 3.39.0 which has a different major version than the requested DD version (4.0.0). IMAS-Python will convert the data automatically, but this does not cover allchanges. See https://imas-python.readthedocs.io/en/latest//multi-dd.html#conversion-of-idss-between-dd-versions @db_entry.py:606
10:55:19 INFO     Parsing data dictionary version 3.39.0 @dd_zip.py:89
10:55:20 WARNING  On-disk data is stored in DD 3.39.0 which has a different major version than the requested DD version (4.0.0). IMAS-Python will convert the data automatically, but this does not cover allchanges. See https://imas-python.readthedocs.io/en/latest//multi-dd.html#conversion-of-idss-between-dd-versions @db_entry.py:606
<imas.db_entry.DBEntry object at 0x7230de6bde20>
>>> 

[olivhoenen]

@munechika-koyo in your build_core.sh you've set the Python bindings to OFF, why is that?

# Python bindings
-D AL_PYTHON_BINDINGS=OFF

[munechika-koyo]

@olivhoenen That is because I aim to produce the al-core library (including the al shared library, build_core.sh) and imas-core Python bindings (build_py.sh) separately.