diff --git a/docs/architecture/ROADMAP.md b/docs/architecture/ROADMAP.md index 12513fbbf..d9cddd36a 100644 --- a/docs/architecture/ROADMAP.md +++ b/docs/architecture/ROADMAP.md @@ -18,9 +18,9 @@ Legend: --- -# 1. Sample Model +# 1. Structure Model -## 1.1 Crystal Structure Parameters +## 1.1 Crystal Structure ### Space Group @@ -48,35 +48,51 @@ Legend: | Occupancy | ✅ | ✅ | | Symmetry _wyckoff_letter_ | ✅ | ✅ | -### Atomic Displacement Parameters (ADP) +### Atomic Displacement (ADP) -| Feature | LIB | APP | -| ----------------------------------------------- | --- | --- | -| Isotropic Biso | ✅ | 🗓 | -| Isotropic Uiso | 🚧 | ✅ | -| Anisotropic Bani _B11, B22, B33, B12, B13, B23_ | 🚧 | 🗓 | -| Anisotropic Uani _U11, U22, U33, U12, U13, U23_ | 🚧 | 🗓 | +| Feature | LIB | APP | +| --------------------------------------------------- | --- | --- | +| Isotropic _Biso_ | ✅ | 🗓 | +| Isotropic _Uiso_ | 🚧 | ✅ | +| Anisotropic _Bani_ (_B11, B22, B33, B12, B13, B23_) | 🚧 | 🗓 | +| Anisotropic _Uani_ (_U11, U22, U33, U12, U13, U23_) | 🚧 | 🗓 | --- -## 1.2 Magnetic Structure Parameters +## 1.2 Magnetic Structure - EPIC -| Feature | LIB | APP | -| ------------------------------------------------------- | --- | --- | -| EPIC (Magnetic space groups, unpolarized and polarized) | 🗓 | 🗓 | +| Feature | LIB | APP | +| ----------------------------------------------------- | --- | --- | +| Magnetic Space Groups | 🗓 | 🗓 | +| Irreducible representations | 🗓 | 🗓 | +| Magnetic propagation vector (_kx, ky, kz_) | 🗓 | 🗓 | +| Magnetic moments (_mx, my, mz_) | 🗓 | 🗓 | +| Local Susceptibility (_𝜒11, 𝜒22, 𝜒33, 𝜒12, 𝜒13, 𝜒23_) | 🗓 | 🗓 | --- # 2. Experiment Model -## 2.1 Powder Diffraction - -### Fitting Methods - -| Feature | LIB | APP | -| ------------------------------------- | --- | --- | -| Rietveld refinement (full pattern) | ✅ | ✅ | -| Le Bail refinement (profile matching) | 🗓 | 🗓 | +| Techniques | LIB | APP | +| ---------------------------------------------------- | ----- | ----- | +| 2.1. Powder Diffraction | ✅/🗓 | ✅/🗓 | +| 2.1.1. Common features | ✅/🗓 | ✅/🗓 | +| 2.1.2. Standard Bragg diffraction (CWL) | ✅/🗓 | ✅/🗓 | +| 2.1.2. Standard Bragg diffraction (TOF) | ✅/🗓 | ✅/🗓 | +| 2.1.3. Total Scattering (Pair-Distribution Function) | ✅/🗓 | 🗓 | +| 2.2. Single-Crystal Diffraction (CWL) | ✅/🗓 | ✅/🗓 | +| 2.2. Single-Crystal Diffraction (TOF) | ✅/🗓 | ✅/🗓 | +| 2.3. Polarized Powder Diffraction | 🗓 | 🗓 | +| 2.3.1. Flipping-rathio method (TOF) | 🗓 | 🗓 | +| 2.3.1. Flipping-rathio method (CWL) | 🗓 | 🗓 | +| 2.4. Polarized Single-Crystal Diffraction | 🗓 | 🗓 | +| 2.4.1. Flipping-rathio method (CWL) | 🗓 | 🗓 | +| 2.4.2. Flipping-rathio method (TOF) | 🗓 | 🗓 | +| 2.4.3. Spherical neutron polarimetry | 🗓 | 🗓 | + +## 2.1. Powder Diffraction + +## 2.1.1 Common features ### Linked Phases @@ -90,7 +106,14 @@ Legend: | ----------------------------------------- | --- | --- | | Multiple regions
_start/end positions_ | ✅ | 🗓 | ---- +## 2.1.1 Standard Bragg diffraction + +### Fitting Methods + +| Feature | LIB | APP | +| ------------------------------------- | --- | --- | +| Rietveld refinement (full pattern) | ✅ | ✅ | +| Le Bail refinement (profile matching) | 🗓 | 🗓 | ### Background @@ -130,20 +153,12 @@ Legend: ### Peak Profile — Time-of-Flight -CrysPy peak_shape options: - -- "Gauss": Jorgensen (back-to-back exponentials ⊗ Gaussian) -- "pseudo-Voigt": Jorgensen-Von Dreele (back-to-back exponentials ⊗ - pseudo-Voigt) -- "type0m": Double back-to-back exponentials ⊗ pseudo-Voigt (Z-Rietveld - type 0m) - -| Feature | LIB | APP | -| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --- | --- | -| Jorgensen (back-to-back exponentials ⊗ Gaussian)
_Gaussian broadening σ₀, σ₁, σ₂
Back-to-back exponential rise α₀, α₁. Back-to-back exponential decay β₀, β₁_
(CrysPy) | ✅ | ✅ | -| Jorgensen-Von Dreele (back-to-back exponentials ⊗ pseudo-Voigt)
_Gaussian broadening σ₀, σ₁, σ₂. Lorentzian broadening γ₀, γ₁, γ₂
Back-to-back exponential rise α₀, α₁. Back-to-back exponential decay β₀, β₁_
(CrysPy) | ✅ | ✅ | -| Double back-to-back exponentials ⊗ pseudo-Voigt [Z-Rietveld type0m]
_Gaussian broadening σ₀, σ₁, σ₂. Lorentzian broadening γ₀, γ₁, γ₂
Rise α₁, α₂. Fast decay β₀₀, β₀₁. Slow decay β₁₀. Switching r₀₁, r₀₂, r₀₃_
(CrysPy) | ✅ | 🗓 | -| Ikeda-Carpenter ⊗ pseudo-Voigt
_Moderator pulse α₀, α₁, β₀, κ
Gaussian broadening σ². Lorentzian broadening γ_
(CrysFML) | 🗓 | 🗓 | +| Feature | LIB | APP | +| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --- | --- | +| Jorgensen (back-to-back exponentials ⊗ Gaussian)
_Gaussian broadening σ₀, σ₁, σ₂
Back-to-back exponential rise α₀, α₁. Back-to-back exponential decay β₀, β₁_
(CrysPy "Gauss") | ✅ | ✅ | +| Jorgensen-Von Dreele (back-to-back exponentials ⊗ pseudo-Voigt)
_Gaussian broadening σ₀, σ₁, σ₂. Lorentzian broadening γ₀, γ₁, γ₂
Back-to-back exponential rise α₀, α₁. Back-to-back exponential decay β₀, β₁_
(CrysPy "pseudo-Voigt") | ✅ | ✅ | +| Double back-to-back exponentials ⊗ pseudo-Voigt [Z-Rietveld type0m]
_Gaussian broadening σ₀, σ₁, σ₂. Lorentzian broadening γ₀, γ₁, γ₂
Rise α₁, α₂. Fast decay β₀₀, β₀₁. Slow decay β₁₀. Switching r₀₁, r₀₂, r₀₃_
(CrysPy "type0m") | ✅ | 🗓 | +| Ikeda-Carpenter ⊗ pseudo-Voigt
_Moderator pulse α₀, α₁, β₀, κ
Gaussian broadening σ². Lorentzian broadening γ_
(CrysFML) | 🗓 | 🗓 | | TOF profile | TOF source | Performance | | ------------------------------------------------------------------- | ----------------------------------------------------------------- | ----------- | @@ -154,13 +169,13 @@ CrysPy peak_shape options: --- -## 2.1.2 Total Scattering (Pair Distribution Function) +## 2.1.3 Total Scattering (Pair Distribution Function) ### Peak Profile -| Feature | LIB | APP | -| ------------------------------------------------------------------------------------------------------ | --- | --- | -| GaussianDampedSinc type
_cutoff q. broadening q. sharpening δ₁, δ₂
damping q, particle diameter_ | ✅ | 🗓 | +| Feature | LIB | APP | +| ------------------------------------------------------------------------------------------------------------------------ | --- | --- | +| Gaussian-damped sinc termination function
_cutoff q. broadening q. sharpening δ₁, δ₂
damping q, particle diameter_ | ✅ | 🗓 | --- @@ -196,11 +211,20 @@ Gauss or Lorentz mosaicity distribution | ------------------------------------ | --- | --- | | Individual wavelength per reflection | ✅ | 🗓 | -## 2.3. Polarized Neutron Diffraction +## 2.3. Polarized Neutron Powder Diffraction - EPIC -| Feature | LIB | APP | -| ---------------------------------------------- | --- | --- | -| EPIC (powders and single crystals, FR and SNP) | 🗓 | 🗓 | +| Feature | LIB | APP | +| ---------------------------- | --- | --- | +| Flipping-rathio method (TOF) | 🗓 | 🗓 | +| Flipping-rathio method (CWL) | 🗓 | 🗓 | + +## 2.3. Polarized Neutron Single Crystal Diffraction - EPIC + +| Feature | LIB | APP | +| ----------------------------- | --- | --- | +| Flipping-rathio method (TOF) | 🗓 | 🗓 | +| Flipping-rathio method (CWL) | 🗓 | 🗓 | +| Spherical neutron polarimetry | 🗓 | 🗓 | --- @@ -215,19 +239,21 @@ Gauss or Lorentz mosaicity distribution # 4. Analysis (Fitting) -### Refinement Algorithms +### Refinement Algorithms (numerical derivatives) -| Feature | LIB | APP | -| --------------------------------------------------------------- | --- | --- | -| Levenberg–Marquardt (numerical derivatives)
LMFIT minimizer | ✅ | ✅ | -| Levenberg–Marquardt (analytical derivatives)
LMFIT minimizer | 🗓 | 🗓 | -| Derivative-free minimization
DFO-LS minimizer | ✅ | ✅ | -| Bayesian analysis
BUMPS minimizer | 🗓 | 🗓 | +| Feature | LIB | APP | +| ---------------------------------------------------- | --- | --- | +| Levenberg–Marquardt
LMFIT minimizer | ✅ | ✅ | +| Levenberg–Marquardt
LMFIT minimizer (scipy-based) | ✅ | ✅ | +| Levenberg–Marquardt
BUMPS minimizer | 🚧 | 🗓 | +| Derivative-free minimization
DFO-LS minimizer | ✅ | ✅ | +| Bayesian analysis
BUMPS minimizer | 🗓 | 🗓 | ### Fit Strategies | Feature | LIB | APP | | ---------------------------------------------------------------------------------------------------- | --- | --- | +| Single fit of one experimental data block to one/multiple structural data block | ✅ | ✅ | | Sequential fit of experimental data blocks | ✅ | 🗓 | | Joint fit of experimental data blocks within the same calculation engine | ✅ | 🗓 | | Joint fit of experimental data blocks using different calculation engines
(e.g. CrysPy + Pdffit2) | ✅ | 🗓 | @@ -292,8 +318,8 @@ Gauss or Lorentz mosaicity distribution | Feature | LIB | APP | CLI | | --------------------------------- | --- | --- | --- | -| List available tutorial notebooks | — | — | ✅ | -| Download tutorial notebooks | — | — | ✅ | +| List available tutorial notebooks | ✅ | — | ✅ | +| Download tutorial notebooks | ✅ | — | ✅ | --- @@ -371,12 +397,10 @@ Gauss or Lorentz mosaicity distribution --- -# 10. Future Topics - -Here, we list features that are not sorted into the above categories, -but are still on our radar for future development. +# 10. Unsorted features - Restrains (soft constraints, e.g. bond lengths, angles) +- Refinement using analytical derivatives - Global optimization algorithms (e.g. simulated annealing) - Incommensurate structures - 2D Rietveld refinement diff --git a/src/easydiffraction/utils/utils.py b/src/easydiffraction/utils/utils.py index 346392640..b44d3e109 100644 --- a/src/easydiffraction/utils/utils.py +++ b/src/easydiffraction/utils/utils.py @@ -25,10 +25,25 @@ pooch.get_logger().setLevel('WARNING') # Suppress pooch info messages +_DATA_REPO = 'easyscience/diffraction' +_DATA_ROOT = 'data' # commit SHA preferred -_DATA_INDEX_REF = '010c69546fa9ec1bd998bdcaa902e1df4f5d10af' +_DATA_INDEX_REF = '927f96547a80c3328f43f2a69cb9c8048286bcb7' # macOS: sha256sum index.json -_DATA_INDEX_HASH = 'sha256:9449dbba0475158bbce9dea1fbb1e5e596c1f63d41fc136a3e3f5d677c5c6779' +_DATA_INDEX_HASH = 'sha256:301d6aafdc1ccf5f97d2edb491a6b350f6195f05106f8f38c9bf5530e592c8ec' + + +def _build_data_url(path: str) -> str: + path = path.lstrip('/') + return f'https://raw.githubusercontent.com/{_DATA_REPO}/{_DATA_INDEX_REF}/{_DATA_ROOT}/{path}' + + +def _record_path(record: dict) -> str: + if 'path' in record: + return record['path'] + + msg = "Index record must contain 'path' key." + raise KeyError(msg) def _validate_url(url: str) -> None: @@ -51,9 +66,13 @@ def _validate_url(url: str) -> None: raise ValueError(msg) -def _filename_for_id_from_url(data_id: int | str, url: str) -> str: - """Return local filename using the extension from the URL.""" - suffix = pathlib.Path(urlparse(url).path).suffix # includes leading dot ('.cif', '.xye', ...) +def _filename_for_id_from_path(data_id: int | str, record_path: str) -> str: + """ + Return local filename using the extension from the record path. + """ + suffix = pathlib.PurePosixPath( + record_path + ).suffix # includes leading dot ('.cif', '.xye', ...) # If URL has no suffix, fall back to no extension. return f'ed-{data_id}{suffix}' @@ -74,10 +93,7 @@ def _normalize_known_hash(value: str | None) -> str | None: def _fetch_data_index() -> dict: """Fetch and cache the diffraction data index.json.""" - index_url = ( - 'https://raw.githubusercontent.com/easyscience/diffraction/' - f'{_DATA_INDEX_REF}/data/index.json' - ) + index_url = _build_data_url('index.json') _validate_url(index_url) destination_dirname = 'easydiffraction' @@ -170,11 +186,10 @@ def download_data( raise KeyError(msg) record = index[key] - url = record['url'] + record_path = _record_path(record) + url = _build_data_url(record_path) _validate_url(url) - - known_hash = _normalize_known_hash(record.get('hash')) - fname = _filename_for_id_from_url(id, url) + fname = _filename_for_id_from_path(id, record_path) dest_path = pathlib.Path(destination) dest_path.mkdir(parents=True, exist_ok=True) @@ -197,6 +212,8 @@ def download_data( log.debug(f"Data #{id} already present at '{file_path}', but will be overwritten.") file_path.unlink() + known_hash = _normalize_known_hash(record.get('hash')) + # Pooch downloads to destination with our controlled filename. pooch.retrieve( url=url, diff --git a/tests/unit/easydiffraction/test___init__.py b/tests/unit/easydiffraction/test___init__.py index e42e801c4..086efa88b 100644 --- a/tests/unit/easydiffraction/test___init__.py +++ b/tests/unit/easydiffraction/test___init__.py @@ -53,7 +53,7 @@ def test_lazy_functions_execute_with_monkeypatch(monkeypatch, capsys, tmp_path): fake_index = { '12': { - 'url': 'https://example.com/data.xye', + 'path': 'data.xye', 'hash': 'sha256:...', 'description': 'Demo dataset', } @@ -72,4 +72,4 @@ def fake_retrieve(**kwargs): result = utils.download_data(id=12, destination=str(tmp_path), overwrite=True) assert Path(result).exists() - assert calls['kwargs']['url'] == 'https://example.com/data.xye' + assert calls['kwargs']['url'] == utils._build_data_url('data.xye') diff --git a/tests/unit/easydiffraction/utils/test_utils_coverage.py b/tests/unit/easydiffraction/utils/test_utils_coverage.py index d4ae58532..9357e2725 100644 --- a/tests/unit/easydiffraction/utils/test_utils_coverage.py +++ b/tests/unit/easydiffraction/utils/test_utils_coverage.py @@ -26,30 +26,37 @@ def test_validate_url_accepts_https(): MUT._validate_url('https://example.com/file.cif') -# --- _filename_for_id_from_url ------------------------------------------------ +# --- _filename_for_id_from_path ----------------------------------------------- -def test_filename_for_id_from_url_with_extension(): +def test_filename_for_id_from_path_with_extension(): import easydiffraction.utils.utils as MUT - result = MUT._filename_for_id_from_url(12, 'https://example.com/data/file.xye') + result = MUT._filename_for_id_from_path(12, 'file.xye') assert result == 'ed-12.xye' -def test_filename_for_id_from_url_cif_extension(): +def test_filename_for_id_from_path_cif_extension(): import easydiffraction.utils.utils as MUT - result = MUT._filename_for_id_from_url('3', 'https://example.com/path/model.cif') + result = MUT._filename_for_id_from_path('3', 'path/model.cif') assert result == 'ed-3.cif' -def test_filename_for_id_from_url_no_extension(): +def test_filename_for_id_from_path_no_extension(): import easydiffraction.utils.utils as MUT - result = MUT._filename_for_id_from_url(7, 'https://example.com/path/noext') + result = MUT._filename_for_id_from_path(7, 'path/noext') assert result == 'ed-7' +def test_record_path_raises_for_missing_path_key(): + import easydiffraction.utils.utils as MUT + + with pytest.raises(KeyError, match="Index record must contain 'path' key"): + MUT._record_path({'url': 'https://example.com/data.xye'}) + + # --- _normalize_known_hash ---------------------------------------------------- @@ -322,7 +329,7 @@ def test_tof_to_d_linear_negative_tof_minus_offset_gives_nan(): def test_download_data_unknown_id(monkeypatch): import easydiffraction.utils.utils as MUT - fake_index = {'1': {'url': 'https://example.com/data.xye', 'hash': None}} + fake_index = {'1': {'path': 'data.xye', 'hash': None}} monkeypatch.setattr(MUT, '_fetch_data_index', lambda: fake_index) with pytest.raises(KeyError, match='Unknown dataset id=999'): MUT.download_data(id=999) @@ -333,7 +340,7 @@ def test_download_data_already_exists_no_overwrite(monkeypatch, tmp_path, capsys fake_index = { '1': { - 'url': 'https://example.com/data.xye', + 'path': 'data.xye', 'hash': None, 'description': 'Test data', } @@ -355,7 +362,7 @@ def test_download_data_success(monkeypatch, tmp_path, capsys): fake_index = { '1': { - 'url': 'https://example.com/data.xye', + 'path': 'data.xye', 'hash': None, 'description': 'Test data', } @@ -383,7 +390,7 @@ def test_download_data_overwrite_existing(monkeypatch, tmp_path, capsys): fake_index = { '1': { - 'url': 'https://example.com/data.xye', + 'path': 'data.xye', 'hash': None, 'description': 'Test data', } @@ -411,7 +418,7 @@ def test_download_data_no_description(monkeypatch, tmp_path, capsys): fake_index = { '1': { - 'url': 'https://example.com/data.xye', + 'path': 'data.xye', 'hash': 'sha256:...', } }