Skip to content

Fix data access to external resources #155

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
AndrewSazonov merged 4 commits into from
Apr 14, 2026
+132 −84
Merged

Fix data access to external resources #155

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
689c5b7
Refactor data access to use 'path' instead of 'url' in index records
AndrewSazonov Apr 14, 2026
b6f4f36
Apply PR feedback for path-based data index helpers
AndrewSazonov Apr 14, 2026
b4f61c4
Update data index reference to latest commit SHA
AndrewSazonov Apr 14, 2026
abe823a
Update roadmap documentation
AndrewSazonov Apr 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
138 changes: 81 additions & 57 deletions docs/architecture/ROADMAP.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,9 @@ Legend:

---

# 1. Sample Model
# 1. Structure Model

## 1.1 Crystal Structure Parameters
## 1.1 Crystal Structure

### Space Group

Expand Down Expand Up @@ -48,35 +48,51 @@ Legend:
| Occupancy | ✅ | ✅ |
| Symmetry _wyckoff_letter_ | ✅ | ✅ |

### Atomic Displacement Parameters (ADP)
### Atomic Displacement (ADP)

| Feature | LIB | APP |
| ----------------------------------------------- | --- | --- |
| Isotropic Biso | ✅ | 🗓 |
| Isotropic Uiso | 🚧 | ✅ |
| Anisotropic Bani _B11, B22, B33, B12, B13, B23_ | 🚧 | 🗓 |
| Anisotropic Uani _U11, U22, U33, U12, U13, U23_ | 🚧 | 🗓 |
| Feature | LIB | APP |
| --------------------------------------------------- | --- | --- |
| Isotropic _Biso_ | ✅ | 🗓 |
| Isotropic _Uiso_ | 🚧 | ✅ |
| Anisotropic _Bani_ (_B11, B22, B33, B12, B13, B23_) | 🚧 | 🗓 |
| Anisotropic _Uani_ (_U11, U22, U33, U12, U13, U23_) | 🚧 | 🗓 |

---

## 1.2 Magnetic Structure Parameters
## 1.2 Magnetic Structure - EPIC

| Feature | LIB | APP |
| ------------------------------------------------------- | --- | --- |
| EPIC (Magnetic space groups, unpolarized and polarized) | 🗓 | 🗓 |
| Feature | LIB | APP |
| ----------------------------------------------------- | --- | --- |
| Magnetic Space Groups | 🗓 | 🗓 |
| Irreducible representations | 🗓 | 🗓 |
| Magnetic propagation vector (_kx, ky, kz_) | 🗓 | 🗓 |
| Magnetic moments (_mx, my, mz_) | 🗓 | 🗓 |
| Local Susceptibility (_𝜒11, 𝜒22, 𝜒33, 𝜒12, 𝜒13, 𝜒23_) | 🗓 | 🗓 |

---

# 2. Experiment Model

## 2.1 Powder Diffraction

### Fitting Methods

| Feature | LIB | APP |
| ------------------------------------- | --- | --- |
| Rietveld refinement (full pattern) | ✅ | ✅ |
| Le Bail refinement (profile matching) | 🗓 | 🗓 |
| Techniques | LIB | APP |
| ---------------------------------------------------- | ----- | ----- |
| 2.1. Powder Diffraction | ✅/🗓 | ✅/🗓 |
| 2.1.1. Common features | ✅/🗓 | ✅/🗓 |
| 2.1.2. Standard Bragg diffraction (CWL) | ✅/🗓 | ✅/🗓 |
| 2.1.2. Standard Bragg diffraction (TOF) | ✅/🗓 | ✅/🗓 |
| 2.1.3. Total Scattering (Pair-Distribution Function) | ✅/🗓 | 🗓 |
| 2.2. Single-Crystal Diffraction (CWL) | ✅/🗓 | ✅/🗓 |
| 2.2. Single-Crystal Diffraction (TOF) | ✅/🗓 | ✅/🗓 |
| 2.3. Polarized Powder Diffraction | 🗓 | 🗓 |
| 2.3.1. Flipping-rathio method (TOF) | 🗓 | 🗓 |
| 2.3.1. Flipping-rathio method (CWL) | 🗓 | 🗓 |
| 2.4. Polarized Single-Crystal Diffraction | 🗓 | 🗓 |
| 2.4.1. Flipping-rathio method (CWL) | 🗓 | 🗓 |
| 2.4.2. Flipping-rathio method (TOF) | 🗓 | 🗓 |
| 2.4.3. Spherical neutron polarimetry | 🗓 | 🗓 |

## 2.1. Powder Diffraction

## 2.1.1 Common features

### Linked Phases

Expand All @@ -90,7 +106,14 @@ Legend:
| ----------------------------------------- | --- | --- |
| Multiple regions<br>_start/end positions_ | ✅ | 🗓 |

---
## 2.1.1 Standard Bragg diffraction

### Fitting Methods

| Feature | LIB | APP |
| ------------------------------------- | --- | --- |
| Rietveld refinement (full pattern) | ✅ | ✅ |
| Le Bail refinement (profile matching) | 🗓 | 🗓 |

### Background

Expand Down Expand Up @@ -130,20 +153,12 @@ Legend:

### Peak Profile — Time-of-Flight

CrysPy peak_shape options:

- "Gauss": Jorgensen (back-to-back exponentials ⊗ Gaussian)
- "pseudo-Voigt": Jorgensen-Von Dreele (back-to-back exponentials ⊗
pseudo-Voigt)
- "type0m": Double back-to-back exponentials ⊗ pseudo-Voigt (Z-Rietveld
type 0m)

| Feature | LIB | APP |
| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --- | --- |
| Jorgensen (back-to-back exponentials ⊗ Gaussian)<br>_Gaussian broadening σ₀, σ₁, σ₂<br>Back-to-back exponential rise α₀, α₁. Back-to-back exponential decay β₀, β₁_<br>(CrysPy) | ✅ | ✅ |
| Jorgensen-Von Dreele (back-to-back exponentials ⊗ pseudo-Voigt)<br>_Gaussian broadening σ₀, σ₁, σ₂. Lorentzian broadening γ₀, γ₁, γ₂<br>Back-to-back exponential rise α₀, α₁. Back-to-back exponential decay β₀, β₁_<br>(CrysPy) | ✅ | ✅ |
| Double back-to-back exponentials ⊗ pseudo-Voigt [Z-Rietveld type0m]<br>_Gaussian broadening σ₀, σ₁, σ₂. Lorentzian broadening γ₀, γ₁, γ₂<br>Rise α₁, α₂. Fast decay β₀₀, β₀₁. Slow decay β₁₀. Switching r₀₁, r₀₂, r₀₃_<br>(CrysPy) | ✅ | 🗓 |
| Ikeda-Carpenter ⊗ pseudo-Voigt<br>_Moderator pulse α₀, α₁, β₀, κ<br>Gaussian broadening σ². Lorentzian broadening γ_<br>(CrysFML) | 🗓 | 🗓 |
| Feature | LIB | APP |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --- | --- |
| Jorgensen (back-to-back exponentials ⊗ Gaussian)<br>_Gaussian broadening σ₀, σ₁, σ₂<br>Back-to-back exponential rise α₀, α₁. Back-to-back exponential decay β₀, β₁_<br>(CrysPy "Gauss") | ✅ | ✅ |
| Jorgensen-Von Dreele (back-to-back exponentials ⊗ pseudo-Voigt)<br>_Gaussian broadening σ₀, σ₁, σ₂. Lorentzian broadening γ₀, γ₁, γ₂<br>Back-to-back exponential rise α₀, α₁. Back-to-back exponential decay β₀, β₁_<br>(CrysPy "pseudo-Voigt") | ✅ | ✅ |
| Double back-to-back exponentials ⊗ pseudo-Voigt [Z-Rietveld type0m]<br>_Gaussian broadening σ₀, σ₁, σ₂. Lorentzian broadening γ₀, γ₁, γ₂<br>Rise α₁, α₂. Fast decay β₀₀, β₀₁. Slow decay β₁₀. Switching r₀₁, r₀₂, r₀₃_<br>(CrysPy "type0m") | ✅ | 🗓 |
| Ikeda-Carpenter ⊗ pseudo-Voigt<br>_Moderator pulse α₀, α₁, β₀, κ<br>Gaussian broadening σ². Lorentzian broadening γ_<br>(CrysFML) | 🗓 | 🗓 |

| TOF profile | TOF source | Performance |
| ------------------------------------------------------------------- | ----------------------------------------------------------------- | ----------- |
Expand All @@ -154,13 +169,13 @@ CrysPy peak_shape options:

---

## 2.1.2 Total Scattering (Pair Distribution Function)
## 2.1.3 Total Scattering (Pair Distribution Function)

### Peak Profile

| Feature | LIB | APP |
| ------------------------------------------------------------------------------------------------------ | --- | --- |
| GaussianDampedSinc type<br>_cutoff q. broadening q. sharpening δ₁, δ₂<br>damping q, particle diameter_ | ✅ | 🗓 |
| Feature | LIB | APP |
| ------------------------------------------------------------------------------------------------------------------------ | --- | --- |
| Gaussian-damped sinc termination function<br>_cutoff q. broadening q. sharpening δ₁, δ₂<br>damping q, particle diameter_ | ✅ | 🗓 |

---

Expand Down Expand Up @@ -196,11 +211,20 @@ Gauss or Lorentz mosaicity distribution
| ------------------------------------ | --- | --- |
| Individual wavelength per reflection | ✅ | 🗓 |

## 2.3. Polarized Neutron Diffraction
## 2.3. Polarized Neutron Powder Diffraction - EPIC

| Feature | LIB | APP |
| ---------------------------------------------- | --- | --- |
| EPIC (powders and single crystals, FR and SNP) | 🗓 | 🗓 |
| Feature | LIB | APP |
| ---------------------------- | --- | --- |
| Flipping-rathio method (TOF) | 🗓 | 🗓 |
| Flipping-rathio method (CWL) | 🗓 | 🗓 |

## 2.3. Polarized Neutron Single Crystal Diffraction - EPIC

| Feature | LIB | APP |
| ----------------------------- | --- | --- |
| Flipping-rathio method (TOF) | 🗓 | 🗓 |
| Flipping-rathio method (CWL) | 🗓 | 🗓 |
| Spherical neutron polarimetry | 🗓 | 🗓 |

---

Expand All @@ -215,19 +239,21 @@ Gauss or Lorentz mosaicity distribution

# 4. Analysis (Fitting)

### Refinement Algorithms
### Refinement Algorithms (numerical derivatives)

| Feature | LIB | APP |
| --------------------------------------------------------------- | --- | --- |
| Levenberg–Marquardt (numerical derivatives)<br>LMFIT minimizer | ✅ | ✅ |
| Levenberg–Marquardt (analytical derivatives)<br>LMFIT minimizer | 🗓 | 🗓 |
| Derivative-free minimization<br>DFO-LS minimizer | ✅ | ✅ |
| Bayesian analysis<br>BUMPS minimizer | 🗓 | 🗓 |
| Feature | LIB | APP |
| ---------------------------------------------------- | --- | --- |
| Levenberg–Marquardt<br>LMFIT minimizer | ✅ | ✅ |
| Levenberg–Marquardt<br>LMFIT minimizer (scipy-based) | ✅ | ✅ |
| Levenberg–Marquardt<br>BUMPS minimizer | 🚧 | 🗓 |
| Derivative-free minimization<br>DFO-LS minimizer | ✅ | ✅ |
| Bayesian analysis<br>BUMPS minimizer | 🗓 | 🗓 |

### Fit Strategies

| Feature | LIB | APP |
| ---------------------------------------------------------------------------------------------------- | --- | --- |
| Single fit of one experimental data block to one/multiple structural data block | ✅ | ✅ |
| Sequential fit of experimental data blocks | ✅ | 🗓 |
| Joint fit of experimental data blocks within the same calculation engine | ✅ | 🗓 |
| Joint fit of experimental data blocks using different calculation engines<br>(e.g. CrysPy + Pdffit2) | ✅ | 🗓 |
Expand Down Expand Up @@ -292,8 +318,8 @@ Gauss or Lorentz mosaicity distribution

| Feature | LIB | APP | CLI |
| --------------------------------- | --- | --- | --- |
| List available tutorial notebooks | — | — | ✅ |
| Download tutorial notebooks | — | — | ✅ |
| List available tutorial notebooks | ✅ | — | ✅ |
| Download tutorial notebooks | ✅ | — | ✅ |

---

Expand Down Expand Up @@ -371,12 +397,10 @@ Gauss or Lorentz mosaicity distribution

---

# 10. Future Topics

Here, we list features that are not sorted into the above categories,
but are still on our radar for future development.
# 10. Unsorted features

- Restrains (soft constraints, e.g. bond lengths, angles)
- Refinement using analytical derivatives
- Global optimization algorithms (e.g. simulated annealing)
- Incommensurate structures
- 2D Rietveld refinement
43 changes: 30 additions & 13 deletions src/easydiffraction/utils/utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,25 @@

pooch.get_logger().setLevel('WARNING') # Suppress pooch info messages

_DATA_REPO = 'easyscience/diffraction'
_DATA_ROOT = 'data'
# commit SHA preferred
_DATA_INDEX_REF = '010c69546fa9ec1bd998bdcaa902e1df4f5d10af'
_DATA_INDEX_REF = '927f96547a80c3328f43f2a69cb9c8048286bcb7'
# macOS: sha256sum index.json
_DATA_INDEX_HASH = 'sha256:9449dbba0475158bbce9dea1fbb1e5e596c1f63d41fc136a3e3f5d677c5c6779'
_DATA_INDEX_HASH = 'sha256:301d6aafdc1ccf5f97d2edb491a6b350f6195f05106f8f38c9bf5530e592c8ec'


def _build_data_url(path: str) -> str:
path = path.lstrip('/')
return f'https://raw.githubusercontent.com/{_DATA_REPO}/{_DATA_INDEX_REF}/{_DATA_ROOT}/{path}'

Comment thread
AndrewSazonov marked this conversation as resolved.

def _record_path(record: dict) -> str:
if 'path' in record:
return record['path']

msg = "Index record must contain 'path' key."
raise KeyError(msg)
Comment thread
AndrewSazonov marked this conversation as resolved.


def _validate_url(url: str) -> None:
Expand All @@ -51,9 +66,13 @@ def _validate_url(url: str) -> None:
raise ValueError(msg)


def _filename_for_id_from_url(data_id: int | str, url: str) -> str:
"""Return local filename using the extension from the URL."""
suffix = pathlib.Path(urlparse(url).path).suffix # includes leading dot ('.cif', '.xye', ...)
def _filename_for_id_from_path(data_id: int | str, record_path: str) -> str:
"""
Return local filename using the extension from the record path.
"""
suffix = pathlib.PurePosixPath(
record_path
).suffix # includes leading dot ('.cif', '.xye', ...)
# If URL has no suffix, fall back to no extension.
return f'ed-{data_id}{suffix}'

Expand All @@ -74,10 +93,7 @@ def _normalize_known_hash(value: str | None) -> str | None:

def _fetch_data_index() -> dict:
"""Fetch and cache the diffraction data index.json."""
index_url = (
'https://raw.githubusercontent.com/easyscience/diffraction/'
f'{_DATA_INDEX_REF}/data/index.json'
)
index_url = _build_data_url('index.json')
_validate_url(index_url)

destination_dirname = 'easydiffraction'
Expand Down Expand Up @@ -170,11 +186,10 @@ def download_data(
raise KeyError(msg)

record = index[key]
url = record['url']
record_path = _record_path(record)
url = _build_data_url(record_path)
_validate_url(url)

known_hash = _normalize_known_hash(record.get('hash'))
fname = _filename_for_id_from_url(id, url)
fname = _filename_for_id_from_path(id, record_path)

dest_path = pathlib.Path(destination)
dest_path.mkdir(parents=True, exist_ok=True)
Expand All @@ -197,6 +212,8 @@ def download_data(
log.debug(f"Data #{id} already present at '{file_path}', but will be overwritten.")
file_path.unlink()

known_hash = _normalize_known_hash(record.get('hash'))

# Pooch downloads to destination with our controlled filename.
pooch.retrieve(
url=url,
Expand Down
4 changes: 2 additions & 2 deletions tests/unit/easydiffraction/test___init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ def test_lazy_functions_execute_with_monkeypatch(monkeypatch, capsys, tmp_path):

fake_index = {
'12': {
'url': 'https://example.com/data.xye',
'path': 'data.xye',
'hash': 'sha256:...',
'description': 'Demo dataset',
}
Expand All @@ -72,4 +72,4 @@ def fake_retrieve(**kwargs):

result = utils.download_data(id=12, destination=str(tmp_path), overwrite=True)
assert Path(result).exists()
assert calls['kwargs']['url'] == 'https://example.com/data.xye'
assert calls['kwargs']['url'] == utils._build_data_url('data.xye')
Loading
Loading