---
title: "Architecture"
vignette: >
  %\VignetteIndexEntry{Architecture}
  %\VignetteEngine{quarto::html}
  %\VignetteEncoding{UTF-8}
knitr:
  opts_chunk:
    collapse: true
    comment: '#>'
---

```{r}
#| include: false
#| eval: false
library(gdalraster.windows)
```

This article explains the systems-level design behind `gdalraster.windows`: why the package exists, how the [GDAL](https://gdal.org/) runtime bundle is built, and why runtime activation works the way it does.

## The root problem

GDAL's [Algorithm API](https://gdal.org/en/stable/api/gdalalg_cpp.html) registers algorithms through static C++ constructors, file-scope objects whose constructors insert entries into a global registry when `libgdal` is loaded. Under some Windows toolchain states (notably Rtools/[MXE](https://mxe.cc/) builds where dependencies are static archives), the linker's dead-code elimination discarded those self-registration translation units, leaving the registry empty (see [firelab/gdalraster#826](https://github.com/firelab/gdalraster/issues/826)).

The visible symptom:

```{r}
#| eval: false
gdalraster::gdal_global_reg_names()
#> character(0)
```

The upstream fix landed in GDAL 3.12.2 ([OSGeo/gdal#13592](https://github.com/OSGeo/gdal/pull/13592)), and [`muparser`](https://beltoforion.de/en/muparser/) (required by parts of the Algorithm API) was added to the Rtools GDAL build in [release 6768](https://cran.r-project.org/bin/windows/Rtools/rtools45/news.html) (see [firelab/gdalraster#858](https://github.com/firelab/gdalraster/issues/858)). Until a fixed GDAL ships in the default toolchain, and for anyone needing a pinned, feature-rich GDAL, this package provides a known good runtime: a custom GDAL build plus the activation logic to load it reliably.

## Toolchain stack

| Term | What it is |
|------|------------|
| [MinGW-w64](https://www.mingw-w64.org/) | GCC-based toolchain producing native Win32 `.exe`/`.dll` (no POSIX emulation layer) |
| [MSYS2](https://www.msys2.org/) | Distribution + `pacman` package manager shipping several MinGW-w64 toolchain variants; the *build environment*, not the runtime target |
| [UCRT](https://learn.microsoft.com/en-us/cpp/porting/upgrade-your-code-to-the-universal-crt) | Microsoft's Universal C Runtime (default since VS2015); mixing binaries linked against different C runtimes is unsafe (incompatible heap allocators, `FILE*`, etc.) |
| [UCRT64](https://www.msys2.org/docs/environments/) | The MSYS2 environment targeting UCRT — the variant compatible with Rtools |
| [Rtools45](https://cran.r-project.org/bin/windows/Rtools/rtools45/rtools.html) | CRAN's Windows toolchain for R 4.5 and later; UCRT64/MinGW-based, so C++ ABI-compatible with MSYS2 UCRT64 builds of the same GCC line |

The invariant this package maintains: GDAL and `gdalraster` are both compiled with compatible MinGW/UCRT toolchains.

## Why `gdalraster` must be rebuilt from source

C++ has no standardized ABI across compilers. Name mangling, vtable layout, exception unwinding, and object layout all differ between MSVC and MinGW/GCC, and can drift between incompatible GCC configurations. [`gdalraster`](https://firelab.github.io/gdalraster/) binds to GDAL's C++ API through [Rcpp](https://www.rcpp.org/) (not a C `extern "C"` shim), so `gdalraster.dll` and `libgdal-*.dll` must come from the same ABI world.

That is why `install_gdalraster()` builds from source against the bundle's headers and import library rather than reusing the CRAN binary (which is linked — statically — against Rtools' own GDAL):

```{r}
#| eval: false
gdalraster.windows::install_gdalraster()
```

Internally this uses [`withr::with_makevars()`](https://withr.r-lib.org/reference/with_makevars.html), which writes a scoped Makevars file and points the `R_MAKEVARS_USER` environment variable at it for the duration of the install — compile/link flags never leak into your persistent configuration.

## Key build flags

From [`tools/build_gdal.sh`](https://github.com/jimbrig/gdalraster.windows/blob/main/tools/build_gdal.sh):

| Flag | Purpose |
|------|---------|
| `-DGDAL_USE_MUPARSER=ON` | Algorithm API support (expression evaluation) |
| `-DGDAL_USE_ARROW/PARQUET/HDF5/NETCDF/GEOS/SPATIALITE=ON` | Extended driver profile beyond the lean Rtools build |
| `-DGDAL_ENABLE_DRIVER_PDF=OFF` (+ poppler/podofo/pdfium `OFF`) | The MSYS2 `libpodofo.dll` fails its DLL initialization routine (Windows error 1114) in any process; a `libgdal` that imports it cannot be loaded at all, so the PDF driver is excluded from the bundle |
| `-DGDAL_HIDE_INTERNAL_SYMBOLS=ON` | Restricts the export table to the public API (PE/COFF DLLs cap export ordinals at 65,535 and GDAL's full symbol set exceeds it — see [OSGeo/gdal#4706](https://github.com/OSGeo/gdal/issues/4706)) |
| `-Wl,--kill-at` | Strips `@N` stdcall decoration from exports so symbol names match what loaders expect |
| `-static-libgcc -static-libstdc++` + whole-archive `winpthread` | Embeds the GCC runtime into the DLLs — end users need neither Rtools nor MSYS2 at runtime |

## Dependency closure: `collect_dlls.sh`

Producing `libgdal-*.dll` is half the job; it imports dozens of transitive DLLs ([GEOS](https://libgeos.org/), [PROJ](https://proj.org/), [Arrow](https://arrow.apache.org/)'s deep tree, [HDF5](https://www.hdfgroup.org/solutions/hdf5/), ...). [`tools/collect_dlls.sh`](https://github.com/jimbrig/gdalraster.windows/blob/main/tools/collect_dlls.sh) walks the full PE import tree with [`ntldd -R`](https://github.com/LRN/ntldd), copies every dependency that resolves to the UCRT64 prefix into the bundle's `bin/`, and **fails** if any non-Windows-system dependency remains unresolved. The bundle is therefore a verified-closed set: the only external imports are Windows system DLLs.

Final bundle layout:

```text
<gdal_home>/
├── bin/        libgdal-*.dll + closed transitive DLL set
├── include/    headers (compile-time, for install_gdalraster())
├── lib/        libgdal.dll.a import library (compile-time)
├── share/      gdal/ + proj/ runtime data
└── python/     osgeo_utils (pure-python, for embedded-python algorithms)
```

No executables ship in the bundle (`BUILD_APPS=OFF`): consumers are R
processes loading the DLL, not command-line users.

## Runtime activation: why loading needs help

Windows resolves a DLL's import table at load time using a fixed search order (executable directory, System32, then `PATH`). When `library(gdalraster)` loads `gdalraster.dll`, Windows immediately needs `libgdal-*.dll` — if the bundle's `bin/` is not discoverable at that exact moment, loading fails with `LoadLibrary failure`.

`activate_gdal_runtime()` handles this, session-scoped:

- prepends `<gdal_home>/bin` to `PATH`
- sets `GDAL_DATA`, `PROJ_LIB`, `PROJ_DATA` (GDAL and PROJ require their runtime data trees for CRS operations)
- prepends `<gdal_home>/python` to `PYTHONPATH` (next section)
- preloads the GDAL DLL via `dyn.load(..., local = FALSE, now = TRUE)` — mapping it into the process's loaded-module list *before* `gdalraster.dll` asks for it; `local = FALSE` loads into the global symbol namespace so subsequent DLLs resolve against it

## The embedded Python layer

Some GDAL algorithms (e.g. [`gdal driver gpkg validate`](https://gdal.org/en/stable/programs/gdal_driver_gpkg_validate.html)) are thin C++ entry points around Python implementations. At first use, `libgdal` locates a `python.exe` on `PATH`, dynamically loads the matching `libpython` DLL (no static CPython link), calls `Py_Initialize()`, and imports `osgeo_utils.samples.validate_gpkg`.

[`osgeo_utils`](https://github.com/OSGeo/gdal/tree/master/swig/python/gdal-utils) (GDAL's `gdal-utils` distribution) is pure Python — no compiled extension modules, hence no CPython version/ABI coupling. The bundle ships it under `<gdal_home>/python`, version-locked to the built GDAL tag, and activation exposes it via `PYTHONPATH`. The compiled `osgeo` SWIG bindings are deliberately **not** bundled: they would pin the bundle to a single CPython ABI, and the Python-implemented validators degrade gracefully without them.

```{r}
#| eval: false
alg <- gdalraster::gdal_alg(cmd = "driver gpkg validate")
alg$setArg("dataset", "file.gpkg")
alg$setArg("full-check", TRUE)
alg$run()
alg$output()
```

## Compile-time vs runtime paths

These are independent concerns, and conflating them is the most common source of confusion:

- **Compile-time** (during `install_gdalraster()`): `PKG_CPPFLAGS` points at `<gdal_home>/include`, `PKG_LIBS` at `<gdal_home>/lib` — scoped via `withr`.
- **Runtime** (every session): the Windows loader resolves DLLs through `PATH`/preload state — handled by `activate_gdal_runtime()`.

A successful link does not imply a loadable session, and vice versa.

## Reproducing the bundle

The bundle is built entirely from the repository — no local machine state. For a new GDAL release:

```bash
git tag gdal-v3.14.0 && git push origin gdal-v3.14.0
```

or dispatch the [build workflow](https://github.com/jimbrig/gdalraster.windows/actions/workflows/build.yml) with `gdal_version=v3.14.0`. The version string drives the GDAL source checkout, cache key, asset name (`gdal-ucrt64-v3.14.0-windows-x64.zip`), and release tag. CI cache keys hash the build scripts, so any build-logic change forces a fresh compile. Published bundles are available on the [releases page](https://github.com/jimbrig/gdalraster.windows/releases).

Local (non-CI) reproduction from an MSYS2 UCRT64 shell:

```bash
export GDAL_VER=v3.14.0
export INSTALL_DIR=/c/gdal-install
export BUNDLE_DIR=/c/gdal-bundle
bash tools/build_gdal.sh
bash tools/collect_dlls.sh
```

## Upstream references

- [firelab/gdalraster#826](https://github.com/firelab/gdalraster/issues/826) — algorithm-registry failure mode
- [firelab/gdalraster#982](https://github.com/firelab/gdalraster/issues/982) — working Windows workflow this package productized
- [OSGeo/gdal#13592](https://github.com/OSGeo/gdal/pull/13592) — upstream registration fix (GDAL 3.12.2)
- [Rtools45 news](https://cran.r-project.org/bin/windows/Rtools/rtools45/news.html)