Super resolving with Sentinel-2

The Sentinel-2 constellation revisits every spot on Earth roughly every 5 days. Over a single cloud-free summer at a mid-latitude site, that’s around 30 looks at the same patch of ground from the same sensor family. Although the delivered L2A products live on a fixed 10m UTM grid, the scene content is not perfectly co-registered across dates. Residual processing and viewing-geometry effects show up as sub-pixel translations between products: typically around 1.5m of standard deviation per axis, and occasionally as much as 6 to 7m. That small difference is what classical multi-frame super-resolution exploits. Given many slightly shifted samples of the same scene, you can solve for the hidden high-resolution image that, once shifted, blurred, and downsampled the way the sensor would, reproduces every observation.

To do this, we represent the ground as a continuous field of 2D Gaussian splats on a fine grid, push it through an analytic forward model, and let a gradient based optimizer (LBFGS) jointly recover the scene weights and per-observation shifts. The implementation is around 500 lines of PyTorch, and the optimizer rediscovers the sub-pixel jitter on its own.

Background: the point-spread function

Every camera and sensor blurs a single point of light into a small fuzzy spot. That spot is the point-spread function, or PSF. It comes from the optics, the atmosphere, and the sensor’s photodetector response.

For a satellite imager, the PSF is typically wider than a single pixel on the ground. The optics smear features smaller than the PSF across several pixels before the sensor digitizes anything, so post-processing alone cannot put that information back without an external prior.

The cleanest analogy is a long-exposure photo of an out-of-focus star: stacking more exposures averages down the noise, but never recovers what’s inside the blurred disk.

For Sentinel-2’s 10m bands, the residual blur after pixel aperture is approximately a 2D Gaussian with a standard deviation of roughly 3 to 5m.¹ Without an external or learned prior, you should not expect to reliably recover arbitrary features much below roughly the 8–10 m scale from Sentinel-2.

Our forward model includes the PSF as an explicit known-Gaussian blur. The assumed PSF width matters more than any other knob. Too narrow and the model invents detail that isn’t there. Too wide and it can’t resolve anything.

The setup

We pull 32 cloud-free L2A scenes from April–October 2025 over a 987 × 1011 pixel area at 10m resolution from the Microsoft Planetary Computer. Most ablations and visual examples below use a centred 256 × 256 pixel crop (2.56 × 2.56 km) of that scene. We co-register four bands (B02, B03, B04, B08, corresponding to Blue, Green, Red, and NIR) into a single time series. Esri World Imagery (~0.8m/px) over the same AOI is used only as a visual reference.

The model treats all observations as views of one latent scene. That only works when temporal variation (phenology, illumination, atmosphere, BRDF) is small relative to the spatial structure we care about. A tighter seasonal window or per-date radiometric correction would help on dynamic landscapes.

The job is an inverse problem: we never see the high-resolution scene directly; we see 32 low-resolution measurements . The forward model that maps the hidden scene to each measurement is

where is a per-observation sub-pixel warp parameterized by a 2D shift vector (the small residual misregistration of L2A product , in meters), is a Gaussian point-spread function (PSF) capturing the sensor’s residual optical blur, and is average-pooling down to the native 10m grid. We want to find the scene and the per-pass shifts that best explain all 32 observations.

The key design choice is how to represent . A learnable dense pixel grid works but tends to alias. Instead, we use 2D Gaussian splats: a continuous field of Gaussian blobs on a regular grid finer than the 10m pixel size. Each splat has a learnable per-band weight; we keep positions and widths fixed. The splat basis is naturally smooth, and a Gaussian integrated against another Gaussian has a closed form, which makes the forward pass cheap.

The forward model

The forward model has a closed form. A single observed pixel at position on pass is the integral of the sensor-blurred scene over the pixel’s footprint:

where folds the splat width and the sensor PSF into a single effective Gaussian (the convolution of two Gaussians is a wider Gaussian). Because the integrand factors along the two axes, the double integral collapses into a product of erf² differences:

That’s the whole forward model: each predicted pixel is a sum over splats of weight × erf-difference-x × erf-difference-y. PyTorch differentiates erf analytically, so we get exact gradients with respect to both the splat weights and the per-observation shifts , with no image discretization.

The structure is also separable along the row and column axes, so rendering reduces to two batched matrix multiplications per band. Even at the default 3 m splat spacing — roughly 730k splats over a 256 × 256 crop — a single render takes milliseconds.

The shifts come from the data

We don’t have to tell the optimizer where the sub-pixel shifts are. We initialize for every observation, let LBFGS jointly optimize the scene weights and the shifts against the data fidelity loss, and the per-pass offsets fall out. Each is a single global translation per observation, which works for our small, relatively flat AOI; larger or more mountainous areas would likely need local shifts or a full deformation field.

We can sanity-check the recovered shifts against an independent estimator. Running skimage.phase_cross_correlation with 100× upsampling on the same observation pairs gives shifts that don’t go through our forward model at all. The two sets agree closely.

Two methods landing on the same numbers is good evidence that the jitter is a real physical signal in the data, not just the optimizer fitting noise.

What you can actually resolve

We swept the assumed sensor PSF width, , across a realistic range for S2’s 10m bands:

The bolded row was our sweet spot on this scene — the sharpest reconstruction without visible overfitting. At the final row’s settings the optimizer began fitting the noise as scene structure. “Mean LBFGS shift from init” is the mean adjustment away from the phase-correlation seed, not the absolute recovered shift.

The published S2 modulation transfer function suggests –4m for the 10m bands, which lands us in the bolded row of the table: in our reconstructions we saw meaningful sharpening down to roughly 8m features from 10m data, and visibly cleaner edges than bicubic upsampling. When we pushed below 3m the optimizer began manufacturing structure the data doesn’t support – overfitting that looked plausible until we compared it against the aerial reference.

There is no neural network filling in plausible-looking texture. What you see is close to what the 32 observations support under the assumed forward model.

More observations reduce noise

Across our N-sweep, adding observations produced cleaner images but didn’t recover new spatial frequencies.

With the explicit sensor model, the fit stopped improving at about the same point whether we used 8, 16, or all 32 observations. More looks average down temporal variation from phenology, illumination, and atmosphere, but the 1.5m sub-pixel jitter is small relative to the 3 to 5m PSF — in this regime, multi-temporal fusion bought us SNR more than it bought us resolution.

Use a few well-chosen observations for sharper edges; add more only when you need denoising.

Fast in practice

The whole pipeline runs on a single GPU. At the 3 m default splat spacing, a 256 × 256 crop (~730k splats) fits in 30–150 seconds depending on configuration, and the full 987 × 1011 scene (~11M splats) fits in around 5 minutes. The biggest single speedup comes from handing the joint optimization to LBFGS after a brief Adam warmup:

A few hundred Adam steps find a sane neighborhood – LBFGS’s quasi-Newton update can struggle with poor initialization – then LBFGS takes over. We found that with exact gradients through erf, LBFGS converged quickly and stably across the configurations we tried, and that the only knob that meaningfully changed outputs was . Varying the rest of the regularization (TV weight, splat spacing, ) had little visual impact in our experiments.

Conclusion

This formulation is a good fit when you want a per-scene, training-free reconstruction that runs end-to-end in minutes on a single GPU and handles multiple bands without extra cost. It is most useful when downstream users need to trust the pixels — change detection over time, visual inspection, or basemap production for areas without sub-meter coverage.

If you genuinely need sub-meter recovery from 10m data, you need something else. A learned prior trained on paired low- and high-resolution data (diffusion models, ESRGAN, and similar approaches), or an external structural reference like an aerial basemap or a sharper sensor (PlanetScope, SkySat). We have experimented with basemap-conditioned versions of this same optimizer and they look sharper, but the sharpness comes from the basemap, not the S2 stack.

The code is around 500 lines of PyTorch, with the analytic erf-based renderer on the bottom and a thin Adam-then-LBFGS loop on top.

Bibliography

[1] Irani, M. and Peleg, S. “Improving resolution by image registration.” CVGIP: Graphical Models and Image Processing, 1991.

[2] Razzak, M. T., Mateo-García, G., Lecuyer, G., Gómez-Chova, L., Gal, Y., and Kalaitzis, F. “Multi-Spectral Multi-Image Super-Resolution of Sentinel-2 with Radiometric Consistency Losses and Its Effect on Building Delineation.” ISPRS Journal of Photogrammetry and Remote Sensing, 2023.

[3] Aybar, C., et al. “A radiometrically and spatially consistent super-resolution framework for Sentinel-2.” Remote Sensing of Environment, 2026.

[4] Sirko, W., Asiedu Brempong, E., Marcos, J. T. C., Annkah, A., Korme, A., Hassen, M. A., Sapkota, K., Shekel, T., Diack, A., Nevo, S., Hickey, J., and Quinn, J. “High-Resolution Building and Road Detection from Sentinel-2.”, 2024.

Methods

We ran every (model × precision × compile-mode × input-shape) combination on each GPU, recorded forward-pass latency, and divided by total wall-clock time to get images/sec. We fix batch size at 512 across the matrix.³ A few details about the protocol:

• We isolate the model from the dataloader. A pre-allocated batch sits on the GPU; we time only the model forward pass plus cuda.synchronize(). Real pipelines have host→device transfer and disk I/O too, but we deliberately exclude both so what’s left is just the model and the precision (we revisit the dataloader cost in Dataloader overhead).
• We warm up before timing. Twenty warmup iterations clear the cudnn autotuner and any JIT compilation; then peak-memory stats reset and timing starts. Each cell runs for at least 30 seconds. Throughput is total images divided by total wall time, not the mean of per-iteration rates.
• The GPU has to be idle. The benchmark harness checks for other processes on the GPU and aborts if it finds any.
• Encoder-only. Every model runs as the classification backbone alone, no segmentation decoder attached, so hierarchical CNNs and plain ViTs pay the same accounting.
• What “fp32” means depends on the GPU. On Ampere-and-newer GPUs (A100, H100), our fp32 rows are actually TF32 — 10-bit-mantissa Tensor Cores; on V100 they’re true IEEE-754 fp32. A tf32_enabled column in the CSV disambiguates. (See the dtype primer below for what TF32 means.)

Models cover 33 architectures spanning five families — classical CNNs (ResNet, EfficientNet, ConvNeXt, MobileNetV3, RegNetY), plain ViTs (ViT, DeiT3, DinoV3), hierarchical ViTs (Swin, BEiT), and CNN-ViT hybrids (CoAtNet) — plus 12 encoders from five recent geospatial foundation model families⁴: DOFA (Xiong et al. 2024), CROMA (Fuller et al., NeurIPS 2023), SenPaMAE (Prexl & Schmitt, DAGM 2024), Galileo (Tseng et al., ICML 2025), and OlmoEarth (Herzog et al., CVPR 2026). Precisions are fp32, fp16, bf16⁵, and amp. Compile modes are none and default (torch.compile with default Inductor settings).⁶

We benchmark each timm model across input sizes that match the geo-FMs we want to compare it against ({64, 120, 128, 144, 224}-pixel side, {2, 3, 10, 12} channels) so we can do shape-for-shape comparisons. We run each geo-FM at its pretraining shape: DOFA-{B,L}/16 at 3×224×224, CROMA-Optical at 12×120×120, CROMA-SAR at 2×120×120, SenPaMAE-B/16 at 3×144×144, Galileo-{Nano,Base,Large}/8 at 10×64×64, OlmoEarth-{Nano,Tiny,Base,Large}/8 at 12×128×128.⁷

Results

Dataloader overhead

Inference throughput numbers are notoriously hard to compare across reports because the measurement protocol is rarely consistent. The most common confound is the input pipeline — how each batch gets to the GPU before the model runs.

To isolate the dataloader cost from any specific image decoder or augmentation stack, we wrote benchmark_dataloader_ipc.py: it runs ResNet-18 against a best-case Dataset that returns a cached random tensor for every __getitem__ — no image decoding, no augmentation, no disk I/O. What’s left in the pipeline is just batching, worker-to-main-process IPC, pinning, and the host→GPU copy. For each batch size and worker count, we measure three timings:

• compute — model forward through a pre-allocated GPU batch, plus cuda.synchronize(). Upper bound, no dataloader involved.
• fetch-only — one batch out of the DataLoader, no model. Pure pipeline cost.
• end-to-end — fetch-only + host→GPU copy + compute. The realistic path.

For a single 1024×3×224×224 batch on V100 with ResNet-18 at num_workers=8, those three paths cost:

The fetch-only path (~222 ms) costs about as much as a full ResNet-18 forward pass (~218 ms) — and a chunk of the total wall time even after overlap. If two benchmark reports use different num_workers, different pin_memory settings, different host/device topology, or different storage backends, their throughput numbers measure different things. Worse, these confounds tend to compress the spread between models — if every backbone is bottlenecked on host→device transfer, ResNet-18 and ViT-L/16 look closer than they really are.

Appendix: precision, compile, and cross-GPU effects

The headline numbers in this post all use each model’s best precision + compile config. This appendix unpacks how that “best” gets there — which families gain the most from torch.compile and half precision, how the GPU type interacts with all of it, and how the geo-FMs respond to the same axes.

The speedup you get from going to half precision and turning on torch.compile is not uniform across model families — and the family-by-family pattern is different on V100 than on H100. The table below is the median speedup over the fp32 + no-compile baseline at 224×3, across all models in each family:

Links

• Code: github.com/calebrob6/throughput-bench
• Interactive viewer: calebrob.com/throughput-bench
• Results CSVs: results/ (V100, H100 NVL)

Study area and data

We used the AEF Mosaic — a global Zarr v3 archive of AlphaEarth embedding fields on S3, built by Jeff Albrecht in collaboration with Taylor Geospatial, with annual composites from 2017–2025. Each pixel is a 64-dimensional signed int8 vector aligned to the Sentinel-2 10m grid.

The per-pixel temporal difference (emb_2024 − emb_2020) gives a 64-d float32 feature vector at every location. Pixels where the surface didn’t change cluster near zero; changed pixels project to non-zero directions in that 64-d space, and the experiment below tests how well a linear model can read those directions back out.

Compressing the difference vectors

Our working hypothesis from pt. 1 is that most of the nominal dimensions in an earth embedding are redundant — the intrinsic dimensionality is much lower — and PCA is the tool we use to collapse down to a compact basis. DeltaBit is a test of whether that same idea holds for difference vectors, where we stand to gain the most from it. We fit PCA on a 1-in-1000 subsample of the full raster diff (~181K pixels) and looked at how variance is distributed across the 64 components.

Variance is broadly distributed. PC1 captures only 18.3%; you need 9 components to cross 50% and 40 to cross 90%. This is much flatter than the static-embedding case from our compression post, where AEF reaches 80% variance in 8 dimensions. One hypothesis: differencing partly cancels the dominant scene-level modes — land cover, broad vegetation state — that two nearby years share, and what’s left is a more isotropic mix of higher-frequency change signals and embedding noise. We haven’t validated that directly.

PCA-8 retains less than half the variance. That’s a strong test of whether per-pixel change detection survives aggressive compression. Variance is not the same as predictive signal, but if the model needs the long tail, PCA-8 will lose it.

Change detection with pixel supervision

We labeled 336 pixels by hand — 190 change, 146 no-change — using a side-by-side Sentinel-2 swipe of 2020 vs 2024 true-color imagery. At each labeled pixel we sample the 2020 and 2024 embeddings, compute the diff, and project it through three feature pipelines:

• PCA-3 — first 3 principal components (3-d)
• PCA-8 — first 8 principal components (8-d)
• Full diff — the raw 64-d difference, no PCA

For each, we run logistic regression with 10-fold stratified nested cross-validation: an outer 10-fold for unbiased performance estimation, and an inner 5-fold sweep to pick the regularization strength on F1.¹

10-fold stratified CV on 336 hand-labeled pixels. Inner 5-fold for C selection on F1. Mean ± std across outer folds. Bolds mark column-wise bests on the metrics that drive the comparison; precision saturates near 1.0 across all three feature sets, so it isn’t a useful tiebreaker.

A few things worth pulling out:

PCA-8 is the practical sweet spot. 96% F1 with 8 features is more than enough for an interactive viewer. The demo’s compressed tiles — int8-quantized PCA-8 at 8 bytes per pixel — are 8× smaller than the source AEF int8 embeddings that went in (and 32× smaller than the raw float32 diff the CV ran on), at essentially no cost to what you can do on top of them. The CV here is on float32 PCA-8; pt1 already showed int8 quantization on top of PCA is statistically free, so the in-browser tiles inherit the same number.

Precision is consistently ≥0.97. The model rarely flags no-change pixels as change. The error mode is missed changes: recall climbs from 0.85 (PCA-3) to 0.94 (PCA-8) as feature dimensionality grows.

PCA-3 is useful even without a model. The first three components already reach 90% F1 on a linear probe, which means the change signal is concentrated enough that just looking at the top 3 PCs — mapped directly to R/G/B — surfaces changed pixels by eye. DeltaBit exposes this as an overlay toggle at the top of the labeling panel, and changed areas jump out with no training at all:

The middle row is what DeltaBit serves. The same logistic regression evaluated above gets re-fit in the browser on whatever you click, in a few hundred milliseconds. 8 bytes per pixel also means the entire Seattle scene fits in 2.60 GiB of XYZ tiles at zoom levels 8–14. The browser fetches GeoTIFFs on demand with GeoTIFF.js, caches them per tile key, and runs a WebGPU dot product over each tile to score every pixel.

What 96% F1 actually looks like, on a rural area south of Seattle:

The same flow, end to end in the viewer — 14 clicks is enough to stand up a deforestation detector on this area:

We zoom into a rural patch south of Seattle, place 8 change clicks on cleared parcels and 6 no-change clicks on untouched forest, hit Train, and switch to the heatmap view. The model generalizes cleanly across the local area, lighting up the other clearings we didn’t label.

One thing we noticed — and which in retrospect had to work this way: the lower zoom levels in the pyramid are built by 2×2-averaging the int8 PCA-8 tiles directly, and the same trained linear model produces coherent change predictions at those zooms. Zoom out in the viewer and cleared parcels and construction still light up. The whole pipeline is linear — differencing, PCA, and the model’s pre-sigmoid logit all commute with averaging — so the logit of an averaged tile equals the mean of its four child logits, modulo a bit of int8 rounding. The sigmoid mildly distorts that on the probability side, but not enough to blur the signal. It’s a free side effect of keeping every stage before the classifier linear.

You can try this for yourself — open DeltaBit, pick a spot in the Seattle metro, click a dozen change and no-change pixels, and see the prediction layer update across the visible map. Arrow keys / scroll to fly around; the same tiles get reused at every zoom level.

Limitations

Single scene pair. One Sentinel-2 tile, one year pair. We haven’t tested other sensors, resolutions, or temporal baselines.

Narrow slice of change types. The 336 labeled pixels are dominated by deforestation and urban changes — new construction, road work, cleared parcels — because that’s what’s conspicuous in a 2020→2024 side-by-side over Seattle. Agricultural cycles, flooding, fires, snow and ice, and seasonal vegetation shifts are barely represented, so the CV scores don’t speak to how well a linear model separates those change modes from each other (or from no-change).

Tested on AEF only. The recipe is model-agnostic — PCA + int8 + XYZ tiles works on any per-pixel embedding — but we only ran the experiment on AEF. We picked AEF because a global Zarr store already exists and we had code to pull from it; the bottleneck was data access, not the pipeline. That friction is decreasing fast. As more per-pixel embedding products ship as global mega-Zarrs, COGs, or APIs, running the same PCA(8) + int8 + XYZ recipe against them becomes a configuration change rather than a project. A global change-detection app over any of those sources isn’t inconceivable — just unbuilt.

Takeaways

PCA-8 + int8 is the sweet spot for interactive change detection. A full Sentinel-2 scene’s worth of change embeddings fits in a few GB of tiles — small enough to stream to a browser and run dense inference on the GPU without any server involvement, while still matching what a model trained on the raw embeddings can do. Concretely: 8 bytes per pixel gets 96% F1, which is 8× smaller than the source AEF int8 embeddings (32× smaller than the float32 diff the CV ran on). The recipe is model-agnostic: DINOv3, OlmoEarth, Tessera, or any other per-pixel embedding from pt. 1 would slot in the same way.

Difference embeddings compress less neatly than static ones — and it doesn’t matter. Subtracting two years of embeddings produces a messier, higher-dimensional signal than either year alone, but enough of what matters for change detection survives the first handful of components. Static AEF reaches 80% variance in 8 dims; the 2020→2024 diff needs 40 components to clear 90%, and PCA-8 only retains 48%. Despite that, PCA-8 already hits 96% F1 on the labeled set — variance isn’t the same as predictive signal, and 8 bytes per pixel is the right place to draw the line for an interactive viewer.

Linear is enough. The embeddings are doing the heavy lifting, so a simple classifier on top is all you need — which, as a bonus, is trivially fast to train and run on the GPU in the browser. Logistic regression on 8 features hits 96% F1; nothing fancier is needed for the webapp to feel like a useful tool.

Links: DeltaBit demo · pt. 1: Compressing Earth Embeddings · pt. 2: TerraBit

TerraBit

To test this, we built TerraBit — a global retrieval demo that runs entirely in the browser with no backend or server-side computation. We binary-quantize the full Clay v1.5 corpus into packed bit vectors, store them as spatially-partitioned cloud-native Parquet on public object storage, and let the browser handle shard discovery, data fetching, and in-memory Hamming scoring. The entire “backend” is a static S3 bucket; all compute happens on your machine.

How it works:

1. You draw one or more regions of interest (ROI) anywhere — each of these are loaded independently; regions can be rectangles or freehand polygons
2. You click to create exemplar patches on the map (one or many); positive exemplars outside the AOI have their embeddings fetched on the fly; negatives work anywhere on the globe for contrastive scoring (pos_dist − neg_dist); you can also invert search (bitwise NOT) to find the opposite of a reference!
3. DuckDB-WASM queries a manifest for intersecting geohash shards; only those shards are fetched via HTTP range requests — no full-corpus scan
4. A Web Worker scores all candidates with brute-force Hamming distance and returns ranked results
5. The results render via MapLibre GL across several view modes (top-k, heatmap, threshold, outlier, surprise, gradient)

Multiple exemplars can be combined via mean distance, by applying bitwise AND / OR / XOR directly on the packed binary vectors before scoring — exact, lossless ops that compose semantically because binary embeddings have nice arithmetic properties.

The 50M embeddings are partitioned into geohash-aligned Parquet shards and published on Source Cooperative, which serves them cloud-natively out of S3 — public HTTP with byte-range support, no egress fees, no intermediate server. A single manifest file records the path, row count, and spatial extent of every shard.

When you draw an ROI, DuckDB-WASM queries the manifest with a bounding-box predicate — manifest-based shard pruning: the manifest acts as a coarse spatial index so the browser never opens metadata on shards outside the ROI. Once the intersecting shard list is resolved, DuckDB streams those shard files over HTTP (via httpfs range requests) and applies a second filter at the row level — a bbox predicate for rectangles, or ST_Intersects for freehand polygons — to extract only patches within the drawn region. Ranking over the candidate slice is exact brute-force Hamming: binary embeddings arrive as packed Uint8Array columns (128 bytes per 1024-dim vector) and are scored in a Web Worker via XOR+popcount, which maps directly to hardware-accelerated popcount instructions and completes in milliseconds for a typical AOI partition.

The binary embeddings are lossy though — we find they have ~65% recall@10 (the fraction of true float32 nearest neighbors recovered by the binary representation) which means roughly a third of true neighbors are missed (Figure 2). Good enough for coarse exploration; not a claim about downstream curation or labeling productivity. How coarse is too coarse though? 65% recall goes further than you’d expect — try the demo on your own region!

A few examples of what this enables: a) click a center-pivot irrigation field in Kansas and separate it from rectangular fields across the state, b) pick a greenhouse cluster in Rotterdam and highlight dense greenhouse and vineyard complexes across the region or c) select a solar installation in northwest India and find others at similar scale. None of these queries require labeled data, a trained classifier, or even a definition of what you’re looking for beyond a single click. This is useful for data exploration, bootstrapping training datasets for supervised models, and narrowing the search space before running expensive high-resolution models over targeted areas. The demo also supports exporting ranked candidates as GeoParquet!

Binary Earth Embedding Retrieval at Planet Scale

Clay v1.5 produces 1024-dimensional embeddings from Sentinel-2 imagery. The global corpus spans two years of observations — roughly 50 million embeddings covering Earth’s land surface — and is 183 GiB on disk in ZSTD-compressed Parquet (≈190 GiB as raw float32 – float32s don’t compress well even if they come from a GeoFM). Serving float32 vectors at this scale to a browser isn’t viable; the question we ask is how aggressively you can compress without destroying retrieval quality.

Binary quantization reduces each dimension to a single sign bit. 1024 floats (4,096 bytes) become 128 bytes — a 32× reduction on the raw payload. End-to-end on disk (Parquet with ZSTD, geometry and STAC metadata columns, row-group overhead), the full 49.8M-row corpus drops from 182.9 GiB to 11.1 GiB — 16.5× compression. The on-disk number is what you pay for on object storage (32× is the raw payload reduction). The web demo corpus is smaller still (~7 GiB) because several columns were dropped and the compression level was increased — a demo-specific optimization on top of the 16.5× quantization win.

Why does aggressive quantization work at all on 1024-dimensional vectors? One diagnostic is the intrinsic dimension (ID) — the degrees of freedom the data actually uses, regardless of ambient dimensionality [Facco et al., 2017; Levina & Bickel, 2004]. This framing is directly motivated by Rao et al., 2025, who find that geographic representations — despite operating in 256–512 dimensional spaces — compress to just 2–10 intrinsic dimensions, and that ID correlates with downstream task performance. For Clay v1.5 we estimate ID ≈ 13–17 (MLE: 17.0, TwoNN: 12.6, Local PCA: 17.0, on a 10k sample subset). Three estimators with different assumptions agree on a narrow range. Low ID is why aggressive compression is worth attempting — the data simply isn’t using most of its dimensions.

TurboQuant aka rotate before you quantize

Binary is the extreme end of the compression spectrum, and the retrieval demo uses it — but what if you need more recall than binary while keeping storage well below float32?

Standard affine quantization at low bit-widths (int2–int4) suffers from high variance disparity across embedding dimensions: some dimensions carry far more signal than others, and a uniform quantization grid wastes bits on low-variance dimensions while clipping high-variance ones. TurboQuant fixes this by applying a fixed random orthogonal rotation (sampled once from a Haar-distributed ensemble via QR decomposition) before symmetric affine quantization: . The rotation spreads variance across dimensions so no channel dominates the bit budget. is generated once, stored with the quantized embeddings, and reused for all queries — one matrix multiply at encode/decode, no retraining.

Earth embeddings have the same property: ID ≈ 13–17 in a 1024-d space leaves a lot of variance to redistribute.

We ran TurboQuant across bit-widths on the Clay v1.5 embeddings. The gains are largest at low precision and vanish at high precision:

Practical takeaway: if binary recall is too coarse but you still want aggressive compression, TurboQuant at int2–int4 is worth trying first. At TurboQuant int4, 95% recall at ~6× on-disk compression (8× on the raw payload). By int8, the affine grid is fine enough on its own and the rotation adds nothing.

Search throughput

We also benchmarked brute-force kNN on a 1M-vector subset (1K queries, k=10) using FAISS on CPU and PyTorch’s torch.cdist on an RTX 3090. While other bit-widths benefit from GPU acceleration, binary search is unreasonably fast on CPU thanks to SIMD acceleration.

Why not just build a backend?

Reasonable reaction: “Cool demo, but real systems need a database and an API.” Maybe — but in geospatial ML, the gap between a working prototype and a deployed tool is almost all infrastructure: vector databases, REST APIs, auth, scaling, monitoring. Each layer is individually reasonable but collectively large enough of a barrier to prevent someone from shipping and maintaining a useful tool.

Furthermore, existing vector DBs primarily partition by embedding similarity; a small AOI query still touches shards scattered across the index with geospatial filtering applied AFTER the expensive approximate nearest neighbor (ANN) step. Getting geo-first partitioning right takes careful co-design, and no existing systems target zero-ops browser-native serving of a static corpus. Our approach sidesteps that: embeddings partitioned spatially by geohash, a manifest for shard pruning, and a throwaway Hamming scan.

To be clear, backends still have their place. Full-corpus ANN, multi-user serving, auth, and strict SLAs are backend territory. But for exploration and dataset curation, the barrier to useful interaction with embeddings should be as close to zero as possible, and for a lot of real problems, client-side is enough.

Links: TerraBit retrieval demo · binarized embedding corpus · pt. 1: Compressing Earth Embeddings

The storage problem

Earth’s land surface covers about 150 million km^2. At Sentinel-2’s 10m resolution, that’s 1.5 trillion pixels. Multiply by embedding dimension and bytes per element:

For context, the entire Sentinel-2 archive — every L1C and L2A product collected since 2015 — was roughly 22 PB in 2022 [5] and exceeded 50 PB by mid-2025 [6]. The archive grows by 3-4 PB per year. A single year of 1024d float32 embeddings (6.1 PB) would exceed the annual Sentinel-2 data volume that produced them. The embeddings are larger than the source imagery.

The embeddings are larger than the source imagery.

And these are per-year numbers. AlphaEarth covers 2017-2025 (9 years). Tessera plans the same. Multi-year archives at these scales reach tens of petabytes even for compact models. So how much can you compress before the embeddings stop being useful?

EO representations are redundant

Two recent papers provide evidence that earth observation representations carry substantial redundancy.

Model-level redundancy. Hackel et al. [7] applied post-hoc “slimming” to remote sensing foundation models — uniformly reducing the width of transformer layers after training. At just 1% of the original FLOPs, these models retained over 71% of their full-scale accuracy (relative retention). An ImageNet-trained MAE dropped below 10% relative retention under the same treatment. Intermediate model sizes sometimes outperformed the full model, suggesting the extra capacity adds noise rather than signal. If the intermediate representations are this redundant, the output embeddings are too.

Image-level redundancy. Papazafeiropoulos et al. [8] applied patch-level masking during training and inference of a ViT model, retaining only a fraction of image patches. On BigEarthNet, 15% patch retention achieved 99.4% of baseline accuracy. Even segmentation tolerated 50% patch removal while recovering ~97% of full performance.

These results suggest that standard embedding compression methods — including quantization and dimensionality reduction — may be effective for remotely sensed data as well. So we tested it!

Experimental setup

We evaluate combinations of quantization (float32, int8, int4, int2, binary, ternary, product quantization) and dimensionality reduction (PCA, truncated SVD, random projection, feature selection) across 5 embedding models and 6 classification datasets.

All experiments in this section use EuroSAT [9] — a 10-class Sentinel-2 land cover dataset with 21,600 images — as the primary benchmark. We use the precomputed embeddings for AEF, OlmoEarth, and Tessera from Isaac’s geopool repository, and we compute DINOv3 and ResNet50 embeddings separately. Then, we validate our findings on 5 additional datasets (RESISC45 [10] and 4 GeoBench [11] benchmarks) with the DINOv3 and ResNet50 based embeddings in the cross-dataset section below.

int8 is always free

The simplest compression: reduce each float32 value to int8. For each dimension, compute the min and max across the training set, then linearly map the range into 256 integer levels (4x compression).

EuroSAT kNN accuracy. Bold row = statistically indistinguishable from float32 baseline.

We find int8 is never statistically distinguishable from float32. McNemar’s test gives p >= 0.12 for every model-dataset pair (smallest p = 0.12). The 95% bootstrap confidence interval for the accuracy difference is within +/-0.2% everywhere. There is no reason to store float32 embeddings.

There is no reason to store float32 embeddings.

We also find int4 loses less than 1% for AEF and DINOv3. Binary quantization (1 bit per dimension, 32x compression) is worth a closer look — DINOv3 at 128 bytes still hits 91.8%! More on this below.

Most embedding dimensions are redundant

PCA variance analysis reveals different spectral structures across models:

Cumulative variance explained by top-k PCA components, fitted on EuroSAT training embeddings.

OlmoEarth-nano spreads its variance more broadly than Tessera, with 77% in 4 dimensions and needing 32 dimensions for 98%. DINOv3 distributes variance more evenly still, needing 256 dimensions for 97%.

DINOv3 spreads its variance across many dimensions, so you might expect it to compress poorly — if no dimension is dispensable, PCA can’t help. But DINOv3 at PCA(64)+int8 (6% of its original dimensions) still hits 93.1% kNN accuracy, only 1.4% below baseline. The dimensions PCA discards carry variance but apparently not much task-relevant information.

Combined compression: the Pareto frontier

The best configurations combine PCA with quantization — reduce dimensions first, then quantize:

EuroSAT accuracy. kNN: k=5, cosine. Linear: logistic regression, C tuned.

The two plots tell different stories. For kNN, AEF dominates at every storage budget — its 64 dimensions are compact enough that int8 at 64 bytes is nearly unbeatable, and larger models can’t overcome the dimensionality tax. For linear probes, DINOv3 pulls ahead once budgets exceed ~16 bytes, because a trained classifier can exploit the richer representation even after PCA compression.

PCA(64)+int8 at 64 bytes/embedding is the sweet spot for DINOv3: 64x compression with only 1.4% kNN loss and 96.4% linear accuracy. That brings a year of global DINOv3 embeddings from 6.1 PB down to 96 TB — the same footprint as AlphaEarth’s native int8 representation. Which model to choose depends on your task: kNN retrieval favors AEF, classification with a trained head favors DINOv3.

Binary quantization on DINOv3

DINOv3 loses only 2.7% kNN accuracy under binary quantization (1 bit per dimension, 32x compression), while AEF and Tessera lose 5.7-6.2%. We hypothesize that this might be due to:

1. High dimensionality. 1,024 binary dimensions give 2^1024 possible codes — enormous capacity for separating 10 classes.

2. Balanced dimensions. DINOv3’s dimensions are nearly symmetric around their means (average imbalance = 0.018). Each threshold bit carries close to 1 bit of entropy. OlmoEarth-nano is also well-balanced (0.052), while AEF’s higher imbalance (0.082) means many bits are nearly constant.

A related finding with the binary quantizations: Hamming distance on raw bits outperforms reconstructing float32 vectors and computing cosine distance. The reconstruction step replaces each bit with a centroid value (the mean of all above-threshold or below-threshold values for that dimension). We find that KNN with a Hamming distance (count the differing bits between the two vectors) outperforms using cosine distance on the reconstructed vectors. This seems to preserve the ranking of neighbor distances better:

Hamming distance is also significantly faster to compute than cosine distance on reconstructed vectors — it reduces to a popcount on XOR’d bit vectors.

Cross-dataset consistency

The compression patterns hold across all 6 datasets:

• int8 is effectively lossless on all datasets, including the 45-class RESISC45.
• PCA(64)+int8 at 64 bytes gives 93.1% on EuroSAT (10 classes) and 82.0% on RESISC45 (45 classes) — proportionally similar retention.
• m-forestnet (deforestation driver classification) is the hardest task at ~40% kNN for DINOv3 and ~36% for ResNet50 — likely because RGB-only embeddings lose the spectral bands needed for this task.

Per-class failure modes

Under aggressive compression, specific classes are disproportionately affected:

Highway and PermanentCrop are consistently the most affected — narrow categories relying on fine-grained spectral or spatial features that aggressive quantization destroys. If your application needs balanced per-class performance (rare category detection, e.g.), avoid extreme compression and verify per-class metrics.

Limitations

The main experiments here use EuroSAT — a 10-class patch classification dataset that most models find relatively easy (baselines already at 94-98%). We have some evidence from DINOv3 and ResNet50 results on RESISC45 (45 classes) and 4 GeoBench benchmarks that the core findings generalize across patch classification tasks — int8 is effectively lossless on all of them. But all 6 datasets are patch classification. We have not tested:

• Semantic segmentation — pixel-level predictions may be more sensitive to per-dimension quantization error
• Pixel regression (e.g., canopy height, biomass estimation) — continuous targets could amplify small reconstruction errors that classification absorbs
• Object detection — localization accuracy may degrade differently than classification accuracy
• Change detection — differencing compressed embeddings across time steps could compound quantization noise
• Retrieval — ranking quality over large databases may be more sensitive to distance distortion than top-1 classification

If you are saving embeddings for one of these tasks, we recommend validating compression effects on a representative sample before committing to a storage format.

We also only test OlmoEarth-nano (1.4M params, 128d) — the smallest model in the OlmoEarth family. The larger variants (Tiny at 192d, Base at 768d, Large at 1024d) may have different compression characteristics. And input normalization and patch size play a role in downstream performance that we haven’t disentangled from the compression effects here.

Takeaways

Some takeaways from these experiments (given the above caveat about patch classification):

1. Always use int8. It is statistically indistinguishable from float32 across every model and dataset we tested (p > 0.12). 4x compression, zero engineering effort, no reason not to.

2. Check intrinsic dimensionality before storing. Many geospatial embeddings carry redundant dimensions. Tessera packs 94% of its variance into 4 dimensions; even DINOv3 can be PCA-reduced to 64d with only 1.5% kNN loss.

3. PCA(64)+int8 is the sweet spot for DINOv3. 64 bytes/embedding, 64x compression, 1.4% kNN loss, 96.4% linear accuracy.

4. For binary search indices, use Hamming distance directly on binary embeddings. Skip dequantization — it introduces correlated noise that hurts more than it helps.

5. Don’t use ternary quantization. Binary is simpler, uses fewer bits, and performs better in every configuration we tested.

6. Tune regularization (C) for linear probes. The default C=1.0 leaves performance on the table: Tessera gains 0.9% from C=10, DINOv3 gains 0.4% from C=0.1.

7. Verify per-class metrics under compression. Highway and PermanentCrop degrade disproportionately — aggregate accuracy can mask category-level failures.

Bibliography

[1] Feng, Z., et al. “Tessera: Global-Scale Pixel Embeddings from Sentinel-2.” arXiv:2506.20380, 2025. [paper] [code]

[2] Herzog, H., et al. “OlmoEarth: Stable Latent Image Modeling for Multimodal Earth Observation.” arXiv:2511.13655, 2025. [paper] [code]

[3] Brown, C.F., et al. “AlphaEarth Foundations: An embedding field model for accurate and efficient global mapping from sparse label data.” arXiv:2507.22291, 2025. [paper] [GEE catalog]

[4] Fang, H., et al. “Earth Embeddings as Products.” arXiv:2601.13134, 2026. [paper] [blog]

[5] Bauer-Marschallinger, B. and Falkner, K. “Wasting Petabytes: A Survey of the Sentinel-2 UTM Tiling Grid and its Spatial Overhead.” ISPRS Journal of Photogrammetry and Remote Sensing, 2023. [paper]

[6] ESA. “Copernicus Sentinels Mission and Data Management.” Living Planet Symposium, 2025. [slides]

[7] Hackel, L., Burgert, T., and Demir, B. “How Much of a Model Do We Need? Redundancy and Slimmability in Remote Sensing Foundation Models.” arXiv:2601.22841, 2026. [paper]

[8] Papazafeiropoulos, T., et al. “Hide and Seek: Investigating Redundancy in Earth Observation Imagery.” arXiv:2603.13524, 2026. [paper]

[9] Helber, P., et al. “EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification.” IEEE JSTARS, 2019. [paper]

[10] Cheng, G., Han, J., and Lu, X. “Remote Sensing Image Scene Classification: Benchmark and State of the Art.” Proceedings of the IEEE, 2017. [paper]

[11] Lacoste, A., et al. “GEO-Bench: Toward Foundation Models for Earth Monitoring.” NeurIPS, 2023. [paper]

[12] Simeoni, O., et al. “DINOv3” arXiv:2508.10104, 2025. [paper] [code]

The dataset

Chesapeake Roads Spatial Context (RSC) contains 30,000 512x512 NAIP patches from Maryland with 4-band imagery (RGB + near-infrared) and labels for three classes: background, road, and tree canopy over road. The class balance is extreme — 96.3% background, 3.0% road, 0.7% tree canopy over road.

The idea is simple: roads pass under tree canopy, and when they do, the local appearance at those pixels looks like trees, not road. A model can only classify those pixels correctly by looking at nearby visible road segments and inferring that the road continues underneath. The distance from each tree-canopy-over-road pixel to the nearest visible road pixel has a median of 4 pixels but a 95th percentile of 107 pixels, so some of these inferences require connecting evidence across a large spatial span.

Other remote sensing datasets with roads (ISPRS Vaihingen/Potsdam, LandCover.ai, DeepGlobe, SpaceNet, RoadTracer) are strong tests of segmentation quality, topology, or connectivity, but none explicitly separate easy road pixels from locally ambiguous ones. Chesapeake RSC partitions the road class by spatial difficulty, which makes it possible to ask not just “how well does this model segment roads?” but “how far away can the model look to make a correct decision?”

Distance-weighted recall

To quantify how well a model uses spatial context, we introduced distance-weighted recall (DWR). For each tree-canopy-over-road pixel, we measure its distance to the nearest visible road pixel, then weight the pixel’s contribution to recall by that distance. A model that only gets the easy nearby pixels right will have a high unweighted recall but a low DWR; a model that correctly classifies tree canopy far from any visible road will score much higher.

Theoretical vs. effective receptive fields

Every segmentation architecture has a theoretical receptive field (TRF) — the maximum region of the input that could influence a given output pixel, determined purely by kernel sizes, strides, and network depth. Araujo et al. (2019) give a clear treatment of how to compute this for convolutional networks.

The effective receptive field (ERF) is what the model actually uses. Luo et al. (2016) showed that in deep CNNs the ERF is typically much smaller than the TRF and has a Gaussian-like concentration around the center pixel. A model can have a 527-pixel theoretical receptive field and still behave as though it only looks at a small local neighborhood. For transformers, self-attention gives a global TRF by construction, but global access does not automatically mean global use.

Models and results

We trained a U-Net with a ResNet-18 backbone (14M params, TRF of 527 pixels) and two SegFormer variants: MiT-B0 (3.7M params) and MiT-B2 (25M params), both with global TRFs via self-attention. All models were trained on a binary task (road vs. background, with canopy-over-road grouped into road) using AdamW, cosine annealing, and cross-entropy loss for 150 epochs.

R = recall, P = precision, TC/Road R = recall on tree canopy over road subgroup. Background metrics omitted (all ~99.5%).

SegFormer MiT-B2 leads on overall metrics — best road recall (84.6%) and best tree canopy recall (63.2%). But the U-Net wins on DWR (44.0 vs 42.3), meaning it’s better at classifying tree canopy pixels that are far from visible road. The SegFormers’ ability to attend to distant tokens doesn’t translate into better performance on the spatially hardest pixels. This isn’t to say the U-Net is good at spatial reasoning (62.4% tree canopy recall is still a 21-point drop from visible road recall) — it’s that the ViT’s global attention doesn’t magically help here.

We also tested cutout augmentation — randomly masking 64x64 patches of the input during training — to force the model to reconstruct missing regions and improve spatial reasoning. We tested several cutout sizes and the story was the same: it doesn’t help. The variant shown here achieves 61.8% tree canopy recall, comparable to the baseline’s 62.4%.

Performance degrades with distance

For each tree-canopy-over-road pixel in the test set, we measure the distance to the nearest visible road pixel, bin into log-spaced groups, and compute recall within each bin.

Both models show monotonic performance degradation. At distance ~1 pixel, the U-Net achieves ~76% recall and the SegFormer ~73%. By ~100 pixels, both are in the 36-43% range. At 400+ pixels, recall falls to 20-28%. The U-Net outperforms the SegFormer MiT-B0 at every distance despite having a narrower effective receptive field.

Measuring the effective receptive field

The distance-stratified recall shows that models fail to use long-range context. Gradient-based ERF analysis shows why.

We computed gradient attributions by backpropagating from pre-softmax road logits to the input for 200 test images, then measured how gradient mass distributes as a function of radius from the output pixel. The effective diameter at a given percentile is the smallest circle centered on the output pixel that encloses that fraction of total gradient mass.

The U-Net concentrates half its gradient mass within a 184-pixel diameter circle despite having a 527-pixel theoretical receptive field. The SegFormer reaches 50% at 292 pixels — 1.6x wider, but the bulk of its attention (90%) still stays within a 542-pixel diameter.

The road class has the widest ERF across all models, which suggests models do allocate more spatial attention for road-related predictions. But tree canopy ERFs are approximately equal to background ERFs — when a model needs to look farther to identify a canopy-covered road pixel, it doesn’t.

Gradient attribution: interactive explorer

The aggregate ERF statistics above summarize behavior across many pixels. The visualizer below lets you explore gradient attributions for individual predictions — hover over any 8x8 block to see which input pixels the model relies on for that block’s prediction. Toggle the mask overlay to see ground truth labels.

Sample

Model

U-Net (ResNet-18) SegFormer (MIT-B0)

Loading…

Block Info

Hover over the image to see gradient attribution

Gradient opacity: 75%

Show mask overlay

Prediction Ground Truth

Legend

Background

Road

Tree Canopy Over Road

Gradient (inferno)

What’s next

Across the architectures we tested, the bottleneck appears to be the training signal, not the architecture. Switching from CNN to transformer, increasing model capacity (MiT-B0 -> B2), and adding cutout augmentation all fail to substantially improve spatial reasoning on the hardest pixels. The binary cross-entropy loss treats all road pixels equally — it doesn’t reward the model for propagating information from distant visible road segments to occluded ones. Distance-aware loss functions or auxiliary connectivity tasks might provide a stronger learning signal.

Chesapeake RSC is a controlled version of a broader challenge in remote sensing, and the effective receptive field tools we use here apply directly to any task where local appearance is ambiguous and the correct label depends on spatial context.

Links

• Paper: Seeing the roads through the trees (arXiv)
• Code: github.com/isaaccorley/ChesapeakeRSC
• Dataset: torchgeo/ChesapeakeRSC on HuggingFace

The data

AEF embeddings are 64-dimensional vectors produced by a geospatial foundation model for every 10m pixel on Earth. See the Google Earth Engine catalog page here. They capture land use, land cover, vegetation, and built environment characteristics from satellite imagery. The data is distributed as cloud-optimized GeoTIFFs tiled at 8192x8192 pixels in UTM projection, with a GeoPackage spatial index mapping tile footprints to file paths across years 2018-2025.

Census block boundaries come from the 2020 US Census TIGER/Line shapefiles — the finest-grained census geography, with attributes like population (POP20), housing units (HOUSING20), land/water area, and an urban/rural flag (UR20).

Method

Results

Takeaways

1. AEF embeddings encode urban/rural character. While not very surprising, these can get 92.5% classification accuracy from a linear model alone.

2. The full 128-dim representation is substantially richer than PCA-10. For log population, R^2 jumps from 0.37 to 0.58 — the higher-order embedding dimensions contain useful information.

3. Block-level embedding statistics are a practical feature engineering approach. Mean and stdev per block compresses many 64-dim pixel vectors into a fixed 128-dim representation per geographic unit — simple enough to throw into any tabular ML pipeline. How to aggregate these was also the question of our recent paper, “From Pixels to Patches: Pooling Strategies for Earth Embeddings” (preprint here).

Reproduction

Both scripts are available in this gist. The precomputed Washington State block-level embedding statistics are available as a GeoParquet file on Hugging Face if you want to skip the ~34 GB download and play with the data instead.

# Download census blocks from:
# https://www.census.gov/cgi-bin/geo/shapefiles/index.php
# (year 2025, layer "Blocks (2020)", state "Washington")

# Download AEF index
curl -O https://data.source.coop/tge-labs/aef/v1/annual/aef_index.gpkg

# Compute block statistics (downloads ~34 GB of tiles)
python3 compute_aef_block_stats.py \
  --year 2020 --max-land-km2 25 \
  --output wa_block_aef_stats.geoparquet

# Quick test on Seattle/Bellevue area (~2 tiles)
python3 compute_aef_block_stats.py \
  --bbox -122.44 47.49 -122.07 47.73 \
  --output seattle_bellevue_aef_stats.geoparquet

# Generate PCA visualization
python3 pca_rasterize.py wa_block_aef_stats.geoparquet -o wa_pca.tif

Why satellite imagery isn’t just “big computer vision”

If you’ve tried to plug satellite imagery into a standard computer vision pipeline, you’ve probably run into the friction. Imagery arrives as large georeferenced scenes (often with more than three bands), labels live in separate files with different coordinate reference systems (CRSs) and resolutions, and you can’t just resize and normalize your way to a training loop. Further, once you have a model you need to run inference across entire scenes, which requires stitching together predictions from overlapping tiles and saving the output as a georeferenced raster.

TorchGeo handles this by providing geospatial-aware datasets, samplers, and transforms that slot into standard PyTorch workflows. The key components are:

• Composable datasets — use | (union) to mosaic tiles and & (intersection) to pair imagery with labels, all lazily evaluated
• Geographic samplers — RandomGeoSampler for training and GridGeoSampler for inference, sampling in projected coordinates rather than pixel indices
• Windowed reads — no pre-tiling (assuming you have data in Cloud Optimized GeoTIFFs or other cloud native formats); TorchGeo reads only the pixels it needs from large rasters on demand

The Earth Surface Water dataset

The Earth Surface Water dataset contains Sentinel-2 patches paired with binary water masks from diverse geographic regions. It’s a good fit for a tutorial because it’s small enough to train on quickly but realistic enough to show the full complexity of an EO workflow: patches span multiple UTM zones, the labels are raster masks in separate files, and the task (water vs. non-water) is easy to interpret visually.

Pairing imagery and labels across UTM zones

The tutorial constructs paired RasterDataset objects for imagery and masks, then combines them with TorchGeo’s intersection operator:

from torchgeo.datasets import RasterDataset

images = RasterDataset(paths=image_dir, crs="EPSG:3395", res=10, transforms=scale)
masks = RasterDataset(paths=mask_dir, crs="EPSG:3395", res=10)
masks.is_image = False  # use nearest-neighbor resampling for discrete labels
dataset = images & masks

Because the patches are distributed globally (often falling in different UTM zones), the notebook specifies a global CRS (World Mercator, EPSG:3395) so that all samples are consistently aligned during sampling and loading.

From 6 bands to 9 channels with spectral indices

Satellite data typically has more than three bands, which breaks standard vision preprocessing pipelines. The Earth Surface Water tutorial uses six Sentinel-2 bands — B02 (blue), B03 (green), B04 (red), B08 (NIR) at 10 m resolution, plus B11 and B12 (SWIR) at 20 m. Raw Sentinel-2 digital numbers are divided by 10,000 to convert to surface reflectance (a small detail that’s easy to forget and will silently wreck your training if you skip it).

From those 6 reflectance bands, the notebook computes three spectral indices using TorchGeo’s built-in transforms: NDWI (Normalized Difference Water Index, using green and NIR), MNDWI (Modified NDWI, using green and SWIR2), and NDVI (Normalized Difference Vegetation Index). The full preprocessing pipeline chains index computation and normalization in a single Sequential:

import kornia.augmentation as K
from torchgeo.transforms import indices

# Compute mean/std over training images for z-score normalization,
# then pad with 0s/1s so the 3 index channels pass through unchanged
mean = np.concatenate([band_mean, [0, 0, 0]])
std = np.concatenate([band_std, [1, 1, 1]])

tfms = torch.nn.Sequential(
    indices.AppendNDWI(index_green=1, index_nir=3),   # NDWI: (Green - NIR) / (Green + NIR)
    indices.AppendNDWI(index_green=1, index_nir=5),   # MNDWI: (Green - SWIR2) / (Green + SWIR2)
    indices.AppendNDVI(index_nir=3, index_red=2),      # NDVI: (NIR - Red) / (NIR + Red)
    K.Normalize(mean=mean, std=std),
)
# Input: 6 bands,  Output: 9 channels (6 normalized bands + 3 indices)

We pad the mean/std vectors with [0, 0, 0] and [1, 1, 1], so that z-score normalization becomes a no-op for the index channels, which are already bounded [-1, 1] by construction.

Adapting an RGB architecture to 9 channels

The model is a DeepLabV3 with a ResNet-50 backbone from torchvision, trained from scratch — ImageNet-pretrained weights expect 3-channel RGB input, so they’re not useful here. The key adaptation is reinitializing the first convolutional layer to accept our 9 input channels:

from torchvision.models.segmentation import deeplabv3_resnet50

model = deeplabv3_resnet50(weights=None, num_classes=2)
backbone = model.get_submodule("backbone")
conv = torch.nn.Conv2d(
    in_channels=9, out_channels=64,
    kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False,
)
backbone.register_module("conv1", conv)

The dataset ships with a pre-defined, geographically separated train/validation split — important for avoiding the over-optimistic metrics that spatial autocorrelation can cause in EO. Within each split, RandomGeoSampler draws 512x512 chips in geographic coordinate space, handling CRS alignment and resolution matching automatically. After 10 epochs with Adam (lr=1e-4, weight_decay=0.01) and a batch size of 4, the model reaches 0.977 overall accuracy and 0.824 IoU on the validation set. Training takes a few minutes on a single GPU.

Inference on a Sentinel-2 scene

This is the part of the tutorial where the model stops being a number on a leaderboard and starts being a useful tool! After training, the notebook downloads a Sentinel-2 scene over Rio de Janeiro, Brazil from the Microsoft Planetary Computer, runs gridded inference across the entire tile, and finally saves the resulting predictions as a georeferenced GeoTIFF.

from torchgeo.datasets import Sentinel2
from torchgeo.samplers import GridGeoSampler

s2_dataset = Sentinel2(paths=scene_dir, bands=bands, res=10, transforms=scale)
grid_sampler = GridGeoSampler(s2_dataset, size=512, stride=448, units=Units.PIXELS)
s2_dataloader = DataLoader(
    s2_dataset, sampler=grid_sampler, batch_size=16, collate_fn=stack_samples
)

The GridGeoSampler tiles the scene into overlapping 512x512 patches (stride=448, so 64 pixels of overlap on each edge). Predictions are stitched back together and saved as a GeoTIFF — tiled, compressed, with overviews — that is pixel-aligned with the input scene:

import rasterio as rio

profile = {
    "driver": "GTiff", "dtype": "uint8", "count": 1,
    "width": img_width, "height": img_height,
    "crs": crs, "transform": transform,
    "compress": "deflate", "tiled": True,
    "blockxsize": 512, "blockysize": 512,
}
with rio.open(output_path, "w", **profile) as dst:
    dst.write(prediction, 1)
    dst.build_overviews([2, 4, 8, 16], rio.enums.Resampling.nearest)

The result is a georeferenced water mask that you can open in QGIS, load into a GIS pipeline, or overlay on the original scene.

This step bridges the gap between “model that scores well on a test set” and “model that produces a useful geospatial product.” It also lets you explore the model’s behavior beyond aggregate metrics: How sharp are the predictions along coastlines? What’s the smallest water feature it can detect? Where does it fail?

Try it yourself

The tutorial is distributed as two executable notebooks, and all you need is a machine with a GPU (a Colab T4 works fine):

• Introduction to TorchGeo — core abstractions (dataset composition, spatiotemporal indexing, geographic samplers)
• Earth Surface Water — the end-to-end case study described in this post

For more detail on the design choices and motivation, see our ICLR 2026 ML4RS Workshop paper. The tutorial also builds on Mauricio Cordeiro’s 3-part Medium series on geospatial analysis with TorchGeo. If you have questions or want to discuss, come find us in the TorchGeo Slack.

What to expect

• Paper highlights — summaries and takes on new GeoAI / GeoML research we’re reading
• Code demos — small, reproducible experiments with TorchGeo and the broader geospatial ML ecosystem
• New models & datasets — quick tours of recently released foundation models, benchmarks, and datasets worth trying
• Geospatial explorations — anything from satellite imagery tricks to fun visualizations to workflow tips

Posts will be short and practical. If something is interesting enough to share in Slack, it’s interesting enough to write up here.

Stay tuned, and come hang out in TorchGeo Slack if you haven’t already.