TerraBit

To test this, we built TerraBit — a global retrieval demo that runs entirely in the browser with no backend or server-side computation. We binary-quantize the full Clay v1.5 corpus into packed bit vectors, store them as spatially-partitioned cloud-native Parquet on public object storage, and let the browser handle shard discovery, data fetching, and in-memory Hamming scoring. The entire “backend” is a static S3 bucket; all compute happens on your machine.

How it works:

1. You draw one or more regions of interest (ROI) anywhere — each of these are loaded independently; regions can be rectangles or freehand polygons
2. You click to create exemplar patches on the map (one or many); positive exemplars outside the AOI have their embeddings fetched on the fly; negatives work anywhere on the globe for contrastive scoring (pos_dist − neg_dist); you can also invert search (bitwise NOT) to find the opposite of a reference!
3. DuckDB-WASM queries a manifest for intersecting geohash shards; only those shards are fetched via HTTP range requests — no full-corpus scan
4. A Web Worker scores all candidates with brute-force Hamming distance and returns ranked results
5. The results render via MapLibre GL across several view modes (top-k, heatmap, threshold, outlier, surprise, gradient)

Multiple exemplars can be combined via mean distance, by applying bitwise AND / OR / XOR directly on the packed binary vectors before scoring — exact, lossless ops that compose semantically because binary embeddings have nice arithmetic properties.

The 50M embeddings are partitioned into geohash-aligned Parquet shards and published on Source Cooperative, which serves them cloud-natively out of S3 — public HTTP with byte-range support, no egress fees, no intermediate server. A single manifest file records the path, row count, and spatial extent of every shard.

When you draw an ROI, DuckDB-WASM queries the manifest with a bounding-box predicate — manifest-based shard pruning: the manifest acts as a coarse spatial index so the browser never opens metadata on shards outside the ROI. Once the intersecting shard list is resolved, DuckDB streams those shard files over HTTP (via httpfs range requests) and applies a second filter at the row level — a bbox predicate for rectangles, or ST_Intersects for freehand polygons — to extract only patches within the drawn region. Ranking over the candidate slice is exact brute-force Hamming: binary embeddings arrive as packed Uint8Array columns (128 bytes per 1024-dim vector) and are scored in a Web Worker via XOR+popcount, which maps directly to hardware-accelerated popcount instructions and completes in milliseconds for a typical AOI partition.

The binary embeddings are lossy though — we find they have ~65% recall@10 (the fraction of true float32 nearest neighbors recovered by the binary representation) which means roughly a third of true neighbors are missed (Figure 2). Good enough for coarse exploration; not a claim about downstream curation or labeling productivity. How coarse is too coarse though? 65% recall goes further than you’d expect — try the demo on your own region!

A few examples of what this enables: a) click a center-pivot irrigation field in Kansas and separate it from rectangular fields across the state, b) pick a greenhouse cluster in Rotterdam and highlight dense greenhouse and vineyard complexes across the region or c) select a solar installation in northwest India and find others at similar scale. None of these queries require labeled data, a trained classifier, or even a definition of what you’re looking for beyond a single click. This is useful for data exploration, bootstrapping training datasets for supervised models, and narrowing the search space before running expensive high-resolution models over targeted areas. The demo also supports exporting ranked candidates as GeoParquet!

Binary Earth Embedding Retrieval at Planet Scale

Clay v1.5 produces 1024-dimensional embeddings from Sentinel-2 imagery. The global corpus spans two years of observations — roughly 50 million embeddings covering Earth’s land surface — and is 183 GiB on disk in ZSTD-compressed Parquet (≈190 GiB as raw float32 – float32s don’t compress well even if they come from a GeoFM). Serving float32 vectors at this scale to a browser isn’t viable; the question we ask is how aggressively you can compress without destroying retrieval quality.

Binary quantization reduces each dimension to a single sign bit. 1024 floats (4,096 bytes) become 128 bytes — a 32× reduction on the raw payload. End-to-end on disk (Parquet with ZSTD, geometry and STAC metadata columns, row-group overhead), the full 49.8M-row corpus drops from 182.9 GiB to 11.1 GiB — 16.5× compression. The on-disk number is what you pay for on object storage (32× is the raw payload reduction). The web demo corpus is smaller still (~7 GiB) because several columns were dropped and the compression level was increased — a demo-specific optimization on top of the 16.5× quantization win.

Why does aggressive quantization work at all on 1024-dimensional vectors? One diagnostic is the intrinsic dimension (ID) — the degrees of freedom the data actually uses, regardless of ambient dimensionality [Facco et al., 2017; Levina & Bickel, 2004]. This framing is directly motivated by Rao et al., 2025, who find that geographic representations — despite operating in 256–512 dimensional spaces — compress to just 2–10 intrinsic dimensions, and that ID correlates with downstream task performance. For Clay v1.5 we estimate ID ≈ 13–17 (MLE: 17.0, TwoNN: 12.6, Local PCA: 17.0, on a 10k sample subset). Three estimators with different assumptions agree on a narrow range. Low ID is why aggressive compression is worth attempting — the data simply isn’t using most of its dimensions.

TurboQuant aka rotate before you quantize

Binary is the extreme end of the compression spectrum, and the retrieval demo uses it — but what if you need more recall than binary while keeping storage well below float32?

Standard affine quantization at low bit-widths (int2–int4) suffers from high variance disparity across embedding dimensions: some dimensions carry far more signal than others, and a uniform quantization grid wastes bits on low-variance dimensions while clipping high-variance ones. TurboQuant fixes this by applying a fixed random orthogonal rotation (sampled once from a Haar-distributed ensemble via QR decomposition) before symmetric affine quantization: . The rotation spreads variance across dimensions so no channel dominates the bit budget. is generated once, stored with the quantized embeddings, and reused for all queries — one matrix multiply at encode/decode, no retraining.

Earth embeddings have the same property: ID ≈ 13–17 in a 1024-d space leaves a lot of variance to redistribute.

We ran TurboQuant across bit-widths on the Clay v1.5 embeddings. The gains are largest at low precision and vanish at high precision:

Practical takeaway: if binary recall is too coarse but you still want aggressive compression, TurboQuant at int2–int4 is worth trying first. At TurboQuant int4, 95% recall at ~6× on-disk compression (8× on the raw payload). By int8, the affine grid is fine enough on its own and the rotation adds nothing.

Search throughput

We also benchmarked brute-force kNN on a 1M-vector subset (1K queries, k=10) using FAISS on CPU and PyTorch’s torch.cdist on an RTX 3090. While other bit-widths benefit from GPU acceleration, binary search is unreasonably fast on CPU thanks to SIMD acceleration.

Why not just build a backend?

Reasonable reaction: “Cool demo, but real systems need a database and an API.” Maybe — but in geospatial ML, the gap between a working prototype and a deployed tool is almost all infrastructure: vector databases, REST APIs, auth, scaling, monitoring. Each layer is individually reasonable but collectively large enough of a barrier to prevent someone from shipping and maintaining a useful tool.

Furthermore, existing vector DBs primarily partition by embedding similarity; a small AOI query still touches shards scattered across the index with geospatial filtering applied AFTER the expensive approximate nearest neighbor (ANN) step. Getting geo-first partitioning right takes careful co-design, and no existing systems target zero-ops browser-native serving of a static corpus. Our approach sidesteps that: embeddings partitioned spatially by geohash, a manifest for shard pruning, and a throwaway Hamming scan.

To be clear, backends still have their place. Full-corpus ANN, multi-user serving, auth, and strict SLAs are backend territory. But for exploration and dataset curation, the barrier to useful interaction with embeddings should be as close to zero as possible, and for a lot of real problems, client-side is enough.

Links: TerraBit retrieval demo · binarized embedding corpus · pt. 1: Compressing Earth Embeddings

The storage problem

Earth’s land surface covers about 150 million km^2. At Sentinel-2’s 10m resolution, that’s 1.5 trillion pixels. Multiply by embedding dimension and bytes per element:

For context, the entire Sentinel-2 archive — every L1C and L2A product collected since 2015 — was roughly 22 PB in 2022 [5] and exceeded 50 PB by mid-2025 [6]. The archive grows by 3-4 PB per year. A single year of 1024d float32 embeddings (6.1 PB) would exceed the annual Sentinel-2 data volume that produced them. The embeddings are larger than the source imagery.

The embeddings are larger than the source imagery.

And these are per-year numbers. AlphaEarth covers 2017-2025 (9 years). Tessera plans the same. Multi-year archives at these scales reach tens of petabytes even for compact models. So how much can you compress before the embeddings stop being useful?

EO representations are redundant

Two recent papers provide evidence that earth observation representations carry substantial redundancy.

Model-level redundancy. Hackel et al. [7] applied post-hoc “slimming” to remote sensing foundation models — uniformly reducing the width of transformer layers after training. At just 1% of the original FLOPs, these models retained over 71% of their full-scale accuracy (relative retention). An ImageNet-trained MAE dropped below 10% relative retention under the same treatment. Intermediate model sizes sometimes outperformed the full model, suggesting the extra capacity adds noise rather than signal. If the intermediate representations are this redundant, the output embeddings are too.

Image-level redundancy. Papazafeiropoulos et al. [8] applied patch-level masking during training and inference of a ViT model, retaining only a fraction of image patches. On BigEarthNet, 15% patch retention achieved 99.4% of baseline accuracy. Even segmentation tolerated 50% patch removal while recovering ~97% of full performance.

These results suggest that standard embedding compression methods — including quantization and dimensionality reduction — may be effective for remotely sensed data as well. So we tested it!

Experimental setup

We evaluate combinations of quantization (float32, int8, int4, int2, binary, ternary, product quantization) and dimensionality reduction (PCA, truncated SVD, random projection, feature selection) across 5 embedding models and 6 classification datasets.

All experiments in this section use EuroSAT [9] — a 10-class Sentinel-2 land cover dataset with 21,600 images — as the primary benchmark. We use the precomputed embeddings for AEF, OlmoEarth, and Tessera from Isaac’s geopool repository, and we compute DINOv3 and ResNet50 embeddings separately. Then, we validate our findings on 5 additional datasets (RESISC45 [10] and 4 GeoBench [11] benchmarks) with the DINOv3 and ResNet50 based embeddings in the cross-dataset section below.

int8 is always free

The simplest compression: reduce each float32 value to int8. For each dimension, compute the min and max across the training set, then linearly map the range into 256 integer levels (4x compression).

EuroSAT kNN accuracy. Bold row = statistically indistinguishable from float32 baseline.

We find int8 is never statistically distinguishable from float32. McNemar’s test gives p >= 0.12 for every model-dataset pair (smallest p = 0.12). The 95% bootstrap confidence interval for the accuracy difference is within +/-0.2% everywhere. There is no reason to store float32 embeddings.

There is no reason to store float32 embeddings.

We also find int4 loses less than 1% for AEF and DINOv3. Binary quantization (1 bit per dimension, 32x compression) is worth a closer look — DINOv3 at 128 bytes still hits 91.8%! More on this below.

Most embedding dimensions are redundant

PCA variance analysis reveals different spectral structures across models:

Cumulative variance explained by top-k PCA components, fitted on EuroSAT training embeddings.

OlmoEarth-nano spreads its variance more broadly than Tessera, with 77% in 4 dimensions and needing 32 dimensions for 98%. DINOv3 distributes variance more evenly still, needing 256 dimensions for 97%.

DINOv3 spreads its variance across many dimensions, so you might expect it to compress poorly — if no dimension is dispensable, PCA can’t help. But DINOv3 at PCA(64)+int8 (6% of its original dimensions) still hits 93.1% kNN accuracy, only 1.4% below baseline. The dimensions PCA discards carry variance but apparently not much task-relevant information.

Combined compression: the Pareto frontier

The best configurations combine PCA with quantization — reduce dimensions first, then quantize:

EuroSAT accuracy. kNN: k=5, cosine. Linear: logistic regression, C tuned.

The two plots tell different stories. For kNN, AEF dominates at every storage budget — its 64 dimensions are compact enough that int8 at 64 bytes is nearly unbeatable, and larger models can’t overcome the dimensionality tax. For linear probes, DINOv3 pulls ahead once budgets exceed ~16 bytes, because a trained classifier can exploit the richer representation even after PCA compression.

PCA(64)+int8 at 64 bytes/embedding is the sweet spot for DINOv3: 64x compression with only 1.4% kNN loss and 96.4% linear accuracy. That brings a year of global DINOv3 embeddings from 6.1 PB down to 96 TB — the same footprint as AlphaEarth’s native int8 representation. Which model to choose depends on your task: kNN retrieval favors AEF, classification with a trained head favors DINOv3.

Binary quantization on DINOv3

DINOv3 loses only 2.7% kNN accuracy under binary quantization (1 bit per dimension, 32x compression), while AEF and Tessera lose 5.7-6.2%. We hypothesize that this might be due to:

1. High dimensionality. 1,024 binary dimensions give 2^1024 possible codes — enormous capacity for separating 10 classes.

2. Balanced dimensions. DINOv3’s dimensions are nearly symmetric around their means (average imbalance = 0.018). Each threshold bit carries close to 1 bit of entropy. OlmoEarth-nano is also well-balanced (0.052), while AEF’s higher imbalance (0.082) means many bits are nearly constant.

A related finding with the binary quantizations: Hamming distance on raw bits outperforms reconstructing float32 vectors and computing cosine distance. The reconstruction step replaces each bit with a centroid value (the mean of all above-threshold or below-threshold values for that dimension). We find that KNN with a Hamming distance (count the differing bits between the two vectors) outperforms using cosine distance on the reconstructed vectors. This seems to preserve the ranking of neighbor distances better:

Hamming distance is also significantly faster to compute than cosine distance on reconstructed vectors — it reduces to a popcount on XOR’d bit vectors.

Cross-dataset consistency

The compression patterns hold across all 6 datasets:

• int8 is effectively lossless on all datasets, including the 45-class RESISC45.
• PCA(64)+int8 at 64 bytes gives 93.1% on EuroSAT (10 classes) and 82.0% on RESISC45 (45 classes) — proportionally similar retention.
• m-forestnet (deforestation driver classification) is the hardest task at ~40% kNN for DINOv3 and ~36% for ResNet50 — likely because RGB-only embeddings lose the spectral bands needed for this task.

Per-class failure modes

Under aggressive compression, specific classes are disproportionately affected:

Highway and PermanentCrop are consistently the most affected — narrow categories relying on fine-grained spectral or spatial features that aggressive quantization destroys. If your application needs balanced per-class performance (rare category detection, e.g.), avoid extreme compression and verify per-class metrics.

Limitations

The main experiments here use EuroSAT — a 10-class patch classification dataset that most models find relatively easy (baselines already at 94-98%). We have some evidence from DINOv3 and ResNet50 results on RESISC45 (45 classes) and 4 GeoBench benchmarks that the core findings generalize across patch classification tasks — int8 is effectively lossless on all of them. But all 6 datasets are patch classification. We have not tested:

• Semantic segmentation — pixel-level predictions may be more sensitive to per-dimension quantization error
• Pixel regression (e.g., canopy height, biomass estimation) — continuous targets could amplify small reconstruction errors that classification absorbs
• Object detection — localization accuracy may degrade differently than classification accuracy
• Change detection — differencing compressed embeddings across time steps could compound quantization noise
• Retrieval — ranking quality over large databases may be more sensitive to distance distortion than top-1 classification

If you are saving embeddings for one of these tasks, we recommend validating compression effects on a representative sample before committing to a storage format.

We also only test OlmoEarth-nano (1.4M params, 128d) — the smallest model in the OlmoEarth family. The larger variants (Tiny at 192d, Base at 768d, Large at 1024d) may have different compression characteristics. And input normalization and patch size play a role in downstream performance that we haven’t disentangled from the compression effects here.

Takeaways

Some takeaways from these experiments (given the above caveat about patch classification):

1. Always use int8. It is statistically indistinguishable from float32 across every model and dataset we tested (p > 0.12). 4x compression, zero engineering effort, no reason not to.

2. Check intrinsic dimensionality before storing. Many geospatial embeddings carry redundant dimensions. Tessera packs 94% of its variance into 4 dimensions; even DINOv3 can be PCA-reduced to 64d with only 1.5% kNN loss.

3. PCA(64)+int8 is the sweet spot for DINOv3. 64 bytes/embedding, 64x compression, 1.4% kNN loss, 96.4% linear accuracy.

4. For binary search indices, use Hamming distance directly on binary embeddings. Skip dequantization — it introduces correlated noise that hurts more than it helps.

5. Don’t use ternary quantization. Binary is simpler, uses fewer bits, and performs better in every configuration we tested.

6. Tune regularization (C) for linear probes. The default C=1.0 leaves performance on the table: Tessera gains 0.9% from C=10, DINOv3 gains 0.4% from C=0.1.

7. Verify per-class metrics under compression. Highway and PermanentCrop degrade disproportionately — aggregate accuracy can mask category-level failures.

Bibliography

[1] Feng, Z., et al. “Tessera: Global-Scale Pixel Embeddings from Sentinel-2.” arXiv:2506.20380, 2025. [paper] [code]

[2] Herzog, H., et al. “OlmoEarth: Stable Latent Image Modeling for Multimodal Earth Observation.” arXiv:2511.13655, 2025. [paper] [code]

[3] Brown, C.F., et al. “AlphaEarth Foundations: An embedding field model for accurate and efficient global mapping from sparse label data.” arXiv:2507.22291, 2025. [paper] [GEE catalog]

[4] Fang, H., et al. “Earth Embeddings as Products.” arXiv:2601.13134, 2026. [paper] [blog]

[5] Bauer-Marschallinger, B. and Falkner, K. “Wasting Petabytes: A Survey of the Sentinel-2 UTM Tiling Grid and its Spatial Overhead.” ISPRS Journal of Photogrammetry and Remote Sensing, 2023. [paper]

[6] ESA. “Copernicus Sentinels Mission and Data Management.” Living Planet Symposium, 2025. [slides]

[7] Hackel, L., Burgert, T., and Demir, B. “How Much of a Model Do We Need? Redundancy and Slimmability in Remote Sensing Foundation Models.” arXiv:2601.22841, 2026. [paper]

[8] Papazafeiropoulos, T., et al. “Hide and Seek: Investigating Redundancy in Earth Observation Imagery.” arXiv:2603.13524, 2026. [paper]

[9] Helber, P., et al. “EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification.” IEEE JSTARS, 2019. [paper]

[10] Cheng, G., Han, J., and Lu, X. “Remote Sensing Image Scene Classification: Benchmark and State of the Art.” Proceedings of the IEEE, 2017. [paper]

[11] Lacoste, A., et al. “GEO-Bench: Toward Foundation Models for Earth Monitoring.” NeurIPS, 2023. [paper]

[12] Simeoni, O., et al. “DINOv3” arXiv:2508.10104, 2025. [paper] [code]

The dataset

Chesapeake Roads Spatial Context (RSC) contains 30,000 512x512 NAIP patches from Maryland with 4-band imagery (RGB + near-infrared) and labels for three classes: background, road, and tree canopy over road. The class balance is extreme — 96.3% background, 3.0% road, 0.7% tree canopy over road.

The idea is simple: roads pass under tree canopy, and when they do, the local appearance at those pixels looks like trees, not road. A model can only classify those pixels correctly by looking at nearby visible road segments and inferring that the road continues underneath. The distance from each tree-canopy-over-road pixel to the nearest visible road pixel has a median of 4 pixels but a 95th percentile of 107 pixels, so some of these inferences require connecting evidence across a large spatial span.

Other remote sensing datasets with roads (ISPRS Vaihingen/Potsdam, LandCover.ai, DeepGlobe, SpaceNet, RoadTracer) are strong tests of segmentation quality, topology, or connectivity, but none explicitly separate easy road pixels from locally ambiguous ones. Chesapeake RSC partitions the road class by spatial difficulty, which makes it possible to ask not just “how well does this model segment roads?” but “how far away can the model look to make a correct decision?”

Distance-weighted recall

To quantify how well a model uses spatial context, we introduced distance-weighted recall (DWR). For each tree-canopy-over-road pixel, we measure its distance to the nearest visible road pixel, then weight the pixel’s contribution to recall by that distance. A model that only gets the easy nearby pixels right will have a high unweighted recall but a low DWR; a model that correctly classifies tree canopy far from any visible road will score much higher.

Theoretical vs. effective receptive fields

Every segmentation architecture has a theoretical receptive field (TRF) — the maximum region of the input that could influence a given output pixel, determined purely by kernel sizes, strides, and network depth. Araujo et al. (2019) give a clear treatment of how to compute this for convolutional networks.

The effective receptive field (ERF) is what the model actually uses. Luo et al. (2016) showed that in deep CNNs the ERF is typically much smaller than the TRF and has a Gaussian-like concentration around the center pixel. A model can have a 527-pixel theoretical receptive field and still behave as though it only looks at a small local neighborhood. For transformers, self-attention gives a global TRF by construction, but global access does not automatically mean global use.

Models and results

We trained a U-Net with a ResNet-18 backbone (14M params, TRF of 527 pixels) and two SegFormer variants: MiT-B0 (3.7M params) and MiT-B2 (25M params), both with global TRFs via self-attention. All models were trained on a binary task (road vs. background, with canopy-over-road grouped into road) using AdamW, cosine annealing, and cross-entropy loss for 150 epochs.

R = recall, P = precision, TC/Road R = recall on tree canopy over road subgroup. Background metrics omitted (all ~99.5%).

SegFormer MiT-B2 leads on overall metrics — best road recall (84.6%) and best tree canopy recall (63.2%). But the U-Net wins on DWR (44.0 vs 42.3), meaning it’s better at classifying tree canopy pixels that are far from visible road. The SegFormers’ ability to attend to distant tokens doesn’t translate into better performance on the spatially hardest pixels. This isn’t to say the U-Net is good at spatial reasoning (62.4% tree canopy recall is still a 21-point drop from visible road recall) — it’s that the ViT’s global attention doesn’t magically help here.

We also tested cutout augmentation — randomly masking 64x64 patches of the input during training — to force the model to reconstruct missing regions and improve spatial reasoning. We tested several cutout sizes and the story was the same: it doesn’t help. The variant shown here achieves 61.8% tree canopy recall, comparable to the baseline’s 62.4%.

Performance degrades with distance

For each tree-canopy-over-road pixel in the test set, we measure the distance to the nearest visible road pixel, bin into log-spaced groups, and compute recall within each bin.

Both models show monotonic performance degradation. At distance ~1 pixel, the U-Net achieves ~76% recall and the SegFormer ~73%. By ~100 pixels, both are in the 36-43% range. At 400+ pixels, recall falls to 20-28%. The U-Net outperforms the SegFormer MiT-B0 at every distance despite having a narrower effective receptive field.

Measuring the effective receptive field

The distance-stratified recall shows that models fail to use long-range context. Gradient-based ERF analysis shows why.

We computed gradient attributions by backpropagating from pre-softmax road logits to the input for 200 test images, then measured how gradient mass distributes as a function of radius from the output pixel. The effective diameter at a given percentile is the smallest circle centered on the output pixel that encloses that fraction of total gradient mass.

The U-Net concentrates half its gradient mass within a 184-pixel diameter circle despite having a 527-pixel theoretical receptive field. The SegFormer reaches 50% at 292 pixels — 1.6x wider, but the bulk of its attention (90%) still stays within a 542-pixel diameter.

The road class has the widest ERF across all models, which suggests models do allocate more spatial attention for road-related predictions. But tree canopy ERFs are approximately equal to background ERFs — when a model needs to look farther to identify a canopy-covered road pixel, it doesn’t.

Gradient attribution: interactive explorer

The aggregate ERF statistics above summarize behavior across many pixels. The visualizer below lets you explore gradient attributions for individual predictions — hover over any 8x8 block to see which input pixels the model relies on for that block’s prediction. Toggle the mask overlay to see ground truth labels.

Sample

Model

U-Net (ResNet-18) SegFormer (MIT-B0)

Loading…

Block Info

Hover over the image to see gradient attribution

Gradient opacity: 75%

Show mask overlay

Prediction Ground Truth

Legend

Background

Road

Tree Canopy Over Road

Gradient (inferno)

What’s next

Across the architectures we tested, the bottleneck appears to be the training signal, not the architecture. Switching from CNN to transformer, increasing model capacity (MiT-B0 -> B2), and adding cutout augmentation all fail to substantially improve spatial reasoning on the hardest pixels. The binary cross-entropy loss treats all road pixels equally — it doesn’t reward the model for propagating information from distant visible road segments to occluded ones. Distance-aware loss functions or auxiliary connectivity tasks might provide a stronger learning signal.

Chesapeake RSC is a controlled version of a broader challenge in remote sensing, and the effective receptive field tools we use here apply directly to any task where local appearance is ambiguous and the correct label depends on spatial context.

Links

• Paper: Seeing the roads through the trees (arXiv)
• Code: github.com/isaaccorley/ChesapeakeRSC
• Dataset: torchgeo/ChesapeakeRSC on HuggingFace

The data

AEF embeddings are 64-dimensional vectors produced by a geospatial foundation model for every 10m pixel on Earth. See the Google Earth Engine catalog page here. They capture land use, land cover, vegetation, and built environment characteristics from satellite imagery. The data is distributed as cloud-optimized GeoTIFFs tiled at 8192x8192 pixels in UTM projection, with a GeoPackage spatial index mapping tile footprints to file paths across years 2018-2025.

Census block boundaries come from the 2020 US Census TIGER/Line shapefiles — the finest-grained census geography, with attributes like population (POP20), housing units (HOUSING20), land/water area, and an urban/rural flag (UR20).

Method

Results

Takeaways

1. AEF embeddings encode urban/rural character. While not very surprising, these can get 92.5% classification accuracy from a linear model alone.

2. The full 128-dim representation is substantially richer than PCA-10. For log population, R^2 jumps from 0.37 to 0.58 — the higher-order embedding dimensions contain useful information.

3. Block-level embedding statistics are a practical feature engineering approach. Mean and stdev per block compresses many 64-dim pixel vectors into a fixed 128-dim representation per geographic unit — simple enough to throw into any tabular ML pipeline. How to aggregate these was also the question of our recent paper, “From Pixels to Patches: Pooling Strategies for Earth Embeddings” (preprint here).

Reproduction

Both scripts are available in this gist. The precomputed Washington State block-level embedding statistics are available as a GeoParquet file on Hugging Face if you want to skip the ~34 GB download and play with the data instead.

# Download census blocks from:
# https://www.census.gov/cgi-bin/geo/shapefiles/index.php
# (year 2025, layer "Blocks (2020)", state "Washington")

# Download AEF index
curl -O https://data.source.coop/tge-labs/aef/v1/annual/aef_index.gpkg

# Compute block statistics (downloads ~34 GB of tiles)
python3 compute_aef_block_stats.py \
  --year 2020 --max-land-km2 25 \
  --output wa_block_aef_stats.geoparquet

# Quick test on Seattle/Bellevue area (~2 tiles)
python3 compute_aef_block_stats.py \
  --bbox -122.44 47.49 -122.07 47.73 \
  --output seattle_bellevue_aef_stats.geoparquet

# Generate PCA visualization
python3 pca_rasterize.py wa_block_aef_stats.geoparquet -o wa_pca.tif

Why satellite imagery isn’t just “big computer vision”

If you’ve tried to plug satellite imagery into a standard computer vision pipeline, you’ve probably run into the friction. Imagery arrives as large georeferenced scenes (often with more than three bands), labels live in separate files with different coordinate reference systems (CRSs) and resolutions, and you can’t just resize and normalize your way to a training loop. Further, once you have a model you need to run inference across entire scenes, which requires stitching together predictions from overlapping tiles and saving the output as a georeferenced raster.

TorchGeo handles this by providing geospatial-aware datasets, samplers, and transforms that slot into standard PyTorch workflows. The key components are:

• Composable datasets — use | (union) to mosaic tiles and & (intersection) to pair imagery with labels, all lazily evaluated
• Geographic samplers — RandomGeoSampler for training and GridGeoSampler for inference, sampling in projected coordinates rather than pixel indices
• Windowed reads — no pre-tiling (assuming you have data in Cloud Optimized GeoTIFFs or other cloud native formats); TorchGeo reads only the pixels it needs from large rasters on demand

The Earth Surface Water dataset

The Earth Surface Water dataset contains Sentinel-2 patches paired with binary water masks from diverse geographic regions. It’s a good fit for a tutorial because it’s small enough to train on quickly but realistic enough to show the full complexity of an EO workflow: patches span multiple UTM zones, the labels are raster masks in separate files, and the task (water vs. non-water) is easy to interpret visually.

Pairing imagery and labels across UTM zones

The tutorial constructs paired RasterDataset objects for imagery and masks, then combines them with TorchGeo’s intersection operator:

from torchgeo.datasets import RasterDataset

images = RasterDataset(paths=image_dir, crs="EPSG:3395", res=10, transforms=scale)
masks = RasterDataset(paths=mask_dir, crs="EPSG:3395", res=10)
masks.is_image = False  # use nearest-neighbor resampling for discrete labels
dataset = images & masks

Because the patches are distributed globally (often falling in different UTM zones), the notebook specifies a global CRS (World Mercator, EPSG:3395) so that all samples are consistently aligned during sampling and loading.

From 6 bands to 9 channels with spectral indices

Satellite data typically has more than three bands, which breaks standard vision preprocessing pipelines. The Earth Surface Water tutorial uses six Sentinel-2 bands — B02 (blue), B03 (green), B04 (red), B08 (NIR) at 10 m resolution, plus B11 and B12 (SWIR) at 20 m. Raw Sentinel-2 digital numbers are divided by 10,000 to convert to surface reflectance (a small detail that’s easy to forget and will silently wreck your training if you skip it).

From those 6 reflectance bands, the notebook computes three spectral indices using TorchGeo’s built-in transforms: NDWI (Normalized Difference Water Index, using green and NIR), MNDWI (Modified NDWI, using green and SWIR2), and NDVI (Normalized Difference Vegetation Index). The full preprocessing pipeline chains index computation and normalization in a single Sequential:

import kornia.augmentation as K
from torchgeo.transforms import indices

# Compute mean/std over training images for z-score normalization,
# then pad with 0s/1s so the 3 index channels pass through unchanged
mean = np.concatenate([band_mean, [0, 0, 0]])
std = np.concatenate([band_std, [1, 1, 1]])

tfms = torch.nn.Sequential(
    indices.AppendNDWI(index_green=1, index_nir=3),   # NDWI: (Green - NIR) / (Green + NIR)
    indices.AppendNDWI(index_green=1, index_nir=5),   # MNDWI: (Green - SWIR2) / (Green + SWIR2)
    indices.AppendNDVI(index_nir=3, index_red=2),      # NDVI: (NIR - Red) / (NIR + Red)
    K.Normalize(mean=mean, std=std),
)
# Input: 6 bands,  Output: 9 channels (6 normalized bands + 3 indices)

We pad the mean/std vectors with [0, 0, 0] and [1, 1, 1], so that z-score normalization becomes a no-op for the index channels, which are already bounded [-1, 1] by construction.

Adapting an RGB architecture to 9 channels

The model is a DeepLabV3 with a ResNet-50 backbone from torchvision, trained from scratch — ImageNet-pretrained weights expect 3-channel RGB input, so they’re not useful here. The key adaptation is reinitializing the first convolutional layer to accept our 9 input channels:

from torchvision.models.segmentation import deeplabv3_resnet50

model = deeplabv3_resnet50(weights=None, num_classes=2)
backbone = model.get_submodule("backbone")
conv = torch.nn.Conv2d(
    in_channels=9, out_channels=64,
    kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False,
)
backbone.register_module("conv1", conv)

The dataset ships with a pre-defined, geographically separated train/validation split — important for avoiding the over-optimistic metrics that spatial autocorrelation can cause in EO. Within each split, RandomGeoSampler draws 512x512 chips in geographic coordinate space, handling CRS alignment and resolution matching automatically. After 10 epochs with Adam (lr=1e-4, weight_decay=0.01) and a batch size of 4, the model reaches 0.977 overall accuracy and 0.824 IoU on the validation set. Training takes a few minutes on a single GPU.

Inference on a Sentinel-2 scene

This is the part of the tutorial where the model stops being a number on a leaderboard and starts being a useful tool! After training, the notebook downloads a Sentinel-2 scene over Rio de Janeiro, Brazil from the Microsoft Planetary Computer, runs gridded inference across the entire tile, and finally saves the resulting predictions as a georeferenced GeoTIFF.

from torchgeo.datasets import Sentinel2
from torchgeo.samplers import GridGeoSampler

s2_dataset = Sentinel2(paths=scene_dir, bands=bands, res=10, transforms=scale)
grid_sampler = GridGeoSampler(s2_dataset, size=512, stride=448, units=Units.PIXELS)
s2_dataloader = DataLoader(
    s2_dataset, sampler=grid_sampler, batch_size=16, collate_fn=stack_samples
)

The GridGeoSampler tiles the scene into overlapping 512x512 patches (stride=448, so 64 pixels of overlap on each edge). Predictions are stitched back together and saved as a GeoTIFF — tiled, compressed, with overviews — that is pixel-aligned with the input scene:

import rasterio as rio

profile = {
    "driver": "GTiff", "dtype": "uint8", "count": 1,
    "width": img_width, "height": img_height,
    "crs": crs, "transform": transform,
    "compress": "deflate", "tiled": True,
    "blockxsize": 512, "blockysize": 512,
}
with rio.open(output_path, "w", **profile) as dst:
    dst.write(prediction, 1)
    dst.build_overviews([2, 4, 8, 16], rio.enums.Resampling.nearest)

The result is a georeferenced water mask that you can open in QGIS, load into a GIS pipeline, or overlay on the original scene.

This step bridges the gap between “model that scores well on a test set” and “model that produces a useful geospatial product.” It also lets you explore the model’s behavior beyond aggregate metrics: How sharp are the predictions along coastlines? What’s the smallest water feature it can detect? Where does it fail?

Try it yourself

The tutorial is distributed as two executable notebooks, and all you need is a machine with a GPU (a Colab T4 works fine):

• Introduction to TorchGeo — core abstractions (dataset composition, spatiotemporal indexing, geographic samplers)
• Earth Surface Water — the end-to-end case study described in this post

For more detail on the design choices and motivation, see our ICLR 2026 ML4RS Workshop paper. The tutorial also builds on Mauricio Cordeiro’s 3-part Medium series on geospatial analysis with TorchGeo. If you have questions or want to discuss, come find us in the TorchGeo Slack.

What to expect

• Paper highlights — summaries and takes on new GeoAI / GeoML research we’re reading
• Code demos — small, reproducible experiments with TorchGeo and the broader geospatial ML ecosystem
• New models & datasets — quick tours of recently released foundation models, benchmarks, and datasets worth trying
• Geospatial explorations — anything from satellite imagery tricks to fun visualizations to workflow tips

Posts will be short and practical. If something is interesting enough to share in Slack, it’s interesting enough to write up here.

Stay tuned, and come hang out in TorchGeo Slack if you haven’t already.