Caching Weather API Responses for Offline Field Operations

Problem statement

A sprayer sitting at the edge of a field needs one answer: is the wind under threshold right now? The weather it depends on comes from a hosted API, and the one place that question gets asked — a tractor cab three kilometres past the last cell tower — is exactly where the network is least reliable. When the request fails, a naive client either blocks the operator behind a spinner or returns nothing, and the application timing decision that depends on it stalls. Meanwhile the same client, back on a strong connection, re-requests identical data for every fix a machine reports, burning through the quota covered by rate-limit handling for crop models.

A caching layer resolves both failures at once, but only if it is built deliberately. Three concrete problems recur:

• Connectivity gaps mid-operation. A field pass runs for hours across variable coverage. A cache that only serves data inside a strict freshness window is useless the moment the link drops; the operator needs the last good reading served as an explicit fallback, not a hard failure.
• Redundant calls hammering the quota. Dozens of fixes land within the same square kilometre in the same minute. Keying the cache on raw coordinates treats each one as a unique request, defeating the cache entirely and pushing straight into the rate limiter. Nearby fixes must collapse onto one key.
• A single provider being a single point of failure. When the primary API returns a 503 or times out, a client with no fallback has nowhere to go even though a secondary provider would answer. Provider order needs to be part of the cache’s refresh path, not bolted on later.

The remedy is one cache abstraction with four properties: a time-to-live (TTL) that defines freshness, spatial grid keys that collapse nearby fixes, a stale-while-offline mode that serves the last good value with an explicit flag, and an ordered provider list so a failed primary falls through to a backup before anything is served stale. The values this cache feeds are consumed by weather window logic, which is why every stale response must be labelled — a downstream timing decision has to know it is acting on aged data.

Parameter reference table

Every value below shifts the balance between freshness, quota consumption, and how long a decision can keep running through an outage. Recommended values assume 15-minute weather refresh cadence and edge controllers that lose the link for minutes at a time.

Keep these in one versioned config object rather than scattering literals through call sites, so a single tuning change re-applies everywhere and the audit trail records which policy was in force.

Runnable implementation

The module below is a self-contained cache layer targeting Python 3.10+ (uses match/case and X | None unions), fully typed, built on httpx and a Protocol-based backend so Redis, SQLite, or an on-disk map all satisfy the same contract. All timestamps are tz-aware UTC.

from __future__ import annotations

import logging
from collections.abc import Awaitable, Callable
from dataclasses import dataclass
from datetime import datetime, timezone
from typing import Any, Protocol

import httpx

logger = logging.getLogger("weather.cache")
UTC = timezone.utc

# A provider is an async callable returning a decoded JSON payload for one cell.
Provider = Callable[[float, float, str], Awaitable[dict[str, Any]]]


@dataclass(frozen=True)
class CacheEntry:
    payload: dict[str, Any]
    stored_at: datetime  # tz-aware UTC; naive datetimes are a programming error
    provider: str


class CacheBackend(Protocol):
    """Minimal async KV contract; satisfied by Redis, SQLite, or a dict."""

    async def get(self, key: str) -> CacheEntry | None: ...
    async def set(self, key: str, entry: CacheEntry) -> None: ...


@dataclass
class CacheConfig:
    ttl_seconds: int = 900          # freshness window for spray / irrigation calls
    grid_deg: float = 0.05          # ~5.5 km cell collapses nearby fixes to one key
    stale_max_seconds: int = 21_600  # 6 h: oldest a stale entry may be served offline
    timeout_seconds: float = 4.0    # per-provider HTTP timeout
    offline: bool = False           # driven by the link monitor; forces stale-serve


def grid_key(lat: float, lon: float, variable: str, cfg: CacheConfig) -> str:
    """Snap a fix to its grid cell so nearby calls share one cache entry."""
    cell_lat = round(lat / cfg.grid_deg) * cfg.grid_deg
    cell_lon = round(lon / cfg.grid_deg) * cfg.grid_deg
    return f"wx:{variable}:{cell_lat:.3f}:{cell_lon:.3f}"


class WeatherCache:
    def __init__(
        self,
        backend: CacheBackend,
        providers: list[tuple[str, Provider]],  # ordered: primary first
        cfg: CacheConfig,
    ) -> None:
        self._backend = backend
        self._providers = providers
        self._cfg = cfg

    def _age_seconds(self, entry: CacheEntry) -> float:
        return (datetime.now(UTC) - entry.stored_at).total_seconds()

    async def get(self, lat: float, lon: float, variable: str) -> dict[str, Any]:
        key = grid_key(lat, lon, variable, self._cfg)
        cached = await self._backend.get(key)
        if cached is not None and self._age_seconds(cached) <= self._cfg.ttl_seconds:
            logger.info('{"event":"cache_hit","key":"%s","age_s":%.0f}',
                        key, self._age_seconds(cached))
            return cached.payload
        if self._cfg.offline:  # link is down: never touch the network, serve last good
            return self._serve_stale(key, cached, reason="offline")
        return await self._refresh(key, lat, lon, variable, cached)

    async def _refresh(
        self, key: str, lat: float, lon: float, variable: str,
        cached: CacheEntry | None,
    ) -> dict[str, Any]:
        for name, provider in self._providers:  # primary, then fallbacks in order
            try:
                payload = await provider(lat, lon, variable)
            except (httpx.HTTPError, ValueError) as exc:
                logger.warning('{"event":"provider_failed","provider":"%s",'
                               '"key":"%s","error":"%s"}', name, key, type(exc).__name__)
                continue  # fall through to the next provider before serving stale
            entry = CacheEntry(payload, datetime.now(UTC), name)
            await self._backend.set(key, entry)
            logger.info('{"event":"cache_refresh","key":"%s","provider":"%s"}', key, name)
            return payload
        return self._serve_stale(key, cached, reason="all_providers_failed")

    def _serve_stale(
        self, key: str, cached: CacheEntry | None, reason: str,
    ) -> dict[str, Any]:
        if cached is None:  # nothing cached and no provider reachable: hard failure
            logger.error('{"event":"cache_miss_no_stale","key":"%s","reason":"%s"}',
                         key, reason)
            raise LookupError(f"no cached weather for {key} and no provider reachable")
        age = self._age_seconds(cached)
        if age > self._cfg.stale_max_seconds:  # too old to trust for a decision
            logger.error('{"event":"stale_expired","key":"%s","age_s":%.0f,'
                         '"limit_s":%d}', key, age, self._cfg.stale_max_seconds)
            raise LookupError(f"stale weather for {key} exceeds limit")
        flagged = dict(cached.payload)
        flagged["stale_while_offline"] = True  # downstream MUST see it is aged
        flagged["stale_age_seconds"] = round(age)
        logger.warning('{"event":"serve_stale","key":"%s","age_s":%.0f,"reason":"%s"}',
                       key, age, reason)
        return flagged


def open_meteo_provider(client: httpx.AsyncClient, cfg: CacheConfig) -> Provider:
    async def _fetch(lat: float, lon: float, variable: str) -> dict[str, Any]:
        resp = await client.get(
            "https://api.open-meteo.com/v1/forecast",
            params={"latitude": lat, "longitude": lon, "hourly": variable},
            timeout=cfg.timeout_seconds,
        )
        resp.raise_for_status()
        return resp.json()

    return _fetch

The design keeps one invariant: nothing is ever served stale without the stale_while_offline flag, and nothing older than stale_max_seconds is served at all. That flag is what lets weather window logic refuse to open a spray window on data it knows is aged, rather than treating a six-hour-old wind reading as current.

Log patterns and observable signals

Every path emits structured JSON so a decision can be traced to the exact entry, provider, and age it acted on.

Fresh hit (the common, healthy case):

{"event": "cache_hit", "key": "wx:wind_speed_10m:41.850:-93.600", "age_s": 240}

Refresh from the primary provider (a miss or an expired entry, link up):

{"event": "cache_refresh", "key": "wx:wind_speed_10m:41.850:-93.600", "provider": "open_meteo"}

Primary failed, fallback about to be tried:

{"event": "provider_failed", "provider": "open_meteo", "key": "wx:wind_speed_10m:41.850:-93.600", "error": "ReadTimeout"}

Serving stale while offline (expected during a coverage gap — but every consumer must honor the flag):

{"event": "serve_stale", "key": "wx:wind_speed_10m:41.850:-93.600", "age_s": 3120, "reason": "offline"}

Hard failure (stale entry exceeded the limit, or nothing was cached at all):

{"event": "stale_expired", "key": "wx:wind_speed_10m:41.850:-93.600", "age_s": 25200, "limit_s": 21600}

When triaging, watch the ratio of serve_stale to cache_hit. A brief spike during a known dead zone is normal; a sustained rise means the link monitor is stuck in offline, or the refresh path is failing silently. A climbing provider_failed rate on the primary with clean fallbacks is your early warning to check the primary’s status before it exhausts the budget tracked in rate-limit handling for crop models.

Safe override protocol

Occasionally an operator must extend the stale window past stale_max_seconds — a multi-hour operation deep in a valley where no provider is reachable but the last reading is still directionally useful. The override must never silently widen the limit for everyone or strip the stale flag; it only grants a bounded, audited exception for one call.

Guard conditions, all mandatory:

1. Explicit ceiling, never unbounded. The override supplies a concrete extended limit; there is no “serve any age” mode.
2. Flag is never removed. The returned payload still carries stale_while_offline and its true age, so the timing engine can still veto a marginal decision.
3. Operator identity and reason recorded. Who approved the extension and why is written to an append-only ledger before the value is served.
4. Single call, no persistence. The extension applies to one lookup and expires; it does not mutate the shared CacheConfig.

from datetime import datetime, timezone

UTC = timezone.utc


async def get_with_extended_stale(
    cache: WeatherCache,
    lat: float,
    lon: float,
    variable: str,
    extended_limit_s: int,  # explicit, bounded; no default "serve anything"
    approver: str,
    reason: str,
) -> dict[str, Any]:
    key = grid_key(lat, lon, variable, cache._cfg)
    cached = await cache._backend.get(key)
    if cached is None:
        raise LookupError(f"no cached weather for {key}; override cannot invent data")
    age = (datetime.now(UTC) - cached.stored_at).total_seconds()
    if age > extended_limit_s:  # even the override has a hard ceiling
        raise LookupError(f"age {age:.0f}s exceeds extended limit {extended_limit_s}s")
    flagged = dict(cached.payload)
    flagged["stale_while_offline"] = True  # flag is never stripped by an override
    flagged["stale_age_seconds"] = round(age)
    flagged["stale_override"] = {"approver": approver, "reason": reason}
    logger.warning('{"event":"stale_override","key":"%s","age_s":%.0f,'
                   '"approver":"%s","reason":"%s"}', key, age, approver, reason)
    return flagged

Requiring extended_limit_s and approver as arguments with no defaults is what stops a convenience escape hatch from quietly becoming the default behavior for every offline read.

Troubleshooting

• Hit rate is near zero and calls track the request count one-to-one. Root cause: grid_deg is too fine (or keys include a timestamp), so every fix is a unique key. Remediation: coarsen grid_deg to ~0.05 and confirm grid_key excludes per-request time; freshness belongs to the TTL, not the key.
• serve_stale fires constantly even with a good link. Root cause: the link monitor left offline=True, so the router never attempts a refresh. Remediation: verify the connectivity probe clears the flag on recovery, and alert on any offline window longer than a few minutes.
• A downstream spray window opened on visibly old data. Root cause: a consumer ignored stale_while_offline. Remediation: treat the flag as load-bearing in weather window logic — a flagged payload should tighten or veto a marginal window, never pass silently.
• The primary provider’s outage takes the whole cache down. Root cause: only one provider is registered, so a failed refresh has nowhere to fall through. Remediation: register at least one fallback in the ordered provider list and confirm provider_failed is followed by a cache_refresh from the backup.
• Intermittent stale_expired errors mid-operation. Root cause: stale_max_seconds is shorter than the longest realistic dead zone. Remediation: size it to the worst-case coverage gap for the region, and route genuine extensions through the audited override rather than raising the global ceiling.

Frequently asked questions

How is stale-while-offline different from just extending the TTL? Extending the TTL makes old data count as fresh for everyone and hides its age. Stale-while-offline keeps freshness strict but adds an explicit, flagged fallback that only triggers when the network is down, so consumers still know the value is aged.

Why key on a spatial grid instead of raw coordinates? Nearby fixes in the same minute would each be a unique key and defeat the cache. Rounding lat/lon to a ~5.5 km cell collapses them onto one key so a single response answers a whole cluster of decisions.

What happens when the primary provider fails? The refresh path falls through an ordered provider list to a backup before anything stale is served; only if every provider fails and a recent value is cached is a flagged stale response returned.

Related

• Handling weather API rate limits for crop models — the quota budget this cache protects by collapsing redundant calls.
• Weather window logic — the timing engine that consumes these responses and must honor the stale flag.

Up: Weather API Integration · Farm Data Ingestion & Field Boundary Synchronization