Connecting the John Deere API to a Python Backend: Integration & Troubleshooting Guide

Problem statement

Integrating the John Deere Operations Center (Axiom) API into a production Python backend is rarely defeated by the initial handshake — it is defeated in the middle of a multi-page sync, when a token silently expires or a connection resets and the backend cannot prove where it stopped. This page covers that exact edge case: preventing silent data drops across the asynchronous request cycles that iterate a field, machine, or operation dataset.

The concrete failure this prevents is a partial sync masquerading as a complete one. A worker fetches page 1 of /organizations/{orgId}/fields, holds a bearer token that crosses its expiry boundary on page 7, receives a 401 Unauthorized, and — because a naive loop treats any non-2xx as “no more data” — exits cleanly with six pages persisted and no error raised. Downstream, the Equipment Telemetry Parsing layer then reconciles machine positions against a field set that is missing a quarter of its boundaries, and prescription zones are generated against stale geometry. The sync looks successful in every log; only the record count is wrong.

The remedy has three parts, all implemented below: refresh the OAuth 2.0 token proactively at 80% of its lifespan under a lock, follow the Axiom API’s HATEOAS nextPage links rather than guessing offsets, and checkpoint the last successfully persisted cursor so an interrupted run resumes instead of restarting. Raw payloads pulled here flow up into the Farm Data Ingestion & Field Boundary Synchronization pipeline, and boundary geometry itself is validated by the ISOXML and shapefile parsing routines once it lands.

Parameter reference table

Every value below changes how the client behaves at the boundary between token lifecycle and pagination. Recommended values assume the Operations Center production tier and an intermittently connected edge gateway.

Parameter	Type	Recommended value	Effect on behavior
`refresh_threshold_pct`	`float`	`0.80`	Fraction of token lifespan after which a proactive refresh fires. Below ~0.5 wastes refreshes; above ~0.9 risks a request riding an expiring token on a slow page.
`max_retry_attempts`	`int`	`3`	Attempts per request across `401`/`429`/timeout before the page is abandoned. Higher masks systemic auth faults; lower drops pages on transient blips.
`backoff_factor`	`float`	`1.5`	Base for exponential backoff (`backoff_factor ** attempt`) when no `Retry-After` header is present.
`cursor_batch_size`	`int`	`50`	Axiom `itemLimit` per page. Values above ~100 inflate serialization time and raise `504` risk on boundary-heavy orgs; drop to `25` under timeouts.
`checkpoint_interval`	`int`	`10`	Pages between checkpoint writes. Smaller means less re-work on resume but more checkpoint I/O.
`timeout_seconds`	`float`	`30.0`	Per-request read timeout. Large spatial payloads need headroom; too low converts slow pages into false failures.
`fallback_on_reset`	`str`	`resume_from_checkpoint`	Recovery behavior after a connection reset. Never `restart` in production sync loops — full re-ingestion can exhaust the rate limit.

Register the OAuth scopes (ag1, ag2, ag3, eq1, org1, offline_access) in the John Deere Developer Portal so they match the request exactly; a scope the token lacks surfaces as 403 Forbidden, not 401, and no amount of retrying fixes it. Backoff and jitter tuning shared with other feeds lives under Async Polling Strategies.

Runnable implementation

The client below serializes token refresh under an asyncio.Lock, refreshes proactively at 80% of lifespan, forces a refresh on any 401, honors Retry-After on 429, and follows the Axiom HATEOAS nextPage link so pagination never depends on a guessed offset. It targets Python 3.10+, is fully typed, and streams field records as an async iterator so the caller can checkpoint between pages.

from __future__ import annotations

import asyncio
import time
from dataclasses import dataclass, field
from typing import Any, AsyncIterator

import httpx

TOKEN_URL = "https://signin.johndeere.com/oauth2/aus78tnlaysMraFhC1t7/v1/token"
API_BASE = "https://partnerapi.deere.com/platform"
AXIOM_ACCEPT = "application/vnd.deere.axiom.v3+json"


@dataclass
class TokenState:
    access_token: str
    refresh_token: str  # John Deere rotates this on every refresh
    refresh_at: float   # monotonic deadline; refresh once we pass it


def _next_link(page: dict[str, Any]) -> str | None:
    # Axiom is HATEOAS: the cursor lives in a 'nextPage' link rel, never an offset.
    for link in page.get("links", []):
        if link.get("rel") == "nextPage":
            return link.get("uri")
    return None  # absence of nextPage is the ONLY reliable end-of-data signal


@dataclass
class JohnDeereClient:
    client_id: str
    client_secret: str
    seed_refresh_token: str
    refresh_threshold_pct: float = 0.80
    max_retry_attempts: int = 3
    backoff_factor: float = 1.5
    _state: TokenState | None = field(default=None, init=False)
    _lock: asyncio.Lock = field(default_factory=asyncio.Lock, init=False)

    async def _refresh(self, http: httpx.AsyncClient) -> None:
        # Serialize refreshes: concurrent page fetches must not each mint a token
        # and invalidate one another's rotated refresh_token.
        async with self._lock:
            if self._state and time.monotonic() < self._state.refresh_at:
                return  # a peer coroutine already refreshed while we waited
            prior = self._state.refresh_token if self._state else self.seed_refresh_token
            resp = await http.post(
                TOKEN_URL,
                data={"grant_type": "refresh_token", "refresh_token": prior},
                auth=(self.client_id, self.client_secret),
            )
            resp.raise_for_status()
            body = resp.json()
            ttl = int(body["expires_in"])
            self._state = TokenState(
                access_token=body["access_token"],
                refresh_token=body["refresh_token"],  # persist the rotated token
                refresh_at=time.monotonic() + ttl * self.refresh_threshold_pct,
            )

    async def _get(self, http: httpx.AsyncClient, url: str,
                   params: dict[str, Any] | None) -> dict[str, Any]:
        for attempt in range(1, self.max_retry_attempts + 1):
            if self._state is None or time.monotonic() >= self._state.refresh_at:
                await self._refresh(http)
            assert self._state is not None
            headers = {
                "Authorization": f"Bearer {self._state.access_token}",
                "Accept": AXIOM_ACCEPT,
            }
            resp = await http.get(url, headers=headers, params=params)
            if resp.status_code == 401:
                self._state = None  # stale despite the clock; force a refresh
                continue
            if resp.status_code == 429:
                wait = float(resp.headers.get("Retry-After",
                                              self.backoff_factor ** attempt))
                await asyncio.sleep(wait)
                continue
            resp.raise_for_status()
            return resp.json()
        raise RuntimeError(f"exhausted {self.max_retry_attempts} attempts for {url}")

    async def iter_fields(self, org_id: str,
                          cursor_batch_size: int = 50) -> AsyncIterator[dict[str, Any]]:
        async with httpx.AsyncClient(base_url=API_BASE, timeout=30.0) as http:
            url: str | None = f"/organizations/{org_id}/fields"
            params: dict[str, Any] | None = {"itemLimit": cursor_batch_size}
            while url:
                page = await self._get(http, url, params)
                for record in page.get("values", []):
                    yield record
                url = _next_link(page)  # follow the cursor, don't compute it
                params = None           # the nextPage URI already carries state

Two choices are load-bearing. First, refresh_at is derived from time.monotonic(), not wall-clock time.time(), so an NTP correction on an edge gateway can never make a valid token look expired or vice-versa. Second, end-of-data is decided only by the absence of a nextPage link — never by an empty values array, which Axiom can legitimately return mid-stream while more pages remain.

Log patterns and observable signals

Every request cycle emits one structured line carrying the endpoint, the token state, and — on success — the cursor that was just persisted, so any interrupted run is reconstructable.

Success path (proactive refresh, then a clean page with a checkpoint):

[INFO]  2026-06-14T14:22:01Z | token.refresh | reason=proactive | ttl_pct=0.80 | ttl_remaining=312s
[INFO]  2026-06-14T14:22:02Z | page.ok | endpoint=/organizations/AB12/fields | items=50 | cursor=eyJpZCI6IjEyMzQ1In0= | checkpoint=written

Warning (a 401 crossed the expiry boundary; the client self-heals):

[WARN]  2026-06-14T14:22:05Z | page.401 | endpoint=/organizations/AB12/fields | stale_token=true | action=force_refresh
[INFO]  2026-06-14T14:22:06Z | page.ok | endpoint=/organizations/AB12/fields | items=50 | retry_count=1

Recovery (connection reset; resume from the last checkpoint rather than restart):

[INFO]  2026-06-14T15:10:33Z | conn.reset | last_cursor=eyJpZCI6IjEyMzQ1In0= | state=checkpoint_loaded
[INFO]  2026-06-14T15:10:34Z | page.resume | cursor=eyJpZCI6IjEyMzQ1In0= | retry_count=1

Error (rate ceiling with no recovery left):

[ERROR] 2026-06-14T15:12:40Z | page.429 | endpoint=/organizations/AB12/fields | rate_limit_exceeded=true | attempts=3 | action=abort_and_checkpoint

When triaging, filter on page.ok and diff the emitted cursor values against the persisted checkpoint log. A gap between two cursor tokens with no intervening checkpoint=written is the signature of a page that was fetched but not durably stored — exactly the silent drop this design exists to prevent.

Safe override protocol

Two overrides are legitimate here, and both must remain auditable rather than becoming a way to skip safety logic.

Forced token refresh. Expose a path that discards _state and re-authenticates immediately, ignoring the 80% clock. Use it only when a 401 persists across a refresh (a revoked or rotated-out token). Guard it: never call it inside the retry loop more than once per page, or a genuinely revoked credential turns into an infinite refresh storm against the token endpoint.
Checkpoint bypass. Only skip checkpoint validation during an initial historical backfill, where re-ingesting from cursor zero is intended. In a production sync loop this is prohibited — set fallback_on_reset = resume_from_checkpoint, because a bypass here triggers full-dataset re-ingestion and can exhaust the daily rate limit for the whole organization.

async def force_refresh(client: JohnDeereClient) -> None:
    # Guard: caller must confirm a 401 survived the normal proactive refresh
    # before invoking; unconditional use defeats the lifespan optimization.
    async with httpx.AsyncClient(base_url=API_BASE, timeout=30.0) as http:
        client._state = None
        await client._refresh(http)

Every override emits its own audit line (override=force_refresh or override=checkpoint_bypass) with the operator identity, and those lines are appended to the same immutable ledger the compliance layer consumes — an override is an auditable deviation, never an exemption.

Troubleshooting

Record count short but every log line reads page.ok. Root cause: pagination terminated on an empty values array instead of a missing nextPage link. Remediation: confirm end-of-data is driven solely by _next_link() returning None; Axiom can return an empty page mid-stream.
page.401 repeats every few pages under concurrency. Root cause: parallel coroutines each refreshed and rotated the refresh_token, invalidating peers. Remediation: ensure all refreshes pass through the single asyncio.Lock and that the rotated refresh_token is written back to _state, as shown above.
page.429 | rate_limit_exceeded=true in bursts. Root cause: retries fire without jitter, so backed-off workers resynchronize and hammer the endpoint together. Remediation: add jitter to the backoff (see Async Polling Strategies) and lower cursor_batch_size to reduce per-page cost.
504 Gateway Timeout on boundary-heavy organizations. Root cause: large spatial payloads exceed the serialization window at itemLimit=50. Remediation: drop cursor_batch_size to 25; the extra pages cost less than repeated timeouts.
403 Forbidden | scope_mismatch that no retry clears. Root cause: the token was minted without a scope the endpoint requires. Remediation: re-register the exact scope set in the Developer Portal and re-authorize; 403 is an authorization defect, not a transient fault, so retrying only wastes quota.

Overlapping records during headland turns, coordinate-precision drift, and polygon self-intersection are resolved downstream — see the ISOXML and shapefile parsing reference for the deduplication and topology repair matrix. For non-blocking I/O patterns behind this client, the Python asyncio documentation is authoritative, and the refresh flow follows the confidential-client guidance in RFC 6749.

Parsing ISOXML and Shapefile Field Boundaries — validates and repairs the geometry this client ingests.
Async Polling Strategies — backoff, jitter, and scheduling shared across every upstream feed.
Handling Weather API Rate Limits — the same 429/backoff discipline applied to a sibling data source.

Up: Equipment Telemetry Parsing · Farm Data Ingestion & Field Boundary Synchronization