How a connector works

The mechanics of a connector: the outbound half that polls, maps and publishes, the inbound bridge that answers identity questions, and the design rules the code holds you to.

Two halves

A connector joins one of your existing systems, such as an ERP or a PIM, to the OpenEPCIS platform. It works in two directions that have almost nothing in common. The outbound half reads records out of the source system, maps them onto GS1 terms and publishes them into the catalog behind the resolver. The inbound half is a small HTTP API, the bridge under /v1, which the source system's own user interface calls when it needs an answer the platform holds: is this GTIN sound, whose prefix is it on, what does the linkset currently say, give me the next number.

The two halves are kept apart because they fail in different ways and serve different people. The outbound half runs unattended on a schedule and must never lose or duplicate a record. The inbound half runs while somebody is looking at a screen, so it has to answer in milliseconds, say honestly when it is guessing, and never make a person wait on a registry that has stopped answering.

Everything below describes the shared core that is identical for every source system: the loop, the mapping reader, the catalog client, the hashing and the scheduling. An adapter adds only what is specific to its system. It says which system it speaks for, lists what changed since a cursor, fetches one record by its id, and answers a cheap authenticated ping for the diagnosis ladder. Two further hooks are optional, and an adapter is complete without either of them: writing a value back into the source, and receiving the identity verdict for a record it published. Where the source system offers webhooks, the adapter's module also brings the endpoint that verifies their signatures.

The outbound half: poll, map, publish

The loop

For each connection and each entity kind the sequence is the same. The loop reads the cursor, asks the adapter what changed since then, runs each record through the mapping, derives the sync key and asks the catalog what it holds for that key. If the catalog already holds what the connector would send, the record is skipped; otherwise it is written with a PUT. Then the cursor advances.

The connector keeps no record of its own about what it has published. Whether a record has been published and whether it has changed are both questions it asks the catalog, which is the master data index, so there is no second copy of that truth that could disagree with it. Instance documents for lots and serials are the one exception. The management API accepts a PUT on /products/{gtin}/10/{lot} and offers no GET, so there is nothing to read back, and the connector keeps one row per instance; without it, every instance would look newly published on every tick.

Two clocks set the timing. A global tick, every ten minutes by default, wakes the scheduler, which then honours each connection's own poll-interval (fifteen minutes unless configured otherwise). A nightly full sweep runs on top of that, at 0 30 3 * * ? by default, and can be switched per connection with full-sync-enabled. The sweep matters because a delta poll only sees what the source system remembered to mark as changed, and the sweep catches every record that changed without being marked.

Four properties of the loop matter when something goes wrong:

  • Each cursor advance commits on its own, right after that record's write, so trouble late in a batch keeps the successes before it.
  • A publish failure stops only that entity kind. The rest of the connection carries on, the cursor stays where it was, and the next tick retries from the failed record.
  • An invalid key skips one record and the batch continues. The cursor moves past it, and the record comes back when it changes or on the nightly sweep.
  • Dry run writes nothing: no PUT, no stamp, no event, only a log line naming the path. Arming the connection afterwards still publishes everything.

Delta and cursor

The delta is taken over whatever the source system offers as its notion of "updated at". Systems differ enough here that each adapter manages its own cursor. UnoPim is filtered on updated_at, and since it pages by id, the cursor is kept as a running maximum. ERPNext keeps the last modified timestamp it saw per DocType, because one catalog kind can be fed by several DocTypes: an organization comes from both Supplier and Customer, and each pages independently. metasfresh treats the since value only as an optimisation hint and stores the poll's own start time, in epoch milliseconds, as the cursor.

The hash skip

Most records that come back from a delta poll have not changed in any way that matters to the catalog. A PIM touches updated_at when somebody opens and saves a product without editing a field, and publishing such records would fill version histories with noise and spend write capacity for nothing.

Before publishing, the connector therefore fetches the catalog's own document for that key and compares it with the payload it is about to send, using SHA-256 over canonical JSON with sorted map keys. If the two are identical, nothing is sent.

Two details make this comparison trustworthy. It is projected: only the keys this connection's mapping produces are compared, because a catalog record also carries other connectors' contributions, an editor's corrections and whatever a registry enriched it with, and comparing the whole document would report every record as changed on every tick. Numbers are also normalised before hashing, in two senses: 30 and 30.0 are the same number, and 250 GRM and 0.25 KGM are the same weight. Without either step, a record would differ from its own stored copy forever.

Alongside the comparison, each connection leaves a stamp in the record's connectorStamps field. The stamp is the hash of exactly what the connection last published, keyed by connection name so that several connections can stamp the same record without erasing one another. Where the comparison tells the connector whether anything changed, the stamp tells it which side made the change, and the section on direction below shows how that is used. The stamp lives in the catalog for the same reason the loop keeps no state of its own.

Publish is a PUT

Publishing is a PUT, which the catalog treats as create-or-update and merges into the existing document. This is what makes retries safe. A repeated write of the same document cannot duplicate anything, so after a failure the loop simply leaves the cursor standing and tries again on the next tick. There is no bookkeeping of which writes got through, no compensating delete, and no window in which a crash produces two of something.

What it does not do

People who expect a sync to be symmetrical are often surprised by two things the loop leaves alone: deleting records and producing passports.

The connector never deletes. A delta poll never sees a record that is gone, and a webhook for a record the source no longer knows is logged while the catalog document stays, because removing master data is an explicit decision that a late delivery should never trigger. The same holds one level up: a withdrawn link stays in the linkset and keeps pointing at a page that no longer serves. Undoing either is manual work on the catalog and linkset APIs. A product that is merely deactivated in the source keeps being published; its status travels in the payload, and nothing reads it.

The connector also does not produce a digital product passport. That is the job of the DPP API, a separate application with its own address, which the connector never calls. Its interface is EN 18222, and the passport it returns is EN 18223. With dpp.masterdata.backend=resolver it derives the passport on read from the master data the connector wrote, so no create call is needed. On its default backend it knows only what its own create call put there, and every read ends in a 404. The resolver advertises a gs1:dpp link only once it is told where that service lives, through openepcis.digital-link-resolver.dpp-base-url, which is empty by default and set as part of each deployment.

Whether a passport exists at all depends on one derivation. economicOperatorId is mandatory, and the DPP API derives it from gs1:manufacturer on the product record, either from the organization's own party Digital Link or from one built from its GLN. Source systems rarely keep the manufacturer on every article, because for most companies it is the same company each time, so the connection carries it once: target.economic-operator-gln names the GLN of the economic operator, and the connector sets it as the manufacturer on every product whose source states none. A manufacturer given per article in the source always takes precedence. Without either, the passport read fails, since the API never returns a passport with a gap in it.

A first sync gives you the whole GS1 master data document in the catalog, a resolving Digital Link and its linkset. For a passport on top of that, the connection needs its economic-operator-gln (or a mapping rule of your own onto manufacturer, which the connector vocabulary types as an organization), and the DPP API has to be configured with the resolver backend.

The webhook is a doorbell

Where a source system can call out when something changes, the connector accepts that call as a signal. It reads the payload for one thing only, the identity of the record that moved. It then fetches that record fresh from the source API and passes it into the same loop the poll uses, at the same function.

This rule keeps a connector predictable. A webhook body is a snapshot of somebody else's transaction, taken at a moment you did not choose, and it can go wrong in ways that are invisible from your side. Measured against UnoPim 3.0.0, creating a product fires two identical deliveries, product.created and product.updated; a failed delivery is never retried, because the delivery service swallows its own error; and the create payload omitted the dpp_gtin that had been set in the very same request. A connector that treated the body as data would pass all of that on to the catalog. Because the connector treats it as a doorbell, the worst a bad payload can do is cause one unnecessary re-read.

Poll and webhook share one code path for the same reason. Every fix, every mapping change and every edge case then has to be got right only once, and there is no rarely exercised second path to carry a hidden bug. Signatures are checked before anything else: HMAC-SHA256, in hex or in Frappe's base64 variant, sent as X-Frappe-Webhook-Signature for ERPNext and as X-Unopim-Signature: sha256=<hex> for UnoPim. An unsigned or wrongly signed call gets a 401 and triggers no re-read.

The mapping is data, not code

Authentication, paging and webhooks are code that rarely changes. What differs from one customer to the next is which field in the source system becomes which GS1 term in the catalog, and that is kept in a YAML file, one per connection. Adapting an installation therefore means reviewing one readable document, with no new release.

The mapping is the only file a connection uses. All its other settings, such as credentials, intervals, locale and dry-run, are Quarkus properties under openepcis.connectors.connections.<name>.*, handed to the container as environment variables. One of those properties, mapping-file, takes the path of the mapping to read. The Kubernetes deployment currently mounts no volume for it, so a connection running there uses the mapping the build ships, and a customer-specific file has no way in yet.

A file holds one document per entity family (product, lot, serial, lot-serial, organization), and each document is a list of rules:

system: unopim
entity: product
description: Default mapping for UnoPim products.
rules:
  - source: name
    target: productName
    kind: localized
    direction: both
    description: The merchant's product name, maintained in either system.

Both sides of a rule are dotted paths. brand.brandName addresses a nested object. Missing objects are created on write, and a missing object on read simply yields an absent value without raising an error.

A target segment ending in [] fans the value out into a list, so targetMarket[].targetMarketCountries.countryCode gives a repeatable GS1 term the one-element list it expects.

kind: localized files a value under a language key. "Leinenhemd" becomes {"de": "Leinenhemd"}, using source.options.locale with the region dropped, so de_DE files under de. A source that already sends language maps passes through untouched and needs no locale, which lets the same rule serve a single-language ERP and a multilingual PIM. Only plain text needs the locale. If it is missing, the record is skipped with a message naming the rule and the setting, because the catalog would reject a value filed under a blank language key.

A unit: stamps a UN/CEFACT code onto a bare number, for sources that keep the unit somewhere outside the data. When a rule has no unit: and the source carries its own unit, that unit is converted into whatever the catalog holds, using the platform's own Recommendation 20 table of 721 units across 51 dimensions. The EPCIS repository converts sensor values against the same table, and there is no second one. A conversion between different measurands fails loudly and is never approximated, and so does a unit the table does not know. A unit: on a kind: localized rule is refused outright, since text has no unit.

Unknown keys are rejected as soon as the file is read, at both levels. A misspelled direciton: that silently fell back to the default would be worse than an error, because the only symptom would be a field that quietly never travels.

direction

direction decides who owns a field, and it takes three values.

to-catalog is the default. The source system owns the field, the value travels one way, and the source is never written into. Writing into a customer's system is something a mapping has to ask for explicitly.

from-catalog means the platform owns the field. It is left out of the outgoing document and written into the source, by those adapters that accept a write-back at all. An ERP is usually the authority for what it sends and simply never opts in. This is the right direction for everything the platform derives itself.

both means the field is genuinely maintained in either place.

For both, the stamp decides which edit is more recent. Timestamps cannot do this job: two systems have two clocks and two notions of "modified", and a PIM that touches updated_at on every save would win every race whether or not anything changed. The stamp distinguishes three cases. If the catalog matches what the connector would send, nothing has moved. If the catalog differs but the stamp still matches what the connector would send, the source has not moved since the last publish, so the change was made in the platform; the value travels back into the source and nothing is published. If the catalog differs and the stamp does not match, the source has moved, and the connector publishes. The write-back is best effort. The catalog is correct either way, and a PIM that refuses a courtesy write must not stop a publication.

The direction is always declared on the rule and never inferred from its shape, because a wrong guess fails silently. When both systems claim the same field by mistake, each tick's publish overwrites the catalog edit that the pull-back has just carried into the source. Nothing fails and nothing is logged; the correction disappears, and both sides settle on the stale value.

Two rule shapes cannot run backwards at all, and a file that asks either of them to do so is refused at read time with the rule named. A target that fans out has no single element to write back to, and a rule with a fixed unit has discarded the information about which unit the source meant.

Identifiers do not appear in the mapping file. The adapter reads the GTIN, the GLN and the batch identifier into the sync key, because they say which record this is, and the catalog derives the Digital Link from that key. The JSON-LD type is likewise stamped on the way out. Every target name is checked against a vocabulary index built from the GS1 web vocabulary and the platform's own regulation extensions. The resolver would silently drop a target that no vocabulary knows, so the build fails on one.

The inbound half: the bridge under /v1

The bridge is what an adapter, or a plugin inside the host system, calls when it needs an answer from the platform. It has seven endpoints:

EndpointWhat it answers
POST /v1/products/resolveA normalised payload in, an identity verdict out
GET /v1/products/{gtin}/linksetThe current linkset (?lot= and ?serial= narrow it)
POST /v1/products/{gtin}/linksRegister or update link-type targets
POST /v1/identifiers/allocateDraw a key from the tenant's own prefix
GET /v1/gpcSearch the GPC classification by words
GET /v1/tenants/{id}/quotaAllocation counts by state and prefix
GET /v1/healthWhat this bridge is configured to serve

A tenant here is a configured connection, and its name is the tenant id. Every call names one, in the body or as ?tenantId=, and there is no other tenancy concept. An unknown tenant returns a 404, so a misconfigured adapter finds out immediately and never quietly receives an empty result. The bridge is not published outside the cluster. Callers reach it through the cluster-internal service address on the connector's own HTTP port, which is why a plugin inside a host system is configured with that internal address and no public URL.

Allocation deserves a closer look, because it is the one call with a consequence outside the platform. A drawn key is held as a candidate in the resolver's ledger and is registered with GS1 only when the record using it is saved. Registration upstream cannot be undone, and an abandoned form would otherwise burn a number permanently.

The payload is host-agnostic. Nothing in it knows what a UnoPim value section or an ERPNext DocType is, because each adapter first flattens its own shape into a NormalizedProduct. That is what lets one bridge serve a PIM and an ERP without a branch per host system, and it gives every new adapter the whole inbound half at no extra cost.

The bridge reimplements no GS1 logic. The identifier grammar comes from the shared aitable.json of the Digital Link translator toolkit, which describes 477 application identifiers with their qualifier chains, formats and check-digit positions, and the prefix boundary comes from that same toolkit's GCP Length Table. Verified by GS1, linkset management and key allocation are calls to the resolver, which already holds the credentials and the ledgers; the connector never holds registry credentials at all. The bridge itself adds four things: the normalised payload, the verdict shape, per-fact provenance and the cache policy.

The shape of a verdict

A verdict carries far more than a boolean:

{
  "ai": "01", "value": "9521123456785",
  "canonicalKey": "09521123456785",
  "digitalLink": "https://id.example.test/01/09521123456785",
  "syntacticallyValid": true, "checkDigitValid": true, "offlineValid": true,
  "gcpLength": 7, "gcpOwnership": "foreign_prefix",
  "registryStatus": "verified", "licenseeName": "Muster Handel GmbH",
  "reason": null, "dataProblem": false, "degraded": false,
  "provenance": {
    "syntax":          {"source": "offline",  "observedAt": "..."},
    "registry_status": {"source": "cache",    "observedAt": "..."}
  }
}

provenance is recorded per fact, because a verdict combines facts from several places. A cached entry carries the timestamp of the registry call that produced it, so what a person sees is the age of the knowledge itself. A silent cache hit that somebody mistakes for a live lookup is exactly the kind of trust failure this design is meant to prevent.

Cache lifetimes are set per fact and per tenant. Licensee identity changes rarely and is kept for 75 days, descriptive attributes read from the registry for 30, licence status for 10, and a negative result for only 2. Licence status is the volatile fact, where a stale "active" would be the most misleading thing this cache could serve, and a GTIN that GS1 does not know today is very often one that gets licensed next week. Partitioning by tenant is a confidentiality rule: a fact read under one tenant's licence is never served to another, and the cache has no global fallback lookup anywhere. The store is per process and bounded, so with several replicas a restart can cost a repeated lookup, but it can never produce a wrong or cross-tenant answer.

Design rules

Three rules recur throughout, and the code enforces each of them.

Readiness informs and never blocks. Downstream registries have their own expectations (GS1 Germany wants a GPC code, for instance), and the platform reports them per channel and per entity kind as requiredTerms, for whichever channel a connection publishes into. This information is advisory and feeds the "still needed" hints in a screen or a report. An unmet requirement never holds data back, because a connector that waits for a perfect record publishes nothing on the day it is most needed.

A data problem and our problem are different types. Nothing caused by an outage, a credential or a quota may present itself as bad customer data. dataProblem is true only when a person editing the record can fix what is wrong: bad syntax, a failed check digit, an AI that admits no such value, a qualifier its primary key does not take, or GS1 answering that it does not know the key. It is false for every outage, credential and quota reason. Callers should branch on dataProblem and never on registryStatus. The rule is drawn once in the code: the reason code itself carries the flag, and the exception raised when a registry lookup does not complete cannot be constructed with a reason that claims to be a data problem.

Degraded mode. When the registry or the resolver does not answer, when a credential is missing or rejected, or when the tenant's allowance is spent, the verdict comes back with registryStatus: "unavailable", degraded: true and a reason, with every offline fact still in place. Syntax, check digit, GCP boundary and the canonical Digital Link are computed before anything can fail, so an outage costs you the registry's opinion and can never cost a merchant their good GTIN.

One operational default belongs alongside these rules. A connection starts with dry-run set to true, and arming it takes one explicit line of configuration. Behind the catalog sit real registrars, GS1 production among them, and a rehearsal that quietly wrote would be worse than no rehearsal at all.

Where the boundaries are

Not every source system can be joined in the same way, and the reasons lie in licensing.

metasfresh is licensed GPL-2.0-only, so the connector couples to it across the network and nowhere else. REST v2 is the only interface, there is no metasfresh dependency in any Maven scope, and nothing is copied in: no code, no schemas, no field lists, no sample payloads. Knowledge may come only from the published documentation, the Swagger description of a running instance, and our own observation of HTTP traffic. This is recorded as an invariant that every future change has to keep.

Odoo works the other way round. It brings its own runtime, its own packaging and its own licensing world, so the Odoo connector is a separate repository of Odoo addons under LGPL-3. Its GS1 helper code is written clean-room against the GS1 General Specifications and shares nothing with the server-side components, which lets the addons carry that licence without qualification.

The shared core and the bridge remain proprietary. Adapters count as distribution and may be published.

Last updated: