Skip to content
OWOX Data Marts

Declarative Connector Manifest Reference

A declarative connector manifest is a single JSON document that tells OWOX Data Marts how to pull data from a third-party HTTP API — no code required. You describe the API’s shape (base URL, authentication, endpoints, pagination, how a raw record maps to output fields) and the engine (ManifestParser + DeclarativeSource) does the rest: making requests, paging through results, applying your date window, filtering and transforming records, and casting fields to their declared types.

This page is the complete grammar the engine accepts, written for a person authoring or reviewing a manifest by hand — every key, every enum value, and the mistakes that most often make the parser reject a manifest. To build a connector step by step in the UI instead, see Connector Builder.

Building a connector with an AI assistant? Give your assistant the authoring guide for AI assistants together with the target API’s documentation, then import the manifest it writes in the connector builder (⋮ → Import JSON…).

Contents

Section titled “Contents”

How the engine turns responses into rows

Section titled “How the engine turns responses into rows”

Every node runs the same pipeline, in this order:

fetch (+ pagination) → recordFilter → transformations → field projection/casting (fields)

The step that most often trips up a new manifest is the first one: how a raw HTTP response becomes a set of rows. recordSelector.recordPath (or, for an async node, download.recordPath) is an array of keys that walks the JSON response down to the node holding your records. From there, the engine turns that node into rows in exactly two ways:

  • An array → one row per element.
  • A single object → exactly one row.

There is no third case. The engine cannot turn an object’s keys into rows, and there is no “current key” available to a field mapping. So never point a node at an endpoint that returns an object keyed by entity id, such as:

{ "bitcoin": { "usd": 65000 }, "ethereum": { "usd": 3400 } }

There is no recordPath that produces one row per currency here. Prefer an endpoint that returns an array instead, or — if the API only offers the dictionary shape — fetch one entity per run via a parameter or a partition router value.

Once records are selected, recordFilter (optional, keep-or-drop) runs first, then transformations (optional, add/remove/reshape), and finally each field’s dataPath/apiName reads its value out of the (post-filter, post-transformation) record and casts it to the declared type.

Top-level manifest keys

Section titled “Top-level manifest keys”
KeyRequiredDescription
versionYesAlways the string "1.0".
nameYesThe connector’s identifier: letters, digits and underscores, starting with a letter — PascalCase by convention, e.g. MolocoCloud. It cannot be changed after the connector is created.
baseUrlYesThe address every request path is appended to, over HTTPS, e.g. https://api.example.com or https://api.example.com/v1. Leave off the trailing slash.
parametersYesAn object of user-supplied inputs — may be {} if the connector needs none. See Parameters.
nodesYesAn object keyed by node name; each value is one data stream. See Nodes.
titleNoA human-friendly display name, e.g. "Frankfurter FX (Declarative)".
descriptionNoA longer description shown in the UI.
docUrlNoA link to the API’s own documentation.
authenticationNoHow requests are authenticated. See Authentication.
accountsNoFan a single run out across multiple account IDs. See Multi-account fan-out.
rateLimitNoA global request-rate cap. See Rate limiting.

Common naming mistakes

Section titled “Common naming mistakes”

The parser does not check for unknown keys. A misspelled key is silently ignored: the manifest publishes, and the mistake only shows when you test the node — for example, a 401 because an auth block was never read. These five are the most common:

  • The auth block is authentication, not auth.
  • A request’s query string is queryParameters, not queryParams.
  • A record’s row-selector is recordSelector.recordPath — an array of keys — not fieldPath.
  • fields is an object keyed by field name, not an array.
  • A field’s dataPath/apiName is a single dot-string (e.g. "stats.spending"), but almost every other “path” in the grammar — recordSelector.recordPath, recordFilter.path, errorHandler’s bodyMatch.path, pagination’s cursor.path/stopCondition.path, an async node’s jobIdPath/statusPath/resultUrlPath/download.recordPath, partitionRouter.parent.recordPath, incremental.request.startPath/endPath, and authentication.exchange.tokenPath — is an array of key segments (e.g. ["stats", "spending"]). The parser refuses a dot-string in any of those array positions, so that mistake fails at publish. The reverse is silent: an array in a field’s dataPath is accepted, and the field is always empty.

Parameters

Section titled “Parameters”

Each entry in parameters describes one user-supplied input, referenced elsewhere via {{ parameters.Name }}:

{
  "ApiKey": {
    "requiredType": "string",
    "isRequired": true,
    "label": "API Key",
    "description": "Found under Settings → API in your account.",
    "attributes": ["SECRET"]
  },
  "Region": {
    "requiredType": "string",
    "isRequired": false,
    "default": "us",
    "label": "Region"
  }
}
FieldMeaning
requiredTypeOne of string, number, boolean, date.
isRequiredBoolean — this, not an attribute, is what makes the parameter mandatory.
defaultOptional default used when the user leaves the field blank.
labelOptional display label shown in the configuration form.
descriptionOptional help text shown next to the field.
attributesOptional flags. The one every author should know is SECRET — it masks the value in the UI and in API responses, stores it apart from the Data Mart’s definition and replaces it with *** in run logs and errors (it is not encrypted); always set it for API keys, client secrets, tokens, and passwords. A few advanced/internal flags also exist (HIDE_IN_CONFIG_FORM, ADVANCED, OAUTH_FLOW, DEPRECATED, PINNED, MANUAL_BACKFILL) but are rarely needed when hand-authoring a manifest.

Marking a parameter SECRET only changes how the manifest declares it — the user still enters the actual value themselves, through the connector’s configuration form in the browser. A manifest should never contain a real credential value.

A parameter name starts with a letter and contains only letters, digits and underscores, so that templates can refer to it. It also cannot be a name the engine or a storage uses for its own settings: Fields, LastRequestedDate, DestinationTableName, DestinationTableNameOverride, MaxBufferSize, or a storage setting such as ProjectID or ServiceAccountJson (BigQuery), Schema or Database (Redshift) or AWSRegion (Athena, Redshift). The parser refuses those names, so Test and Publish fail until the parameter is renamed.

Authentication

Section titled “Authentication”

If the API needs no authentication, omit authentication entirely. Otherwise, authentication.type must be one of apiKey, bearer, basic, tokenExchange, oauth2, selective.

API key

Section titled “API key”

Injects a templated value into a header or query parameter:

{
  "type": "apiKey",
  "inject": { "into": "header", "name": "Api-Key", "format": "{{ parameters.ApiKey }}" }
}

inject.into is "query" or "header"; inject.name is the parameter/header name; inject.format is a template string — often just {{ parameters.X }}, but it can wrap it, e.g. "ApiKey {{ parameters.ApiKey }}".

Bearer token

Section titled “Bearer token”

The same injection mechanism, fixed to an Authorization: Bearer <token> header:

{
  "type": "bearer",
  "inject": { "into": "header", "name": "Authorization", "format": "Bearer {{ parameters.Token }}" }
}

Basic auth

Section titled “Basic auth”

HTTP Basic auth — no inject block; the engine base64-encodes username:password itself:

{
  "type": "basic",
  "username": "{{ parameters.Username }}",
  "password": "{{ parameters.Password }}"
}

Some APIs, such as Stripe, take the API key as the username with an empty password. Leave password out for them: a username without a password is treated as the credential, and its parameter is marked SECRET.

Token exchange

Section titled “Token exchange”

Exchanges a credential for a server-issued token via one POST, then injects that token. Use this for APIs with a simple “trade my API key for a session token” step and a fixed-length token lifetime:

{
  "type": "tokenExchange",
  "exchange": {
    "method": "POST",
    "url": "https://api.example.com/auth/tokens",
    "body": { "api_key": "{{ parameters.ApiKey }}" },
    "tokenPath": ["token"],
    "ttlSeconds": 3600
  },
  "inject": { "into": "header", "name": "Authorization", "format": "Bearer {{ auth.token }}" }
}

exchange.tokenPath is an array locating the token in the JSON response. The token is cached for exchange.ttlSeconds (300 if unset) and re-issued once it expires. inject templates read the cached token via {{ auth.token }}.

OAuth2

Section titled “OAuth2”

A standard OAuth2 token endpoint (refresh_token or client_credentials grant). Prefer this over tokenExchange for Google/Microsoft/Facebook/LinkedIn-class APIs, and whenever the provider returns an expires_in or may rotate the refresh token:

{
  "type": "oauth2",
  "tokenUrl": "https://oauth2.googleapis.com/token",
  "grantType": "refresh_token",
  "clientId": "{{ parameters.ClientId }}",
  "clientSecret": "{{ parameters.ClientSecret }}",
  "refreshToken": "{{ parameters.RefreshToken }}",
  "scope": "https://www.googleapis.com/auth/spreadsheets.readonly",
  "ttlSeconds": 300,
  "inject": { "into": "header", "name": "Authorization", "format": "Bearer {{ auth.token }}" }
}
FieldNotes
tokenUrlRequired.
grantType"refresh_token" (default) or "client_credentials".
clientId / clientSecretRequired.
refreshTokenRequired when grantType is "refresh_token".
scopeOptional, space-separated OAuth scopes.
ttlSecondsOptional fallback cache TTL, used only when the token response omits expires_in — defaults to 300s if also unset. When the response does include expires_in, that value (minus a 60s safety skew) drives the cache lifetime instead.
injectRequired, same shape as the other auth types; always reads {{ auth.token }}.

On a refresh_token grant, if the provider rotates the refresh token (returns a new one in the response), the engine keeps using the new one for the rest of the run and reports it back so it is persisted for the next run — but only for a connector saved with credentials in the platform’s credential store, not for values typed inline as plain parameters. A GeneratedRefreshToken parameter is auto-registered by the engine for every oauth2 manifest; don’t declare it yourself.

Selective

Section titled “Selective”

Picks one of several authentication branches at runtime based on a parameter’s value — useful when the same connector supports multiple auth modes (e.g. “API Key” vs “OAuth”):

{
  "type": "selective",
  "selectionParameter": "AuthMode",
  "authenticators": {
    "apikey": {
      "type": "apiKey",
      "inject": { "into": "header", "name": "Api-Key", "format": "{{ parameters.ApiKey }}" }
    },
    "oauth": {
      "type": "oauth2",
      "tokenUrl": "https://oauth2.googleapis.com/token",
      "clientId": "{{ parameters.ClientId }}",
      "clientSecret": "{{ parameters.ClientSecret }}",
      "refreshToken": "{{ parameters.RefreshToken }}",
      "inject": { "into": "header", "name": "Authorization", "format": "Bearer {{ auth.token }}" }
    }
  }
}

selectionParameter names a manifest parameter whose runtime value picks the branch key — here, {{ parameters.AuthMode }} must resolve to "apikey" or "oauth". Each branch is itself apiKey | bearer | basic | tokenExchange | oauth2 — never another selective (no nesting).

Nodes

Section titled “Nodes”

nodes is an object keyed by node name. Node names, field names and destinationName become table and column names, so they contain only letters, digits and underscores; a node name starts with a letter, the others with a letter or an underscore. uniqueKeys and defaultFields list the node’s own field names. For an upstream key such as created-at, name the field created_at and set its dataPath to created-at.

Each node describes one data stream:

{
  "overview": "One-line human description of what this node returns.",
  "uniqueKeys": ["date", "id"],
  "destinationName": "my_table_name",
  "isTimeSeries": true,
  "defaultFields": ["date", "id", "name"],
  "request": {
    "method": "GET",
    "path": "/v1/things",
    "queryParameters": { "limit": "50" },
    "body": {}
  },
  "recordSelector": { "recordPath": ["data"], "responseFormat": "json" },
  "fields": { "...": { "type": "string" } }
}
FieldMeaning
overviewOptional one-line description, shown in the builder UI.
uniqueKeysField names forming the row’s unique key: later runs update the rows with these values instead of adding duplicates. Required to publish, and every name must be one of the node’s fields.
destinationNameOptional; the destination table name (defaults to the node name).
isTimeSeriesBoolean; enables date-window incremental processing for this node. A node with an incremental strategy other than none is treated as time-series even without it.
defaultFieldsOptional field names pre-selected by default (defaults to all declared fields).
request{ method, path, queryParameters?, headers?, body? }. method is GET or POST. path is relative to baseUrl and must start with /. headers is an object of extra request headers; its values can be templates, like queryParameters.
recordSelector.recordPathArray of keys locating the row(s) — see How the engine turns responses into rows. recordSelector.responseFormat is optional: json (default), csv, or jsonl. A 204 response or an empty JSON body counts as no records.
fieldsObject keyed by field name. See Fields.

A node can also optionally carry pagination, incremental, partitionRouter, transformations, recordFilter, and errorHandler — each covered in its own section below.

An async node (see Asynchronous retriever) replaces request + recordSelector with a retriever: { type: "async", submit, poll, download } block instead. A node-level request on an async node is ignored.

Fields

Section titled “Fields”

fields is an object keyed by the output field name; each value describes how to read it from a raw API record:

{
  "date": { "type": "date" },
  "id": { "type": "integer" },
  "spending": { "type": "number", "dataPath": "stats.spending" },
  "raw_json": { "type": "object", "description": "Full nested payload for debugging." }
}
  • type — one of eight lowercase types: string, integer, number, boolean, date, datetime, object, array.
  • dataPath (preferred) or apiName (legacy alias, same meaning) — a dot-string path into the raw record, e.g. "total_market_cap.usd" reaches a nested object. Where the record has no such nested path, a key spelled with the dots ("total_market_cap.usd" as one key, as flatten with the separator "." writes it) is read instead. When the row itself is an array (for example [[timestamp, price], ...] selected via recordPath), use the positional index as the path: "0", "1". If both are omitted, the field name itself is used as the key — the API’s field is assumed to already be named exactly that.
  • description — optional help text.

Casting special-cases number/integer/boolean/date/datetime. A boolean reads true, 1, yes, y, t and on as true, their opposites as false, and any other text as empty. A date or datetime becomes a date, and one written without a time zone, in any format, is read as UTC. An integer reads any number notation (1e5 is 100000) and drops a fraction; an id beyond 2^53 loses digits as a number, so declare it as string. object, array, and string all fall through to the same default branch, which JSON.stringifys object/array values instead of keeping them structured — so an array/object-typed field is written out as a JSON string, not a nested value.

Remember the dot-string-vs-array distinction from Common naming mistakes: dataPath is "a.b.c", never ["a", "b", "c"].

Pagination

Section titled “Pagination”

Optional, node-level, and only for sync nodes. pagination.type is one of none, offset, page, cursor.

No pagination

Section titled “No pagination”
{ "type": "none" }

Offset pagination

Section titled “Offset pagination”
{ "type": "offset", "offsetParam": "offset", "pageSize": 100 }

Stops once a page returns fewer records than pageSize; each subsequent request adds pageSize to the running offset. The engine does not send the page size itself: request it in queryParameters (e.g. "limit": "100", as the Cốc Cốc Ads example below does). If the API returns its default, smaller page instead, pagination stops after the first page.

Page pagination

Section titled “Page pagination”
{ "type": "page", "pageParam": "page", "startPage": 1 }

Stops once a page returns zero records; increments the page number by 1 each time, starting from startPage (an integer, default 1).

Cursor pagination

Section titled “Cursor pagination”
{
  "type": "cursor",
  "cursorParam": "cursor",
  "cursor": { "from": "body", "path": ["meta", "next_cursor"] }
}

Reads the next cursor value either from the response body (cursor.from: "body", cursor.path as an array) or a response header (cursor.from: "header", cursor.header as the name, with an optional cursor.linkRel to parse a Link: header’s rel="next" URL). Pagination stops once no cursor value is found, or when the response hands back the cursor that was just sent (the last page repeating itself, or a cursor parameter the API ignores).

Where the next-page value goes

Section titled “Where the next-page value goes”

All four types accept an optional inject describing where the next-page value is written on the following request:

"inject": { "into": "query", "name": "offset" }
"inject": { "into": "header", "name": "X-Page-Token" }
"inject": { "into": "body", "path": ["paging", "offset"] }
"inject": { "into": "path" }

inject.into is one of query (default), header, body (needs inject.path, a deep-set array), or path — which replaces the node’s request path/URL outright with the value, used when the API returns a full next-page URL to follow verbatim.

An optional stopCondition halts pagination early, regardless of type, when a response field matches a fixed value:

"stopCondition": { "path": ["meta", "has_more"], "equals": false }

A run reads at most 10000 pages of one request. Pagination still going after that fails the node, because the rest of its records would be missing: check that the pagination stops on the last page. A Test run stops at its own, much lower page limit without an error.

Incremental (date-windowed) extraction

Section titled “Incremental (date-windowed) extraction”

Optional, node-level; drives date-windowed (time-series) extraction. incremental.strategy is one of none, day-by-day, range.

No incremental strategy

Section titled “No incremental strategy”
{ "strategy": "none" }

Day-by-day

Section titled “Day-by-day”
{
  "strategy": "day-by-day",
  "request": { "into": "query", "startName": "date", "format": "YYYY-MM-DD" }
}

The run is split into one request per calendar day. The window’s start and end are the same date, so startName/startPath alone is usually enough; an endName/endPath, if set, gets the same date.

Range

Section titled “Range”
{
  "strategy": "range",
  "request": {
    "into": "query",
    "startName": "start_date",
    "endName": "end_date",
    "format": "YYYY-MM-DD"
  }
}

One request per configured date range; use startName + endName (query) or startPath + endPath (body).

request.into is "query" (adds startName/endName query parameters) or "body" (deep-sets startPath/endPath, arrays, into the request body). request.format uses UPPERCASE date-format tokens: YYYY, MM, DD (time components, if present, are 00 for the start and 23:59:59 for the end, so a one-day window covers the whole day); X/x mean unix epoch seconds/milliseconds, read the same way. Omitted, or YYYY-MM-DD, means “pass the date through unchanged.” Non-token characters pass through literally, so avoid formats containing a token’s letters as ordinary text (for example, don’t put mm inside a literal word).

Inside the node’s own request (or retriever.submit), the current window is also available directly as {{ dateWindow.start }} / {{ dateWindow.end }} (both YYYY-MM-DD strings) — this is how transformations.add stamps a date field onto records the API itself doesn’t return dated.

Partition router: substream and list

Section titled “Partition router: substream and list”

Optional, node-level; fans a single node out into one child request per “slice” value, exposed to the child request as {{ stream_slice.<partitionField> }}. Mutually exclusive with an async retriever. The node’s own request / recordSelector / pagination describe the child (per-slice) request.

Substream

Section titled “Substream”

Fetches a parent list first, extracts a key from each parent record, and runs the child request once per distinct key value:

{
  "type": "substream",
  "parent": {
    "request": { "method": "GET", "path": "/accounts" },
    "recordPath": ["data"],
    "key": "id"
  },
  "partitionField": "account_id"
}

parent.request is a full request spec (optionally with its own parent.pagination, if the parent list itself is paginated). parent.recordPath extracts the parent rows the same way recordSelector.recordPath extracts rows. parent.key is a dot-string path into each parent record naming the value to fan out on. partitionField names the key under stream_slice that the child request can reference, e.g. {{ stream_slice.account_id }}.

List

Section titled “List”

Fans out over a static or parameter-supplied list of values instead of a parent fetch — no parent is allowed:

{ "type": "list", "values": ["US", "EU", "APAC"], "partitionField": "region" }
{ "type": "list", "valuesFromParameter": "RegionList", "partitionField": "region" }

Exactly one of values (a non-empty array of strings) or valuesFromParameter (a parameter holding a comma-separated string) must be present.

Multi-account fan-out

Section titled “Multi-account fan-out”

Optional, top-level (accounts); runs every node once per account ID instead of once per connector run:

{
  "from": "{{ parameters.AdAccountId }}",
  "parse": { "split": "[,;]", "trim": true }
}
  • from — a template string (typically {{ parameters.X }}) resolving to one account id, or several separated by the parse.split pattern.
  • parse.split — optional regex string used to split from into multiple ids (default splits on , or ;).
  • parse.trim — optional boolean (default true); trims whitespace off each id.
  • parse.strip / parse.prefix — optional: strip removes any of the given characters from each id; prefix prepends a fixed string to each id.

Each resolved id becomes {{ account.id }} inside that account’s requests, and node fetching runs once per id. If accounts is omitted, the node runs exactly once with no account.id scope.

One account’s failure does not stop the others: the engine attempts every account, then decides the run’s outcome. The policy is not configurable in the manifest.

  • An account the API turns away with 401 or 403 is skipped. The run logs a warning, and the date window still counts as loaded, so recover that account’s data with a manual backfill. An errorHandler filter with action: "FAIL" for that status fails the run instead.
  • Any other failure is logged as an error, and the window is requested again on the next run. The run then fails, naming the accounts that did not import. An account that fails the same way on three days of a day-by-day window is not asked again for the rest of that run.
  • If no account imported anything, the run fails.
  • If every account is turned away on the same day, the run stops at that day, which usually means the credentials stopped working. The days before it stay loaded.

Asynchronous retriever

Section titled “Asynchronous retriever”

For APIs that generate a report asynchronously: submit a job, poll until it’s ready, then download the result. An async node replaces the node-level request + recordSelector entirely with retriever: { type: "async", submit, poll, download }:

{
  "type": "async",
  "submit": {
    "method": "POST",
    "path": "/reports",
    "body": { "ad_account_id": "{{ account.id }}" },
    "jobIdPath": ["id"]
  },
  "poll": {
    "method": "GET",
    "path": "/reports/{{ job.id }}/status",
    "statusPath": ["status"],
    "readyValue": "READY",
    "failedValue": "FAILED",
    "resultUrlPath": ["location_json"],
    "backoff": { "maxAttempts": 180, "initialMs": 3000, "maxMs": 15000 }
  },
  "download": { "recordPath": ["rows"] }
}
  • submit — a request spec plus jobIdPath (array), locating the newly created job’s id in the submit response.
  • poll — a request spec (its path/templates may reference {{ job.id }}) plus statusPath (array), readyValue, optional failedValue, and resultUrlPath (array) locating a download URL once the job succeeds. poll.backoff bounds the polling loop: maxAttempts (default 180, at most 1000), initialMs (default 3000) and maxMs (default 15000), each at most 300000 — the delay doubles each attempt up to maxMs. The status is compared with readyValue and failedValue as text, so "true" or "2" matches an API that answers true or 2. A response matching failedValue throws immediately; exhausting maxAttempts without reaching readyValue also throws. A poll that carries no status at statusPath (a job still queued) is polled again; when none of them ever carries one, the error names statusPath, since the path is then the likely mistake.
  • jobIdPath, statusPath and resultUrlPath must each name at least one key, and readyValue is required. The parser refuses an async node without them: an empty path reads the whole response, which never equals readyValue, so the poll loop would run out all its attempts first.
  • download.recordPath — array; the downloaded JSON is extracted the same way recordSelector.recordPath extracts rows. The download URL comes from the API, so it may be on any public HTTPS host, not only the manifest’s own.
  • A submit, poll or download request that fails with a 5xx, a 429 or a network error is sent again, as a sync request without an errorHandler is. For the submit, an API that created the job before failing is left with one extra job, which the run never reads. Without the retry the day would fail, and the next run would submit the report again anyway.

This poll.backoff is a completely separate mechanism from a node’s errorHandler.backoff — the former paces the async job-status loop, the latter paces HTTP-error retries. An async node must not carry an errorHandler at all: the engine wires that for sync retrievers only, so the parser refuses the pairing at publish.

Transformations

Section titled “Transformations”

Optional, node-level array, applied in order, after recordFilter and before field projection/casting:

[
  { "type": "add", "field": "date", "value": "{{ dateWindow.start }}" },
  { "type": "remove", "field": "internal_debug_id" },
  { "type": "keysToLower" },
  { "type": "flatten", "separator": "_" }
]
  • add — { field, value }; sets a top-level field to a templated value. Templating here is uniquely lenient — an unresolved path renders as an empty string instead of throwing — and, uniquely, also exposes the record itself: {{ parameters.X }}, {{ dateWindow.start }} / {{ dateWindow.end }}, {{ account.id }}, and {{ record.<field> }} (a field already present on this same record) are all available.
  • remove — { field }; deletes a top-level field.
  • keysToLower — no options; lowercases every top-level key (last one wins on a collision).
  • flatten — { separator? } (default "_"); recursively flattens nested objects into separator-joined top-level keys (e.g. {"stats":{"clicks":5}} → {"stats_clicks":5}); arrays are left intact, not flattened. Fields read the new keys, not the old paths: with "_", stats.clicks becomes the key stats_clicks; with ".", a field’s dataPath "stats.clicks" reads the new key as before.

Record filter

Section titled “Record filter”

Optional, node-level; keeps or drops each raw record before transformations run. path is an array (see Common naming mistakes) evaluated against the raw record. Only one recordFilter per node — it’s a single object, not an array.

{ "path": ["status"], "operator": "equals", "value": "ACTIVE" }
{ "path": ["deleted_at"], "operator": "isNull" }
{ "path": ["region"], "operator": "inList", "value": "US,EU,APAC" }
OperatorBehavior
equals / notEqualsString-compares the value at path (coerced with String(...)) against value.
containsSubstring match: String(value_at_path).includes(value).
isNull / isNotNullNo value needed; true when the path resolves to null/undefined (or not, respectively).
inListTrue when the value at path is one of a resolved list, supplied either as a literal comma-string value or a parameter name via valuesFromParameter (also comma-split). Exactly one of the two must be present.

A record where path is missing never matches equals, contains or inList, and always matches notEquals — so notEquals keeps records that lack the field.

Error handling

Section titled “Error handling”

Optional, node-level, and applies to sync nodes only — an errorHandler on an async node is refused at publish, because an async node paces itself through poll.backoff instead. errorHandler matches an HTTP error response against an ordered list of filters; the first matching filter decides what happens:

{
  "responseFilters": [
    {
      "httpCodes": [429],
      "action": "RETRY",
      "backoff": { "type": "waitTimeFromHeader", "header": "Retry-After" }
    },
    { "httpCodes": [404], "action": "IGNORE" },
    {
      "bodyMatch": { "path": ["error", "code"], "equals": "ACCOUNT_DISABLED" },
      "action": "FAIL"
    }
  ],
  "backoff": { "type": "exponential", "factor": 2, "baseMs": 1000 }
}

Each filter needs at least one of httpCodes (an array of HTTP status codes, 100–599), messageContains (string, matched against the raw error message/body text), or bodyMatch ({ path: [...], equals?: <string>, contains?: <string> }, matched against the parsed JSON body) — plus a required action:

ActionEffect
RETRYRetry the request, delayed by this filter’s backoff (or the handler’s top-level backoff if the filter has none).
IGNORETreat the failed request as if it returned zero records and stop paginating there: records on this and any later page are not imported. The run does not fail, and its log names the skipped request.
FAILDo not retry; the error fails the run.

An error that matches no filter gets the default treatment: a 5xx or 429 response is retried, any other error fails the run at once.

backoff.type is one of:

TypeFields
constantRequires a numeric delayMs.
exponentialOptional factor (default 2) and baseMs (default is the retry loop’s own initial delay); delay = baseMs * factor^attempt.
waitTimeFromHeaderOptional header (default "Retry-After"); reads either a number of seconds or an HTTP date from that header.
waitUntilTimeFromHeaderRequires header (an epoch-seconds value); optional regex to extract the number out of a larger header string, optional minMs floor on the computed delay.

A backoff may also be set directly on errorHandler (no responseFilters match required) as the node’s default retry pacing.

Every retry delay is capped at 5 minutes: a constant or exponential one, and one read from a response header (waitTimeFromHeader, waitUntilTimeFromHeader), including after the minMs floor. A run has no deadline of its own, so an uncapped delay would hold it for as long as it said.

Rate limiting

Section titled “Rate limiting”

Optional, top-level (rateLimit); a simple global cap shared by every request the connector makes:

{ "requests": 60, "perSeconds": 60 }

requests is a positive integer; perSeconds is a positive number of at most 3600. For a daily quota, retry the API’s 429 with an errorHandler instead. The example above means “no more than 60 requests per 60 seconds.”

Templating scopes

Section titled “Templating scopes”

Every string field that accepts a template uses {{ scope.path }} syntax (double curly braces; conventionally one space on each side).

ScopeAvailable
{{ parameters.X }}Any declared parameter’s resolved value.
{{ account.id }}The current account id, when the connector declares accounts. Without accounts there is no account id.
{{ auth.token }}The token issued by tokenExchange/oauth2 — only meaningful inside that same authenticator’s inject.format.
{{ dateWindow.start }} / {{ dateWindow.end }}The current incremental run’s date window (YYYY-MM-DD strings).
{{ node.selectedFields }}The names of the fields the Data Mart selected for this node, comma-separated, e.g. date,rate — for an API that takes the list of fields to return, such as a fields query parameter. These are the manifest’s field names, not their dataPaths.
{{ stream_slice.<partitionField> }}The current partition value inside a partition-router child request.
{{ job.id }}The async job id, inside retriever.async.poll.
{{ record.<field> }}A field of the current record, only inside transformations.add.value.

What happens to an unresolved path depends on where it is:

  • In the request path, the request body and the authentication fields, it throws and fails the run.
  • In queryParameters and headers, the parameter is not sent at all, the same as when its value is empty. An optional parameter the user left blank simply drops out of the request.
  • Inside transformations.add.value, it renders as an empty string.

Authoring workflow

Section titled “Authoring workflow”
  1. Read this reference before writing or editing a manifest. An AI assistant gets the same grammar from the authoring guide for AI assistants.
  2. Research the target API and author the manifest: pick the authentication type, define one node per data stream you need, and add pagination, incremental extraction, filters, transformations, or error handling only where the API actually needs them.
  3. Open the connector builder and bring the manifest in: ⋮ → Import JSON… for a .json file, or paste it into Code mode.
  4. Run Test on one node. Enter an API key or a token only into a parameter marked SECRET or used by the authentication: the builder uses those values for the test and never saves them, while other test values are saved in the browser.
  5. If the test fails, read the returned error and make the smallest change that fixes it — correct a typo in an existing baseUrl/path/queryParameters/field name rather than rewriting working parts, renaming nodes, or switching to a different API — then test again.
  6. Once it passes, Publish the connector.

Real credentials are connected separately, afterward: the person setting up the connector enters their API key/token on the Data Mart page, or in the connector’s Test. Credentials are never typed into a manifest or into an AI assistant.

Worked examples

Section titled “Worked examples”

1. Simple GET, no auth

Section titled “1. Simple GET, no auth”

A minimal connector with no authentication block at all, one node, and an empty recordPath because the response body itself is the record:

{
  "version": "1.0",
  "name": "RatesDeclarative",
  "title": "Frankfurter FX (Declarative)",
  "baseUrl": "https://api.frankfurter.dev",
  "parameters": {
    "Base": { "requiredType": "string", "isRequired": true, "default": "EUR", "label": "Base Currency" }
  },
  "nodes": {
    "latest": {
      "overview": "Latest exchange rates snapshot",
      "destinationName": "frankfurter_latest",
      "isTimeSeries": false,
      "uniqueKeys": ["date", "base"],
      "defaultFields": ["date", "base"],
      "request": { "method": "GET", "path": "/v1/latest", "queryParameters": { "base": "{{ parameters.Base }}" } },
      "recordSelector": { "recordPath": [] },
      "fields": {
        "date": { "apiName": "date", "type": "date" },
        "base": { "apiName": "base", "type": "string" }
      }
    }
  }
}

2. tokenExchange auth + async retriever + accounts (Moloco)

Section titled “2. tokenExchange auth + async retriever + accounts (Moloco)”

Combines a token exchange, a range incremental strategy writing into the request body, an async submit/poll/download retriever, and account fan-out:

{
  "version": "1.0",
  "name": "MolocoCloud",
  "baseUrl": "https://api.moloco.cloud",
  "parameters": {
    "ApiKey": { "requiredType": "string", "isRequired": true },
    "AdAccountId": { "requiredType": "string", "isRequired": true }
  },
  "authentication": {
    "type": "tokenExchange",
    "exchange": {
      "method": "POST",
      "url": "https://api.moloco.cloud/cm/v1/auth/tokens",
      "body": { "api_key": "{{ parameters.ApiKey }}" },
      "tokenPath": ["token"],
      "ttlSeconds": 57600
    },
    "inject": { "into": "header", "name": "Authorization", "format": "Bearer {{ auth.token }}" }
  },
  "nodes": {
    "performance_report": {
      "destinationName": "moloco_performance_report",
      "isTimeSeries": true,
      "uniqueKeys": ["date", "campaign"],
      "fields": {
        "date": { "apiName": "date", "type": "date" },
        "campaign": { "apiName": "campaign", "type": "string" },
        "spend": { "apiName": "metric.spend", "type": "number" }
      },
      "incremental": {
        "strategy": "range",
        "request": {
          "into": "body",
          "startPath": ["date_range", "start"],
          "endPath": ["date_range", "end"],
          "format": "YYYY-MM-DD"
        }
      },
      "retriever": {
        "type": "async",
        "submit": {
          "method": "POST",
          "path": "/cm/v1/reports",
          "body": { "ad_account_id": "{{ account.id }}", "dimensions": ["DATE", "CAMPAIGN"] },
          "jobIdPath": ["id"]
        },
        "poll": {
          "method": "GET",
          "path": "/cm/v1/reports/{{ job.id }}/status",
          "statusPath": ["status"],
          "readyValue": "READY",
          "failedValue": "FAILED",
          "resultUrlPath": ["location_json"],
          "backoff": { "maxAttempts": 180, "initialMs": 3000, "maxMs": 15000 }
        },
        "download": { "recordPath": ["rows"] }
      }
    }
  },
  "accounts": { "from": "{{ parameters.AdAccountId }}", "parse": { "split": "[,;]", "trim": true } }
}

3. oauth2 refresh_token grant (minimal, illustrative)

Section titled “3. oauth2 refresh_token grant (minimal, illustrative)”

A minimal manifest built directly from the oauth2 shape above — swap baseUrl/tokenUrl/paths for the real target API’s:

{
  "version": "1.0",
  "name": "ExampleOAuth2Api",
  "title": "Example OAuth2 API",
  "baseUrl": "https://api.example.com",
  "parameters": {
    "ClientId": { "requiredType": "string", "isRequired": true, "label": "Client ID", "attributes": ["SECRET"] },
    "ClientSecret": { "requiredType": "string", "isRequired": true, "label": "Client Secret", "attributes": ["SECRET"] },
    "RefreshToken": { "requiredType": "string", "isRequired": true, "label": "Refresh Token", "attributes": ["SECRET"] }
  },
  "authentication": {
    "type": "oauth2",
    "tokenUrl": "https://oauth2.googleapis.com/token",
    "grantType": "refresh_token",
    "clientId": "{{ parameters.ClientId }}",
    "clientSecret": "{{ parameters.ClientSecret }}",
    "refreshToken": "{{ parameters.RefreshToken }}",
    "scope": "https://www.googleapis.com/auth/spreadsheets.readonly",
    "ttlSeconds": 300,
    "inject": { "into": "header", "name": "Authorization", "format": "Bearer {{ auth.token }}" }
  },
  "nodes": {
    "items": {
      "overview": "Items visible to the authenticated account.",
      "uniqueKeys": ["id"],
      "request": { "method": "GET", "path": "/v1/items", "queryParameters": { "pageSize": "100" } },
      "recordSelector": { "recordPath": ["items"] },
      "pagination": { "type": "cursor", "cursorParam": "pageToken", "cursor": { "from": "body", "path": ["nextPageToken"] } },
      "fields": {
        "id": { "type": "string" },
        "name": { "type": "string" },
        "updatedAt": { "type": "datetime", "dataPath": "updated_at" }
      }
    }
  }
}

4. apiKey header + day-by-day incremental + offset pagination + nested dataPath + transformations.add (Cốc Cốc Ads)

Section titled “4. apiKey header + day-by-day incremental + offset pagination + nested dataPath + transformations.add (Cốc Cốc Ads)”

A larger, realistic manifest: a header-injected API key, three sibling nodes, day-by-day incremental extraction, offset pagination, metrics nested under stats.*, and a transformations.add step that stamps the window’s date onto every record:

{
  "title": "Cốc Cốc Ads (Declarative, v2)",
  "name": "CocCocAds",
  "version": "1.0",
  "docUrl": "https://api.qc.coccoc.com/docs/v2/",
  "baseUrl": "https://api.qc.coccoc.com",


  "authentication": {
    "type": "apiKey",
    "inject": { "into": "header", "name": "Api-Key", "format": "{{ parameters.ApiKey }}" }
  },


  "rateLimit": { "requests": 500, "perSeconds": 3600 },


  "parameters": {
    "ApiKey": { "requiredType": "string", "isRequired": true, "label": "API Key", "attributes": ["SECRET"] },
    "CampaignIds": { "requiredType": "string", "isRequired": false, "label": "Campaign IDs filter (comma-separated), e.g. 123,456" }
  },


  "nodes": {


    "account": {
      "overview": "Account snapshot: balance and budget limit (VND).",
      "uniqueKeys": ["id"],
      "request": { "method": "GET", "path": "/v2/users/info" },
      "recordSelector": { "recordPath": ["data"] },
      "fields": {
        "id": { "type": "integer" },
        "balance": { "type": "number" },
        "budget_limit": { "type": "number" }
      }
    },


    "campaigns": {
      "overview": "Daily campaign performance + cost. cost = stats.spending (VND).",
      "uniqueKeys": ["date", "id"],
      "incremental": { "strategy": "day-by-day", "request": { "into": "query", "startName": "start", "endName": "end", "format": "YYYY-MM-DD" } },
      "request": { "method": "GET", "path": "/v2/campaigns", "queryParameters": { "limit": "50" } },
      "recordSelector": { "recordPath": ["data"] },
      "pagination": { "type": "offset", "pageSize": 50, "inject": { "into": "query", "name": "offset" } },
      "transformations": [ { "type": "add", "field": "date", "value": "{{ dateWindow.start }}" } ],
      "fields": {
        "date": { "type": "date" },
        "id": { "type": "integer" },
        "name": { "type": "string" },
        "status": { "type": "string" },
        "shows": { "type": "integer", "dataPath": "stats.shows" },
        "clicks": { "type": "integer", "dataPath": "stats.clicks" },
        "spending": { "type": "number", "dataPath": "stats.spending" },
        "ctr": { "type": "number", "dataPath": "stats.ctr" },
        "cpc": { "type": "number", "dataPath": "stats.cpc" },
        "cpm": { "type": "number", "dataPath": "stats.cpm" }
      }
    },


    "ads": {
      "overview": "Daily ad performance + cost.",
      "uniqueKeys": ["date", "id"],
      "incremental": { "strategy": "day-by-day", "request": { "into": "query", "startName": "start", "endName": "end", "format": "YYYY-MM-DD" } },
      "request": { "method": "GET", "path": "/v2/ads", "queryParameters": { "limit": "50", "campaign_id": "{{ parameters.CampaignIds }}" } },
      "recordSelector": { "recordPath": ["data"] },
      "pagination": { "type": "offset", "pageSize": 50, "inject": { "into": "query", "name": "offset" } },
      "transformations": [ { "type": "add", "field": "date", "value": "{{ dateWindow.start }}" } ],
      "fields": {
        "date": { "type": "date" },
        "id": { "type": "integer" },
        "status": { "type": "string" },
        "campaign_id": { "type": "integer" },
        "title": { "type": "string" },
        "shows": { "type": "integer", "dataPath": "stats.shows" },
        "clicks": { "type": "integer", "dataPath": "stats.clicks" },
        "spending": { "type": "number", "dataPath": "stats.spending" }
      }
    }
  }
}

(Shortened for readability: this shows 3 of the connector’s 4 nodes — the ad_groups node follows the same pattern — and a subset of each node’s fields. For the complete, byte-for-byte manifest, see the authoring guide for AI assistants.)

5. partitionRouter: substream, expanded into a full node

Section titled “5. partitionRouter: substream, expanded into a full node”

Expands the substream shape into one complete node: fetch the list of ad accounts first, then fetch each account’s campaigns, with the parent id available to the child request as {{ stream_slice.account_id }}:

{
  "version": "1.0",
  "name": "ExampleSubstreamApi",
  "baseUrl": "https://api.example.com",
  "parameters": {
    "ApiKey": { "requiredType": "string", "isRequired": true, "label": "API Key", "attributes": ["SECRET"] }
  },
  "authentication": {
    "type": "apiKey",
    "inject": { "into": "header", "name": "Authorization", "format": "Bearer {{ parameters.ApiKey }}" }
  },
  "nodes": {
    "campaigns_by_account": {
      "overview": "Campaigns for every ad account visible to this API key.",
      "uniqueKeys": ["account_id", "id"],
      "partitionRouter": {
        "type": "substream",
        "parent": {
          "request": { "method": "GET", "path": "/accounts" },
          "recordPath": ["data"],
          "key": "id"
        },
        "partitionField": "account_id"
      },
      "request": {
        "method": "GET",
        "path": "/accounts/{{ stream_slice.account_id }}/campaigns",
        "queryParameters": { "limit": "100" }
      },
      "recordSelector": { "recordPath": ["data"] },
      "pagination": { "type": "offset", "offsetParam": "offset", "pageSize": 100 },
      "fields": {
        "id": { "type": "string" },
        "account_id": { "type": "string", "dataPath": "account.id" },
        "name": { "type": "string" },
        "status": { "type": "string" }
      }
    }
  }
}