Data Feed FAQ

1. Do you turn on individual modules for the data feed or do you get all objects?

For the load stream, when you put in a customer support ticket, you can pick which objects you want. For the change stream, you currently get all the objects. An object subscription UI is in the roadmap for later.

2. How is last id used for the data feed connection?

If there is a client side interruption for any reason, such as an internet failure or a client processing failure, the client script should use the last_id to reconnect as shown in the example code. More info on last event id here https://www.w3.org/TR/eventsource/.

3. How should the data be consumed and loaded into a database once received?

Clients should create the data model in their database based on our Logical Object Model. Once the data model is ready, the client should write a script to consume the data (sample perl script) and populate the database based on which Entity they are reading from the feed.

4. How secure is the Feed connection? It looks like an http connection and not an https connection.

The feed service URLs always uses HTTPS protocol. Data is encrypted over SSL in the normal way.

5. Can you provide examples of the full JSON message? I see payload examples but no header detail.

Here’s a sample full JSON payload. The first ‘data’ is the header and second ‘data’ is the payload.

► JSON Payload

data:{"ref":{"type":"Account","ids":{"account_no":21119957,"userid":"43739988"}},"tstamp":"2020-06-08T19:53:30+02:00","tick":29580691950}
data:{"account_no":21119957,"userid":"43739988","client_account_id":null,"balance":0,"status":{"type":"Account_Status","ids":{"status_no":1,"enum":1}},"status_tstamp":"2121-12-31T19:53:32+01:00","parent_account":null,"created_tstamp":"2121-12-31T19:53:32+01:00","updated_tstamp":"2121-12-31T19:53:32+01:00","secret_question":null,"secret_question_answer":null,"is_test_account":false,"is_invoice_approval_required":false,"currency":{"type":"Currency","ids":{"currency_cd":"usd","code":"usd"}},"updated_receipt_id":null,"created_receipt_id":null,"sequence_functional_group":null,"start_date":"2121-12-31","admin_contact":{"type":"Contact","ids":{"contact_no":23436896}},"is_consumer_account":false,"locale":{"type":"Locale","ids":{"locale_no":10001}},"legal_entity":null,"is_anonymized":null,"notify_method":{"type":"Notify_Method","ids":{"method_no":2,"enum":2}},"functional_account_groups":[{"type":"Functional_Account_Group","ids":{"group_no":10095894,"client_group_id":"FA_-_DE"}}],"collection_account_groups":[],"account_fields":[{"account_field":{"type":"Account_Field","ids":{"field_no":22776,"field_name":"supp_field_1587439747321"}},"field_values":["Account_Field_value_1"]}],"tax_exemption_level":null,"notification_overrides":[],"notification_template_group":null,"surcharges":[],"revrec_profile_number":null,"purchase_order_number":null,"cc_email_addresses":[],"bcc_email_addresses":[]}
event:create
id:H4sIAAAAAAAAAKtWMjRUsjI0MTA0Ma8FAG50BJUNAAAA

6. Is tick the event id?

The timestamp and tick in the SSE event headers identify when the data change happened. The tick is simply a tie breaker if two events come in at the exact same timestamp. The smaller tick represents the earlier event.

7. How do you handle when the feed fails or times out? How do you handle monitoring?

The feed service simply drops the connection and waits for the next request. For client failures, the client script should used the last event id to reconnect.

8. How are supplemental fields handled in the data model?

Supplemental fields are included in the feed. Below are two samples.

► Account Fields

{
  "account_no": 21119729,
  "userid": "uid_1591636971086",
  "client_account_id": "caccid_1591636971086",
  "account_fields": [
    {
      "account_field": {
        "type": "Account_Field",
        "ids": {
          "field_no": 22782,
          "field_name": "supp_field_1587439754010"
        }
      },
      "field_values": [
        "1"
      ]
    },
    {
      "account_field": {
        "type": "Account_Field",
        "ids": {
          "field_no": 22784,
          "field_name": "supp_field_1587439756560"
        }
      },
      "field_values": [
        "1"
      ]
    }
  ]
}

 

► Transaction Supplemental Fields (known as Optional Transaction Qualifiers)

{
  "transaction_no": 159241559,
  "transaction_type": {
    "type": "Transaction_Type",
    "ids": {
      "type_no": 9,
      "enum": 9
    }
  },
  "created_tstamp": "2114-06-24",
  "source_no": 381465,
  "account": {
    "type": "Account",
    "ids": {
      "account_no": 21112250,
      "userid": "uid_1591377970950",
      "client_account_id": "caid_1591377970950"
    }
  },
  "currency": {
    "type": "Currency",
    "ids": {
      "currency_cd": "usd",
      "code": "usd"
    }
  },
  "amount": 10,
  "applied_amount": 0,
  "fully_applied_tstamp": null,
  "statement": null,
  "master_plan_instance": {
    "type": "Plan_Instance",
    "ids": {
      "plan_instance_no": 25211656,
      "client_plan_instance_id": "25211656"
    }
  },
  "api_receipt_id": null,
  "transaction_supp_fields": [
    {
      "field_name": "aapm_Qualifier_name_1591377973588",
      "field_value": "aapm_Qualifier_value_1591377973588"
    }
  ],
  "voided_transaction": null,
  "voiding_transaction": null,
  "applications": [],
  "comments": "Balance forward recorded during account creation for plan instance via API"
}

9. How do you know the order of the events? Will the event order be guaranteed?

Event order is guaranteed only within an account. Please refer to our best practices if ordering matters to you.

10. What is the retry window on a failure? Is there something that will resend a missed event?

When there is a client failure, the client application should use the last_event_id to start refeeding. The client app can reconnect and refeed from a given event_id as often as it wants up to the 48 hour retention period.

11. What is the unique ID or an object? Is it the object ID and tick?

The unique ID of an object is its ‘ref’. Every Data Feed SSE event has a header line in its data field. That header line contains the ‘ref’ of the object impacted by that event, right next to the timestamp and tick. SSE event_ids (last event ids) are solely for reconnecting to the SSE server is case of client or connection failure or if you want to get the data again. It plays no part at all in identifying data objects.

12. What are some possible values for the ‘from’ query parameter?

The ‘from’ query parameter takes values like 1day, 2hour, 20minute etc.