Usage Data Ingestion

Usage Data Ingestion Using Files

Usage file management is usually automated. Manual loading may be necessary for:

• Manual loading during testing or demos.
• Loading files for exception scenarios in production.

Clients may need setup to automate usage file ingestion, including access to required folders.

Aria Allegro supports both push and pull methods for files, requiring:

• Clients push files to Allegro S3 folders or SFTP sites for multiple usage source platforms, requiring access control and scripts to manage files.
• Aria Allegro pulls files from usage source platforms via S3 or SFTP folders, requiring access control and management scripts.

Aria Allegro supports (as covered in Allegro Mediation setup):

• Delimited files
• Position-based files
• XML files require a pre-processor to convert XML to CSV. This setup is quick for clients.

Aria Allegro supports files structured with any combination of record types (Header, Detail, sub-records, Trailer).

Scripts and automation place files in AWS S3 (or similar cloud setups). Apache NIFI monitors input folders per source platform and processes files upon arrival.

S3 Folder Structure With Aria Allegro

The S3 folder structure for Aria Allegro and each client source platform is:

• [Home Directory]/Client/mediation/Source Platform/Service Type/Usage Record Type/input
• For example, the home directory for a development environment is “aria-allegro-dev.”
• Other sub-folders at the same level as input include:
  • processing—files move from input to processing as Apache NIFI picks them up
  • completed—successfully processed files move to the processed folder
  • error—files with errors move to the error folder

Usage Flow for File Processing

The high-level flow for usage file processing using Aria Allegro is shown in the following diagram:

<diagram>

This includes the following steps:

1. Apache NIFI service monitors the input S3 folder for files and performs these activities:
   1. Create an entry to manage file stages and stats, updating statistics in real time as usage records process.
   2. Load raw usage data into the table set up via the Allegro Mediation → Mapping exercise.
      1. Audit raw data in its original format to record what was sent and when.
      2. File mapping adds fields to track source platform, filename, service type, usage record type, created date, and status for each record.
      3. If loading fails, move the usage file to an error folder in S3.
   3. Create and process batches of usage file records.
   4. As batches process, push raw usage records to a Kafka Topic for Mediation.
2. Mediation service subscribes to the Kafka topic from the Apache NIFI process. Upon receiving usage records, mediation service:
   1. Converts raw usage data into the harmonized Allegro Usage container model using mapping configuration.
   2. Evaluates and executes rules to update usage container data for:
      1. Filter rules—to exclude usage records not requiring monetization.
      2. Enrichment rules—to normalize data and add fields. The Allegro usage container includes standard attributes plus 25 flexible fields. This includes formalizing date formats.
      3. Splitting rules—(deferred, to be managed in client setup) to split one usage record into multiple.
      4. Aggregation rules—(deferred, to be managed in client setup) to combine multiple records into one.
      5. Note that splitting and aggregation rules change usage record counts for subsequent phases.
   3. After rules engine processing, mediation service pushes the usage container to a Kafka Topic for Usage Processing (Rating, Discounting, etc.).
3. Usage Processing service subscribes to the Kafka topic from the Mediation service and performs:
   1. Uses Usage Processing Setup configuration to execute streams and modules as configured.
   2. If processing succeeds,
      1. Create a monetized usage record.
      2. Update balances for currency, accumulator, and allowance impacts.
      3. Optionally update the bill-unit for a run-time view, linking each usage record to a bill-unit for billing cycle tracking.
   3. If processing errors occur,
      1. Create or update Usage suspense for the file with the error code. Users can reprocess failed usage records.
   4. Usage file statistics update to reflect the true processing state.
   5. Separate from usage processing (for SLA reasons), another process aggregates and sends monetized usage to core Aria for invoicing and statement generation.

Usage Data Audit and Reconciliation

Aria Allegro provides complete, bi-directional audit and reconciliation. This includes:

• Traceability of the usage record from the raw usage data to invoice line and
• Traceability of the invoice line back to the set of raw usage records.

This is managed as follows:

• A raw usage record is part of a file or (in future) a batch (API, queue, database query).
• Each raw usage record has a record ID, called usage Id in Aria Allegro, available in the raw usage object. This includes,
  • a usage id provided by the source platform, identified in the mapping configuration with the unique constraint attribute OR
  • a usage id created by Aria Allegro when multiple attributes have unique constraint attributes as a composite key. Aria Allegro stores the concatenated value as the usage Id.
• Mediation rules (in the future) can aggregate usage records into a single usage record. The aggregated usage id is created and stored as reference.
• Mediation rules (in the future) can split a raw usage record into multiple usage records. The split records store the original usage id attribute and create a usage id for each split record by appending a numeric index to the original usage id.
• Aria Allegro creates a monetized usage record storing the usage id, original usage id, and aggregated usage id.
• If the monetized usage record gets suspended, then
  • Reprocessing of the suspended record will result in a monetized usage record being created.
• Synchronization of monetized usage records with core Aria for invoicing, statement generation, and more includes,
  • Monetized usage records are not aggregated before sending to core Aria. The usage id of the record is passed as-is OR
  • Monetized usage records are aggregated before sending to core Aria. Aria Allegro stores the aggregated id for billing and passes it as the usage record id.
• The File (and future Batch) statistics provide count-level traceability across three usage processing phases: a) Parse and Load, b) Mediation, and c) Rating.

This process and related attributes provide forward and backward audit and reconciliation between invoice lines and raw usage records.