Usage Processing Block Architecture

The following diagram shows a high-level block architecture for Aria Allegro usage processing framework.

Summary of the block diagram is as follows:

1. Usage File sources—Usage files can be generated and sent from network elements, devices, homegrown applications, and Allegro UI (for operational exceptions).
2. Usage File setup—This mechanism handles file transfer via SFTP, S3 link/copy, and more.
3. Aria Allegro S3—The Aria Allegro S3 folder structures are created automatically using source platform configurations. User access control must be set by DevOps and deployment teams. The client folder structure is created automatically when the Usage Source Platform configuration is added for one or more sources. The folder structure is as follows:
   
   It is advisable to not use spaces in the respective names
4. Aria Allegro Microservices—These micro-services enable independent, on-demand scaling by configuring resources per pod and setting minimum and maximum pod counts for auto scaling. Each pod uses multiple threads to process batches of usage records. Below is a summary of Allegro Micro-services for usage processing:
   1. Apache NIFI—Serves as a gateway to pick files from Aria Allegro S3 input folders.
   2. Mediation service—Applies configured mediation rules.
   3. Usage Processing Service—Handles rating and discounting of usage records.
   4. Cache Service—Caches relevant data in memory and syncs bi-directionally with the Allegro database.
   5. Allegro UI—For manual file upload and tracking all the usage related Allegro data
   6. Allegro core—(not shown in the diagram)—It is an interface service for all UI interaction or graph*QL API calls from outside of Aria Allegro.
5. Additional components
   1. Allegro Database—Uses PostgreSQL Aurora RDS with partitioned client data.
   2. Mediation Kafka Topic—Apache NIFI publishes usage record batches for the mediation service to consume and apply rules.
   3. Usage Processing Kafka Topic—Mediation service publishes usage records with applied rules for rating and discounting by the usage processing service.
6. High level flow
   1. File ingestion—Usage File setup or Allegro UI transfers files from various sources to the Allegro S3 folder structure based on the source platform configurations. The scheduler uses the Source Platform configuration to manage periodic file push/pull.
   2. Apache Nifi actively monitors input folders for each usage source and performs these activities:
      1. Moves the file to the processing (transient) folder.
      2. Splits the file into batches per configuration.
      3. Automatically loads batches into the database per the record structure and mapping configuration.
      4. If loading succeeds, moves the file to the processed folder; if errors occur, moves it to the error folder. Errors arise when the file or some records don't conform to the mapping configuration, due to:
         1. Field values not matching the datatype for each attribute, 
         2. Field length too short (if applicable), 
         3. Attributes marked for unique constraint management, and
         4. Mandatory attributes missing required values.
      5. Creates a record to manage file statistics and inserts one row for each Parse and Load, Mediation and Rating phase.
      6. Publishes each usage batch to the Mediation Kafka Topic.
   3. Mediation service picks up usage record batches and
      1. applies Mediation rules (optional), including:
         1. Filter Rules—exclude records not to be monetized
         2. Enrichment Rules—enhance usage record attributes
         3. Splitting Rules (roadmap)—split a usage record into multiple records for monetization.
         4. Aggregation Rules (roadmap)—combine multiple records into one for monetization.
      2. Updates the status of raw usage records in the Allegro database in batches.
      3. Maps and converts raw usage data into the Allegro usage container model using the mapping configuration.
      4. Publishes each usage record to a Usage Processing Kafka Topic.
   4. Usage Processing or the Rating/discounting engine service,
      1. uses the Usage Processing stream configuration to execute sub-streams and modules in the configured sequence.
      2. Errors cause the usage record to suspend for Suspense re-processing.
      3. Successful monetization results in:
         1. Currency impacts, allowance consumptions, currency offsets for allowances, and accumulator impacts are added to the usage container.
         2. A monetized usage record (transaction unit) is created detailing all currency, allowance, and accumulator resources.
         3. Balance Unit (balance group) updates for the corresponding currency, allowance buckets, and accumulator buckets.
   5. Allegro UI navigates usage files, usage statistics, raw usage data, suspended usage, monetized usage records, and more.