Usage Processing Block Architecture
The usage processing framework ingests usage files, applies mediation rules, and rates and discounts usage records across a set of coordinated microservices.
Usage File Sources
Usage files can originate from network elements, devices, homegrown applications, and Allegro UI (for operational exceptions).
Usage File Setup
This mechanism handles file transfer via SFTP, S3 link/copy, and similar methods.
Aria Allegro S3 Folder Structure
The Aria Allegro S3 folder structure is created automatically based on source platform configurations. When you add a Usage Source Platform configuration for one or more sources, Aria Allegro creates the client folder structure automatically.
Note: Do not use spaces in source platform configuration names. These names determine the S3 folder structure.
Aria Allegro Microservices
Aria Allegro microservices enable independent, on-demand scaling. You can configure resources per pod and set minimum and maximum pod counts for auto-scaling. Each pod uses multiple threads to process batches of usage records. The following microservices support usage processing.
- Apache NiFi—Serves as a gateway that picks up files from Aria Allegro S3 input folders.
- Mediation Service—Applies configured mediation rules to incoming usage records.
- Usage Processing Service—Handles rating and discounting of usage records.
- Cache Service—Caches relevant data in memory and synchronizes bi-directionally with the Allegro database.
- Allegro UI—Supports manual file upload and provides a view of all usage-related Allegro data.
- Allegro Core—Serves as the interface service for all UI interactions and GraphQL API calls from outside Aria Allegro.
Additional Components
- Allegro Database—Uses PostgreSQL Aurora RDS with partitioned client data.
- Mediation Kafka Topic—Apache NiFi publishes usage record batches here for the Mediation Service to consume and apply rules.
- Usage Processing Kafka Topic—The Mediation Service publishes usage records with applied rules here for rating and discounting by the Usage Processing Service.
High-Level Usage Processing Flow
The following describes how a usage file moves through the system from ingestion to monetization.
Phase 1: File Ingestion
Usage file setup or Allegro UI transfers files from various sources to the Allegro S3 folder structure based on your Source Platform configurations. The scheduler uses those configurations to manage periodic file push and pull.
Phase 2: Apache NiFi Processing
Apache NiFi actively monitors input folders for each usage source and performs the following.
- Moves the file to the processing (transient) folder.
- Splits the file into batches per configuration.
- Automatically loads batches into the database per the record structure and mapping configuration.
- If loading succeeds, moves the file to the processed folder. If errors occur, moves it to the error folder. Errors occur when a file or its records do not conform to the mapping configuration due to:
- Field values that do not match the defined data type.
- Field length that is too short (if applicable).
- Duplicate values in attributes marked for unique constraint management.
- Missing values in mandatory attributes.
- Creates a record to manage file statistics and inserts one row each for the Parse and Load, Mediation, and Rating phases.
- Publishes each usage batch to the Mediation Kafka Topic.
Phase 3: Mediation Service
The Mediation Service picks up usage record batches and performs the following.
- Applies mediation rules (optional), including:
- Filter Rules—Exclude records that are not to be monetized.
- Enrichment Rules—Enhance usage record attributes.
- Splitting Rules (roadmap)—Split a usage record into multiple records for monetization.
- Aggregation Rules (roadmap)—Combine multiple records into one for monetization.
- Updates the status of raw usage records in the Allegro database in batches.
- Maps and converts raw usage data into the Allegro usage container model using the mapping configuration.
- Publishes each usage record to the Usage Processing Kafka Topic.
Phase 4: Usage Processing (Rating and Discounting)
The Usage Processing Service uses the usage processing stream configuration to execute sub-streams and modules in the configured sequence. Errors cause the usage record to suspend for Suspense re-processing. Successful monetization produces the following results.
- Currency impacts, allowance consumptions, currency offsets for allowances, and accumulator impacts are added to the usage container.
- A monetized usage record (transaction unit) is created detailing all currency, allowance, and accumulator resources.
- Balance Unit (balance group) updates are applied to the corresponding currency, allowance, and accumulator buckets.
Allegro UI lets you navigate usage files, usage statistics, raw usage data, suspended usage, monetized usage records, and more.