Data Limits Rate Limiter

Data Limits Rate Limiter

Data Limits rate limiters control the total amount of data (in bytes) that can be transferred during a subscription. They provide a straightforward way to monetize your APIs based on data volume rather than just request count, making them ideal for data-intensive services.

Types of Data Limits

ApiCharge provides two distinct data limit rate limiters:

Type	Description	Best For
DataLimitUpload	Limits the total amount of data that clients can send to the API	File uploads, data ingestion, POST/PUT APIs
DataLimitDownload	Limits the total amount of data that clients can receive from the API	Content delivery, data export, GET APIs

These can be used independently or together to control both directions of data flow.

How Data Limits Work

Data Limit rate limiters work by:

1. Starting with a configured maximum byte limit (the Bytes property)
2. Tracking each byte transferred in the relevant direction
3. Stopping data transfer once the total byte count is reached

Unlike Stream Rate limiters which control bandwidth (bytes per second), Data Limits control the total cumulative data transfer over the entire duration of the subscription, regardless of how quickly or slowly it is consumed.

Key Features

Data Limit rate limiters provide these essential capabilities:

• Direction-specific control: Separate limits for uploads and downloads
• Precise metering: Accurate tracking of data transferred in each direction
• Protocol support: Works with standard HTTP, WebSockets, and streaming protocols
• Volume-based pricing: Enables charging based on data consumption rather than request count
• Graceful enforcement: Clear responses when limits are reached

Data Tracking and Monitoring

ApiCharge accurately tracks data transfer across all supported protocols:

• Precise byte counting for both upload and download directions
• Real-time tracking accessible through response headers
• Consistent counting across HTTP and WebSocket protocols
• Accurate measurement regardless of content type or encoding

Clients can monitor their data usage through response headers, allowing them to anticipate when they'll need to purchase additional capacity.

Configuration

To configure Data Limit rate limiters in your quote definition, use the following syntax:

{
  "ApiChargeQuotes": {
    "Quotes": [
      {
        "RouteId": "content-api",
        "Duration": 86400,
        "MicroUnitPrice": 500000,
        "RateLimiters": [
          {
            "$type": "DataLimitUpload",
            "Bytes": 10485760
          },
          {
            "$type": "DataLimitDownload",
            "Bytes": 104857600
          }
        ]
      }
    ]
  }
}

In this example:

• "$type": "DataLimitUpload" - Limits data sent from the client to the server
• "Bytes": 10485760 - Allows up to 10 MB of upload data
• "$type": "DataLimitDownload" - Limits data sent from the server to the client
• "Bytes": 104857600 - Allows up to 100 MB of download data

Tiered Service Example

Creating tiered data services is a common use case for Data Limit rate limiters:

{
  "ApiChargeQuotes": {
    "Quotes": [
      {
        "RouteId": "media-service",
        "Duration": 2592000,
        "MicroUnitPrice": 1000000,
        "RateLimiters": [
          {
            "$type": "DataLimitDownload",
            "Bytes": 1073741824
          }
        ]
      },
      {
        "RouteId": "media-service",
        "Duration": 2592000,
        "MicroUnitPrice": 5000000,
        "RateLimiters": [
          {
            "$type": "DataLimitDownload",
            "Bytes": 5368709120
          }
        ]
      },
      {
        "RouteId": "media-service",
        "Duration": 2592000,
        "MicroUnitPrice": 9000000,
        "RateLimiters": [
          {
            "$type": "DataLimitDownload",
            "Bytes": 10737418240
          }
        ]
      }
    ]
  }
}

This configuration creates three monthly content delivery tiers:

• Basic: 1 GB download for 0.1 currency units
• Standard: 5 GB download for 0.5 currency units
• Premium: 10 GB download for 0.9 currency units

Response When Limit Is Reached

When a client reaches a Data Limit, the behavior depends on the context:

• For standard requests: HTTP 429 Too Many Requests with the message "Too Many Requests"
• For streaming connections: The connection is typically closed with an appropriate error code
• For upload limits: Can return HTTP 413 Payload Too Large if the limit would be exceeded by the current request

Clients should handle these responses by either waiting for the next subscription period or purchasing a new subscription with a higher data limit.

Performance Considerations

Data Limit rate limiters have minimal performance overhead:

• Memory overhead: Very low, requiring only a long counter per active subscription
• CPU overhead: Negligible for most workloads, with simple increment operations
• I/O impact: No measurable impact on transfer speeds when limits aren't being approached
• Redis usage: When using Redis for state persistence, data limit counters require minimal storage and network

Use Cases

Content Delivery

The most common use case is for content delivery services where billing is based on the volume of data transferred:

• Video streaming platforms with monthly data allowances
• Image hosting services with tiered download limits
• Document repositories with controlled download volumes

Data Storage and Processing

For data-focused APIs, data limits can help control both ingestion and retrieval costs:

• Cloud storage APIs with data transfer quotas
• Analytics APIs that process uploaded data
• ETL services with defined data processing limits

Combining with Call Count

For comprehensive API monetization, combine Data Limits with Call Count:

{
  "RouteId": "data-api",
  "Duration": 2592000,
  "MicroUnitPrice": 2000000,
  "RateLimiters": [
    {
      "$type": "CallCount",
      "Count": 10000
    },
    {
      "$type": "DataLimitUpload",
      "Bytes": 104857600
    },
    {
      "$type": "DataLimitDownload",
      "Bytes": 1073741824
    }
  ]
}

This configuration limits the API to 10,000 calls, 100 MB of uploads, and 1 GB of downloads, providing protection against various usage patterns that could otherwise lead to unexpected resource consumption.

Next Steps

Now that you understand Data Limit rate limiters, you can explore other rate limiter types:

• Call Count - Control the number of API calls
• Stream Rate - Control bandwidth for streaming connections