Async Iterator Pattern

Process asynchronous data streams using modern iteration protocols

Pattern Overview

⚡ Async Iterator Pattern

The Async Iterator Pattern provides a standardized way to iterate over asynchronous data sources using JavaScript's native async iteration protocol. It enables processing streams of data that arrive over time.

Core Concepts

🔹 AsyncIterableIterator Interface - Implements Symbol.asyncIterator method
🔹 Async/Await Integration - Works seamlessly with for await...of loops
🔹 Lazy Evaluation - Data is fetched/processed only when needed
🔹 Stream Processing - Handle continuous data flows efficiently

Real-World Applications

API Pagination - Iterate through large datasets without loading everything into memory File Processing - Read large files in chunks without blocking the event loop WebSocket Streams - Process real-time messages as they arrive Database Cursors - Stream database results for large queries

Modern JavaScript Features

for await...of - Native syntax for async iteration Symbol.asyncIterator - Well-known symbol for async iteration protocol AsyncGenerator Functions - Function* async syntax for creating iterators Stream APIs - Integration with Node.js streams and Web Streams

Memory Efficiency

Streaming Data - Process large datasets without loading into memory Backpressure Handling - Control flow to prevent overwhelming consumers Lazy Loading - Fetch data only when iteration demands it Garbage Collection - Items can be collected after processing

Implementation Benefits

✅ Memory efficient - Stream processing without loading entire datasets
✅ Non-blocking - Async operations don't block event loop
✅ Composable - Can be chained with async utility functions
✅ Native integration - Works with modern JavaScript iteration syntax

Examples:

Paginated Data Stream Processing

Input:

createPaginatedIterator(5)

Output:

Stream of DataRecord objects with automatic pagination

Large File Chunk Processing

Input:

createFileIterator(content, 150)

Output:

Stream of FileChunk objects with data and metadata

Real-Time WebSocket Message Stream

Input:

createWebSocketIterator('ws://chat-server')

Output:

Stream of WebSocketMessage objects as they arrive

Concepts

design patternssoftware architecturecode organizationobject-oriented programming

Complexity Analysis

Time:O(n)

Space:O(1)

Implementation

paginated-data

Time: O(n) | Space: O(k)

// Async Iterator for paginated data
class PaginatedAsyncIterator implements AsyncIterableIterator<DataRecord> {
  private currentPage = 0;
  private currentPageData: DataRecord[] = [];
  private currentIndex = 0;
  private hasMorePages = true;

  constructor(
    private dataSource: AsyncDataSource,
    private pageSize: number = 10
  ) {}

  [Symbol.asyncIterator](): AsyncIterableIterator<DataRecord> {
    return this;
  }

  async next(): Promise<IteratorResult<DataRecord>> {
    // If we've consumed all items in current page, fetch next page
    if (this.currentIndex >= this.currentPageData.length && this.hasMorePages) {
      await this.fetchNextPage();
    }

    // If no more data available
    if (this.currentIndex >= this.currentPageData.length) {
      return { done: true, value: undefined };
    }

    // Return next item from current page
    const value = this.currentPageData[this.currentIndex];
    this.currentIndex++;
    return { done: false, value };
  }

  private async fetchNextPage(): Promise<void> {
    const result = await this.dataSource.fetchPage(this.currentPage, this.pageSize);
    
    // If this is a new page, reset index and append data
    if (this.currentPageData.length === this.currentIndex) {
      this.currentPageData = result.data;
      this.currentIndex = 0;
    } else {
      // Append to existing data
      this.currentPageData.push(...result.data);
    }
    
    this.hasMorePages = result.hasMore;
    this.currentPage++;
  }
}

// Usage
const dataSource = new AsyncDataSource(50);
const iterator = new PaginatedAsyncIterator(dataSource, 5);

// Stream through data without loading everything into memory
for await (const record of iterator) {
  console.log(`${record.id}: ${record.name} - ${record.value.toFixed(2)}`);
  if (record.id >= 10) break; // Process first 10 records
}