Conveyor Documentation

Welcome to Conveyor, a high-performance Python library for building efficient data processing pipelines with automatic batching, streaming capabilities, and robust error handling.

Quick Start

Getting Started

• Installation
• Quick Start
• Getting Started

User Guide

• User Guide
• Tasks
• Pipelines
• Streaming
• Context
• Side Inputs
• Error Handling
• Best Practices

Examples

• Examples
  • Basic Usage
  • Overview
  • Quick Examples
  • Available Examples
• Basic Usage
  • Single Task Example
  • Batch Task Example
  • Pipeline Composition
  • Mixed Pipeline Example
  • Error Handling
  • Using Context
  • Generator Input

API Reference

• API Reference

Features

• Automatic Batching: Efficiently process data in configurable batches

• Streaming Support: Handle continuous data streams with backpressure

• Error Handling: Robust error management with retry mechanisms

• Side Inputs: Support for auxiliary data in processing pipelines

• High Performance: Optimized for throughput and low latency

• Type Safety: Full type hints and validation support

Installation

Install Conveyor using pip:

pip install conveyor-streaming

Basic Example

import conveyor

@conveyor.task
def process_item(item):
    return item * 2

# Process a stream of data
pipeline = conveyor.Pipeline()
pipeline.add_task(process_item)

results = pipeline.run([1, 2, 3, 4, 5])
print(results)  # [2, 4, 6, 8, 10]

Getting Help

• Check the User Guide for detailed documentation

• Browse Examples for common use cases

• See the API Reference for complete function documentation