Pickle Piper contains the following configuration files: - conf.YAML - pipeline.YAML

conf.yaml

conf.yaml is the base config file that details the system configuration in YAML format.

pipeline.YAML

pipeline.yaml is the configuration file for pipelines.

Pipeline.YAML contains 3 sections:

1. Input
2. Processor
3. Output

The following is an example configuration file that uses Kafka Topics and SASL PLAIN authentication mechanism:

Input:
  Kafka:
    topic: input_topic
    bootstrap_servers:
      - kafka_broker1:9092
      - kafka_broker2:9092
    group_id: stream_processor_group
    security_protocol: SASL_PLAINTEXT
    sasl_mechanism: PLAIN
    sasl_plain_username: your_username
    sasl_plain_password: your_password
    consumer_config:
      auto_offset_reset: latest
      enable_auto_commit: true
      auto_commit_interval_ms: 5000

Processor:
  MLModel:
    model_type: pickled_file
    model_path: /path/to/your/model.pkl
    # OR
    # model_type: safe_tensors
    # model_name: your_model_name

Output:
  Kafka:
    topic: output_topic
    bootstrap_servers:
      - kafka_broker1:9092
      - kafka_broker2:9092
    security_protocol: SASL_PLAINTEXT
    sasl_mechanism: PLAIN
    sasl_plain_username: your_username
    sasl_plain_password: your_password
    producer_config:
      acks: all
      retries: 3