Skip to content

Streamlet

A Streamlet is a processor that handles streaming data flows. Streamlets are the building blocks for data pipelines, connecting sources of data to consumers through transformations.

In RIDDL 2.0 a streamlet is declared with the generic processor keyword and an optional shape ascription. The shape is otherwise derived from how many inlets and outlets the processor declares.

processor TemperatureProcessor as split is {
  inlet readings is type TemperatureReading
  outlet alerts is type TemperatureAlert
  outlet metrics is type TemperatureMetric

  handler ProcessReading is {
    on reading: event TemperatureReading {
      when reading.value > AlertThreshold then
        send event TemperatureAlert(reading.value) to outlet alerts
      end
      send event TemperatureMetric(reading.value) to outlet metrics
    }
  }
}

Shapes

Shape Inlets Outlets Description Synonym
source 0 1+ Generates data (external systems, timers)
sink 1+ 0 Consumes data (database writes, notifications)
flow 1 1 Transforms data from input to output cascade
merge 2+ 1 Combines data from several inputs into one fanin
split 1 2+ Routes data from one input to several outputs broadcast, fanout
router 1 2+ Routes data based on content or rules
void 0 0 No ports (placeholder or utility)

The dedicated shape keywords are deprecated

source, sink, flow, merge, split and router still parse as standalone keywords, but each emits a [deprecated] message telling you to write processor <id> as <keyword> instead. They are slated for removal in 3.0. Prettified output normalizes them, so running riddlc prettify over a 1.x model migrates them for you.

Sources

Sources generate data without receiving input. They might poll external systems, listen for external events, generate data on timers, or read from files and databases.

processor OrderEventSource as source is {
  outlet orders is type OrderEvent

  handler GenerateEvents is {
    on init {
      do "Subscribe to order queue and emit events"
    }
  }
}

Sinks

Sinks consume data without producing output. They might write to databases, send notifications, update external systems, or log and archive data.

processor NotificationSink as sink is {
  inlet notifications is type UserNotification

  handler SendNotifications is {
    on event UserNotification {
      do "Send notification via email or push"
    }
  }
}

Flows

Flows transform data from one shape to another:

processor OrderEnricher as flow is {
  inlet rawOrders is type RawOrder
  outlet enrichedOrders is type EnrichedOrder

  handler EnrichOrder is {
    on raw: event RawOrder {
      do "Look up customer details and product info"
      send event EnrichedOrder(raw.id) to outlet enrichedOrders
    }
  }
}

Connecting Processors

Processors are wired together with Connectors, which link an outlet to an inlet:

context DataPipeline is {
  processor Ingest    as source is { outlet events is type RawOrder }
  processor Transform as flow   is {
    inlet input is type RawOrder
    outlet output is type EnrichedOrder
  }
  processor Store     as sink   is { inlet data is type EnrichedOrder }

  connector IngestToTransform is
    from outlet Ingest.events to inlet Transform.input
  connector TransformToStore is
    from outlet Transform.output to inlet Store.data
}

Exactly one connector may attach to any given port. To fan out, declare more outlets rather than more connectors. To discard output you genuinely do not need, route it to the standard module's BottomlessPit.

Use Cases

  • Event Processing: React to events in real time
  • Data Integration: Move data between systems
  • ETL Pipelines: Extract, transform and load data
  • Monitoring: Collect and process metrics
  • Notifications: Route alerts to appropriate channels

Streamlets vs. Entities

Use Case Streamlet Entity
Stateless transformation Yes No
Long-lived business state No Yes
High-throughput data flow Yes Maybe
Complex business rules with state No Yes
Data enrichment/filtering Yes No
Order processing with lifecycle No Yes

Rule of thumb: If you need to remember something between messages, use an Entity. If you're transforming or routing messages without persistent state, use a streaming processor.

This is a question of purpose, not of capability: an entity may declare ports too, and often does — that is how it publishes its events into a stream.

Occurs In

Contains