Build and Test Pipelines
Use the Cloud pipeline editor to connect inputs, transformations, and outputs, then test the transformation logic before choosing deployment targets. The visual graph and YAML editor provide two ways to work with the pipeline configuration.
Start a pipeline
From the workspace's Pipelines view, create a pipeline. Start from scratch, upload a YAML configuration, or choose a public or organization template. Templates let teams reuse a configuration and adapt it for a new task.
Use the component picker and configuration panel to add or edit inputs, processors, and outputs. The component catalog explains each component's fields and behavior. Use the YAML editor when direct configuration editing is more convenient.
The current creation wizard expects an input, an output, and at least one processor before continuing. Consult the configuration formats guide when moving between a pipeline configuration and the full job format used by the CLI.
Validate before continuing
Cloud lints the configuration and displays issues in the editor. The wizard validates again before advancing to deployment settings. Correct reported errors before proceeding.
Configuration validation checks what can be established from the definition. It does not establish that a production node can reach an endpoint, obtain a credential, or write to a destination. Check those dependencies in the intended deployment environment as well.
Test transformations
Provide sample input to test transformation behavior while editing. Cloud returns transformed output together with errors and warnings. Adjust the configuration and sample to investigate parsing, field mapping, filtering, and other processing logic.
Use examples that exercise the cases you expect in production: normal records, missing fields, malformed values, and records that should be rejected or routed differently. The test excludes inputs and outputs, can skip unsupported processors, and returns at most 100 output messages. Review warnings to see which processing steps were exercised. A successful sample test helps you review the logic; it is not a throughput benchmark or an end-to-end connector test.
Choose test data
Transformation tests run in a browser worker when available and can fall back to Cloud's testing service. Use synthetic or redacted records when the original data should remain on your infrastructure. The same care applies to configuration and samples pasted into the pipeline assistant.
Production execution follows a different path: the deployed pipeline runs on your selected edge nodes and sends records to its configured destinations.
Reuse pipeline configuration
YAML files and templates make pipeline logic reusable. Before deploying a reused configuration, review source and destination addresses, secret references, node selectors, and rollout settings for the new environment. A template does not provision its external dependencies.
Maintain organization templates
Use the pipeline catalog to create an organization template with a name, description, tags, and YAML content. Edit an existing template as your team's preferred configuration evolves, or download its configuration for review and reuse outside the catalog. Public templates provide additional starting points.
A template is a reusable starting configuration. Editing it is not the same as releasing an update to pipelines previously created from it; review and deploy those pipelines through their normal lifecycle.
When the configuration is ready, continue to Deployments and rollback. For local validation and additional debugging tools, see Testing and debugging.
