Build a connector that processes files

Learn how to build a custom connector from the Grand Central connector template that picks up batch files from external storage, processes the records, and publishes status events to Sync Hub. Use this pattern when a vendor or internal system exchanges data as files, for example settlement files or bulk payment orders. For the concepts behind this pattern, see File processing. For delivery behavior and conventions, see File-processing conventions.

​

Prerequisites

Before you start, make sure you have:

• A connector repository scaffolded from the connector template. For more information, see Set up the project.
• Sync Hub deployed in your target runtime, with a topic for your status events. For more information, see Sync Hub deployment and configuration.
• Connection details and credentials for the external storage, for example an SFTP server or Azure Blob Storage container.
• A sample file that matches the agreed format (CSV, XML, or fixed-width).
• Java and Maven installed for local builds.

​

Build the connector

1. Configure the connection to external storage. Store credentials with SOPS and reference them from the connector’s values.yaml file. For more information, see Secrets management with SOPS.
2. Implement the file pickup route. Poll the storage location on a schedule, detect new files, and mark processed files so they aren’t picked up again.
3. Parse and validate each file. Reject malformed files early and route them to an error flow rather than processing partial content. For transformation guidance, see Transform and validate data.
4. Process each record in the file. Keep per-record processing independent so one failed record doesn’t block the rest of the file.
5. Publish status events to Sync Hub so downstream systems stay synchronized. Set the Sync Hub properties in the connector’s application.properties file:
   asb.event.type=com.backbase.payment.event.spec.v1.BatchOrderStatusEvent
   asb.event.source=com.backbase.payment
   asb.event.version=1.0.0
   asb.event.topic=batch-orders
   asb.producer.apiUri=http://asb-producer.synchub-producer.svc.cluster.local
   
   The connector builds the publish endpoint as ${asb.producer.apiUri}/${asb.event.topic}/publish. Set identifier and traceParent per request, because they change for each event. The asb.event.type value shown is an example. Use the event spec class that matches your event contract. Confirm the producer release name in the Application manifest for your runtime. Some runtimes deploy a versioned release name such as asb-producer-v0.
6. Add the Sync Hub pod labels to the connector’s values.yaml file so the runtime network policy allows the call to the producer:
   connector:
     customLabels:
       app.gcservices.io/synchub-enabled: "true"
       app.gcservices.io/synchub-type: "producer"
   
7. Build and run the connector locally:
   mvn clean verify
   mvn quarkus:dev
   

For a working example of a file-based connector that uses Sync Hub, see the OBPM batch connector.

​

Verify the connector

To confirm the connector processes files and publishes events:

1. Place a sample file in the storage location of your development environment.
2. Check the connector logs for file detection, record counts, and publish activity.
3. Check the producer logs in Grafana for the matching publish activity. For queries, see Logs.
4. Confirm the status events arrive on the topic with Azure Service Bus tooling. For more information, see Service Bus explorer.
5. Repeat with a file that contains an invalid record, and confirm the error flow behaves as designed.

​

Next steps

• Configure a custom connector: Secrets, properties, and traits for deployment.
• Test and validate: Add automated tests for parsing and error handling.
• Sync Hub go-live checklist: Verify your setup before production.