Files Ingest
The File Ingest Task allows you to ingest files from remote storage systems into Gaio DataOS. It is commonly used to automate data ingestion from external servers such as SFTP or FTP, enabling repeatable and reliable file-based pipelines. This task focuses on connecting to a remote source, selecting files using flexible patterns, and storing them locally for downstream processing.
The File Ingest Task is designed to:
Connect to external file servers.
Download one or multiple files.
Support dynamic file selection patterns.
Store files locally for further processing.
Enable automated and scheduled ingestion workflows.
How to configure Files Ingest
Provide a descriptive name for the task.
Select the protocol used to access the remote server.
Select the authentication method used to connect.
Password-based authentication (SFTP and FTP).
Key-based authentication (SFTP and FTP).
Anonymous (without password).
Enter the hostname or IP address of the remote server.
Specify the port used by the selected protocol.
Enter the username used for authentication.
Imports a specific file, all files with the given extension or all files from the folder.
When importing more than one file, the remote folder path must end with /*
Specify the path to the file or folder on the remote server.
Define where the files will be stored locally.
Save and execute.
Best Practices (File Ingest)
Use explicit filenames when possible
Use wildcards only when needed
Always validate remote paths
Keep local paths organized by pipeline
Parameterize filenames for dynamic ingestion
Test connectivity before production use
Last updated