Automated and scheduled transfers

A scheduled job that moves files is the most valuable and the most fragile thing anybody sets up here. It works for months and then fails at 3am, quietly.

Five decisions make the difference.

1. Give the job its own account

Ask your administrator for an account used only by this job, not a person's account.

Then:

• The job survives that person leaving.
• Its activity is distinguishable in the log.
• Disabling it does not disable a human.

2. Use a key, never a password

Generate a key pair with no passphrase, because nothing can type one at 3am, and give the public half to your administrator. See Use an SSH key.

Then ask them to remove the password from that account. A job account with a password and a key is protected by the weaker of the two.

Protect the private key file with filesystem permissions: readable only by the user the job runs as.

3. Restrict where it can connect from

Ask your administrator to limit the account to the addresses your job actually runs from. They set this per user.

An account restricted to one address is worth far more than a long password.

4. Pin the host key

Your job must verify the server, and it cannot answer a prompt.

• On the command line, add the server's host key to your known hosts file once, on the machine the job runs on, as the user it runs as. Then the job connects without prompting and fails if the key ever changes.
• In a graphical client's scheduler, connect interactively once as the same user, accept the fingerprint after comparing it, and save the site.

Never disable host key checking

Turning off the check makes the error go away and makes the connection meaningless. If the key changes, you want the job to fail loudly, not to carry on talking to whoever answered.

5. Make failure visible

A transfer job that fails silently is worse than no job.

• Check the exit code and alert on it.
• Log to somewhere a person reads.
• Alert on nothing happening, not just on errors. A job that stopped running produces no errors at all.

A worked example

#!/bin/sh
set -eu

KEY=/etc/transfers/id_ed25519
USER=nightly@company
HOST=company.sftp.cloud
SRC=/var/exports
DEST=/incoming

sftp -b - -i "$KEY" -o User="$USER" -o BatchMode=yes "$HOST" <<'BATCH'
cd /incoming
lcd /var/exports
put *.csv
bye
BATCH

-b - reads the commands from standard input. BatchMode=yes makes the client fail rather than prompt for anything. set -e makes the script stop on the first failure, so your scheduler sees a non zero exit code.

What will eventually break it, and what to do

Cause	Symptom	Prevention
The host key changed	Every connection refused after a server move	Ask your administrator to tell you before a move, and publish the new fingerprint
The account was disabled	Sign in refused	Use a job account nobody would disable by accident
Permissions changed	The connection works and the operation fails	Alert on operation failures, not just on connection failures
The storage on their side is offline	Connection works; the folder on that storage refuses to open (Custom folders) or is absent (Automatic folders) while the rest keeps working	Alert on the failed operation or the missing folder, not only on a failed connection
A cloud credential expired behind the scenes	Operations fail suddenly	Nothing you can do from here. Alerting is what turns this into a phone call rather than a discovery

Things that will not work

• Shell commands over SSH. The connection is SFTP only: no shell, no remote command execution, no SCP, no port forwarding.
• Interactive prompts. Nothing can answer them. Use BatchMode=yes and a saved host key.
• Moving a file between two folders that live on different storage. The move is refused. Copy and delete instead.

Letting the service do the copying

A schedule that only moves files between your own storages does not need a client at all: a script on your Connector can ask the service to copy, move or keep a folder in sync between any two of your storages, as an automation identity, with the result recorded on the Connector. See Transfers from a script.

If your tooling only speaks FTPS

That works too; see Connect with an FTPS client. Use passive mode and require TLS explicitly.

Be aware that FTPS has no key based sign in, so the account will depend on a password. Restricting it to the job's own source addresses matters more there, not less.