File Transfer Scripts

Minimal File Transfer Script

The following is an extremely minimal script to demonstrate a file transfer using the TransferClient.

It uses the tutorial client ID from the tutorials.

Note

You will need to replace the values for source_collection_id and dest_collection_id with UUIDs of collections that you have access to.

transfer_minimal.py [download]

import globus_sdk

# tutorial client ID (we recommend replacing this with your own client)
CLIENT_ID = "61338d24-54d5-408f-a10d-66c06b59f6d2"

# Replace these with your own collection UUIDs
SOURCE_COLLECTION_ID = "..."
DEST_COLLECTION_ID = "..."

# create a Transfer task consisting of one or more items
task_data = globus_sdk.TransferData(SOURCE_COLLECTION_ID, DEST_COLLECTION_ID)
task_data.add_item(
    "/share/godata/file1.txt",  # source
    "/~/minimal-example-transfer-script-destination.txt",  # dest
)

# create an app to manage login, use it to create a client, and submit,
# getting back the task ID
with globus_sdk.UserApp(
    "minimal-transfer-example",
    client_id=CLIENT_ID,
    # we set the 'auto_redrive_gares' flag, which enables handling for missing
    # auth requirements when the script is run against a changing set of collection IDs
    config=globus_sdk.GlobusAppConfig(auto_redrive_gares=True),
) as app:
    with globus_sdk.TransferClient(app=app) as transfer_client:
        task_doc = transfer_client.submit_transfer(task_data)

task_id = task_doc["task_id"]
print(f"submitted transfer, task_id={task_id}")

Best-Effort Proactive Handling of ConsentRequired

The above example works in most cases, and especially when there is a low cost to failing and retrying an activity. The auto_redrive_gares flag enables a behavior which will prompt the user for a fresh login if they are missing consents for access to various collections.

However, in some cases, responding to missing consents when the task is submitted is not acceptable. For example, for scripts used in batch job systems, the user cannot respond to the error until the job is already executing. The user would rather handle such issues when submitting their job.

The service still relies on ConsentRequired errors to indicate that some additional user consent is needed. But we can intentionally trigger them early to control when the user is prompted to resolve them.

The example below tries an ls operation before starting to build the task data. If the ls fails with ConsentRequired, the user can be put through the relevant login flow. Other errors (e.g., bad permissions) are suppressed, as they probably aren’t relevant to the user.

Note

The UserApp object is instantiated a second time, later in the script, to actually start the transfer. This loads the same tokens from the earlier login via the default token storage in ~/.globus/.

To manage tokens in another way, please see the documentation on Token Storages.

transfer_consent_required_proactive.py [download]

import argparse

import globus_sdk
from globus_sdk.scopes import TransferScopes

# do basic argument parsing
parser = argparse.ArgumentParser()
parser.add_argument("SRC")
parser.add_argument("DST")
args = parser.parse_args()

# tutorial client ID (we recommend replacing this with your own client)
CLIENT_ID = "61338d24-54d5-408f-a10d-66c06b59f6d2"
APP_NAME = "proactive-transfer-consent-example"


# Try an ls on the source and destination to see if ConsentRequired errors are raised --
# if they are, a fresh login flow will *not* be triggered.
#
# This is more sophisticated than handling with `redrive_gares=True` and makes
# sure that the user is only prompted to login *one* extra time, even if both
# collections require additional consent.
def probe_for_consent_required(
    transfer_client: globus_sdk.TransferClient, targets: list[str]
) -> list[str]:
    consent_required_scopes: list[str] = []

    for target in targets:
        try:
            transfer_client.operation_ls(target, path="/")
        # catch all errors and discard those other than ConsentRequired
        # e.g. ignore PermissionDenied errors as not relevant
        except globus_sdk.TransferAPIError as err:
            if err.info.consent_required:
                consent_required_scopes.extend(
                    err.info.consent_required.required_scopes
                )

    return consent_required_scopes


with globus_sdk.UserApp(APP_NAME, client_id=CLIENT_ID) as app:
    with globus_sdk.TransferClient(app=app) as transfer_client:
        consent_required_scopes = probe_for_consent_required(
            transfer_client, [args.SRC, args.DST]
        )

# the block above may or may not populate this list
# but if it does, handle ConsentRequired with a new login
if consent_required_scopes:
    print(
        "One of your endpoints requires consent in order to be used.\n"
        "You must login a second time to grant consents.\n\n"
    )
    with globus_sdk.UserApp(
        APP_NAME,
        client_id=CLIENT_ID,
        scope_requirements={
            TransferScopes.resource_server: consent_required_scopes
            + [TransferScopes.all]
        },
    ) as app:
        app.login()


# From this point onwards, the example is exactly the same as the previous scripts.
# We will *not* set `redrive_gares=True`, on the grounds that if you want to use this
# in a context like a job submission system, a prompt for login is not helpful if the
# consent was revoked or insufficient.

task_data = globus_sdk.TransferData(
    source_endpoint=args.SRC, destination_endpoint=args.DST
)
task_data.add_item(
    "/share/godata/file1.txt",  # source
    "/~/example-transfer-script-destination.txt",  # dest
)

with globus_sdk.UserApp(APP_NAME, client_id=CLIENT_ID) as app:
    with globus_sdk.TransferClient(app=app) as transfer_client:
        task_doc = transfer_client.submit_transfer(task_data)

task_id = task_doc["task_id"]
print(f"submitted transfer, task_id={task_id}")