Initiating a Transfer

Moving data within the Globus Ecosystem is performed by submitting a Transfer Task against the Globus Transfer service.

The below examples demonstrate how to do that using a globus sdk TransferClient. They are split into two categories:

  1. Transferring data between two known collections - both source and destination collections are known in advance and are likely be hardcoded into your script.

  2. Transferring data where at least one collection is unknown - either the source or destination collection will be determined at runtime (e.g. by script argument).

We differentiate these examples because certain collections have special auth requirements which must either be defined up front or fixed reactively if omitted. Certain collections (mapped non-high assurance ones) require that a special scope (“data_access”) to be attached to the transfer request to grant Transfer access to that collection’s data. If both collections are known this can be done proactively with a call to the add_app_data_access_scope method. If, however, one or more collections are unknown, the script must reactively solve the ConsentRequired error that is raised when the transfer is submitted.

Transferring data between two known collections

Note

The script references two globus hosted “tutorial” collections. Replace these ids & paths with your own collection ids and paths to move your own data.

Note

Some collections require you to attach a “data_access” scope to your transfer request. You should evaluate whether this is necessary for both your source and destination collections and omit the transfer_client.add_app_data_access_scope calls as needed.

A collection requires “data_access” if it is (1) a mapped collection and (2) is not high assurance.

submit_transfer_collections_known.py [download]
import globus_sdk
from globus_sdk.experimental.globus_app import UserApp

# Tutorial Client ID - <replace this with your own client>
NATIVE_CLIENT_ID = "61338d24-54d5-408f-a10d-66c06b59f6d2"
USER_APP = UserApp("my-simple-transfer", client_id=NATIVE_CLIENT_ID)

# Globus Tutorial Collection 1
# https://app.globus.org/file-manager/collections/6c54cade-bde5-45c1-bdea-f4bd71dba2cc
SRC_COLLECTION = "6c54cade-bde5-45c1-bdea-f4bd71dba2cc"
SRC_PATH = "/share/godata/file1.txt"

# Globus Tutorial Collection 2
# https://app.globus.org/file-manager/collections/31ce9ba0-176d-45a5-add3-f37d233ba47d
DST_COLLECTION = "31ce9ba0-176d-45a5-add3-f37d233ba47d"
DST_PATH = "/~/example-transfer-script-destination.txt"


def main():
    transfer_client = globus_sdk.TransferClient(app=USER_APP)

    # Comment out each of these lines if the referenced collection is either
    #   (1) A guest collection or (2) high assurance.
    transfer_client.add_app_data_access_scope(SRC_COLLECTION)
    transfer_client.add_app_data_access_scope(DST_COLLECTION)

    transfer_request = globus_sdk.TransferData(
        source_endpoint=SRC_COLLECTION,
        destination_endpoint=DST_COLLECTION,
    )
    transfer_request.add_item(SRC_PATH, DST_PATH)

    task = transfer_client.submit_transfer(transfer_request)
    print(f"Submitted transfer. Task ID: {task['task_id']}.")


if __name__ == "__main__":
    main()

Transferring data where at least one collection is unknown

In the case where your script does not know the full set of collections that it will be interacting with, you may need to reactively solve the ConsentRequired errors instead of proactively attaching the “data_access” scope.

This script demonstrates how to do that by:

  1. Attempting to submit the transfer without any “data_access” scopes.

  2. Intercepting any raised ConsentRequired errors if the request fails.

  3. Attaching any scope requirements detailed in the error.

  4. Retrying the transfer which implicitly puts your user through a consent flow to resolve their auth state.

Note

The script references two globus hosted “tutorial” collections. Replace these ids & paths with your own collection ids and paths to move your own data.

Note

Given that this script reactively fixes auth states, it can involve two user login interactions instead of the one required by the above proactive approach.

submit_transfer_collections_unknown.py [download]
import globus_sdk
from globus_sdk.experimental.globus_app import UserApp

# Tutorial Client ID - <replace this with your own client>
NATIVE_CLIENT_ID = "61338d24-54d5-408f-a10d-66c06b59f6d2"
USER_APP = UserApp("my-simple-transfer", client_id=NATIVE_CLIENT_ID)

# Globus Tutorial Collection 1
# https://app.globus.org/file-manager/collections/6c54cade-bde5-45c1-bdea-f4bd71dba2cc
SRC_COLLECTION = "6c54cade-bde5-45c1-bdea-f4bd71dba2cc"
SRC_PATH = "/share/godata/file1.txt"

# Globus Tutorial Collection 2
# https://app.globus.org/file-manager/collections/31ce9ba0-176d-45a5-add3-f37d233ba47d
DST_COLLECTION = "31ce9ba0-176d-45a5-add3-f37d233ba47d"
DST_PATH = "/~/example-transfer-script-destination.txt"


def main():
    transfer_client = globus_sdk.TransferClient(app=USER_APP)

    transfer_request = globus_sdk.TransferData(
        source_endpoint=SRC_COLLECTION,
        destination_endpoint=DST_COLLECTION,
    )
    transfer_request.add_item(SRC_PATH, DST_PATH)

    try:
        task = transfer_client.submit_transfer(transfer_request)
    except globus_sdk.TransferAPIError as err:
        if not err.info.consent_required:
            raise

        print("Additional consent required.")
        transfer_client.add_app_scope(err.info.consent_required.required_scopes)

        task = transfer_client.submit_transfer(transfer_request)
    print(f"Submitted transfer. Task ID: {task['task_id']}.")


if __name__ == "__main__":
    main()