Globus Transfer#

Helper Objects#

These helper objects make it easier to correctly create data for consumption by a TransferClient.

class globus_sdk.TransferData(transfer_client=None, source_endpoint=None, destination_endpoint=None, *, label=None, submission_id=None, sync_level=None, verify_checksum=False, preserve_timestamp=False, encrypt_data=False, deadline=None, skip_activation_check=None, skip_source_errors=False, fail_on_quota_errors=False, recursive_symlinks=None, delete_destination_extra=False, notify_on_succeeded=True, notify_on_failed=True, notify_on_inactive=True, source_local_user=None, destination_local_user=None, additional_fields=None)[source]#

Bases: PayloadWrapper

Convenience class for constructing a transfer document, to use as the data parameter to submit_transfer.

At least one item must be added using add_item.

If submission_id isn’t passed, one will be fetched automatically. The submission ID can be pulled out of here to inspect, but the document can be used as-is multiple times over to retry a potential submission failure (so there shouldn’t be any need to inspect it).

Parameters:

transfer_client (globus_sdk.TransferClient | None) – A TransferClient instance which will be used to get a submission ID if one is not supplied. Should be the same instance that is used to submit the transfer.
source_endpoint (UUIDLike | None) – The endpoint ID of the source endpoint
destination_endpoint (UUIDLike | None) – The endpoint ID of the destination endpoint
label (str | None) – A string label for the Task
submission_id (UUIDLike | None) – A submission ID value fetched via get_submission_id . Defaults to using transfer_client.get_submission_id
sync_level (_StrSyncLevel | int | None) – The method used to compare items between the source and destination. One of "exists", "size", "mtime", or "checksum" See the section below on sync-level for an explanation of values.
verify_checksum (bool) – When true, after transfer verify that the source and destination file checksums match. If they don’t, re-transfer the entire file and keep trying until it succeeds. This will create CPU load on both the origin and destination of the transfer, and may even be a bottleneck if the network speed is high enough. [default: False]
preserve_timestamp (bool) – When true, Globus Transfer will attempt to set file timestamps on the destination to match those on the origin. [default: False]
encrypt_data (bool) – When true, all files will be TLS-protected during transfer. [default: False]
deadline (datetime.datetime | str | None) – An ISO-8601 timestamp (as a string) or a datetime object which defines a deadline for the transfer. At the deadline, even if the data transfer is not complete, the job will be canceled. We recommend ensuring that the timestamp is in UTC to avoid confusion and ambiguity. Examples of ISO-8601 timestamps include 2017-10-12 09:30Z, 2017-10-12 12:33:54+00:00, and 2017-10-12
recursive_symlinks (str | None) – Specify the behavior of recursive directory transfers when encountering symlinks. One of "ignore", "keep", or "copy". "ignore" skips symlinks, "keep" creates symlinks at the destination matching the source (without modifying the link path at all), and "copy" follows symlinks on the source, failing if the link is invalid. [default: "ignore"]
skip_activation_check (bool | None) – When true, allow submission even if the endpoints aren’t currently activated
skip_source_errors (bool) – When true, source permission denied and file not found errors from the source endpoint will cause the offending path to be skipped. [default: False]
fail_on_quota_errors (bool) – When true, quota exceeded errors will cause the task to fail. [default: False]
delete_destination_extra (bool) – Delete files, directories, and symlinks on the destination endpoint which don’t exist on the source endpoint or are a different type. Only applies for recursive directory transfers. [default: False]
notify_on_succeeded (bool) – Send a notification email when the transfer completes with a status of SUCCEEDED. [default: True]
notify_on_failed (bool) – Send a notification email when the transfer completes with a status of FAILED. [default: True]
notify_on_inactive (bool) – Send a notification email when the transfer changes status to INACTIVE. e.g. From credentials expiring. [default: True]
source_local_user (str | None) – Optional value passed to the source’s identity mapping specifying which local user account to map to. Only usable with Globus Connect Server v5 mapped collections.
destination_local_user (str | None) – Optional value passed to the destination’s identity mapping specifying which local user account to map to. Only usable with Globus Connect Server v5 mapped collections.
additional_fields (dict[str, t.Any] | None) – additional fields to be added to the transfer document. Mostly intended for internal use

Sync Levels

The values for sync_level are used to determine how comparisons are made between files found both on the source and the destination. When files match, no data transfer will occur.

For compatibility, this can be an integer 0, 1, 2, or 3 in addition to the string values.

The meanings are as follows:

value	behavior
`0`, `exists`	Determine whether or not to transfer based on file existence. If the destination file is absent, do the transfer.
`1`, `size`	Determine whether or not to transfer based on the size of the file. If destination file size does not match the source, do the transfer.
`2`, `mtime`	Determine whether or not to transfer based on modification times. If source has a newer modified time than the destination, do the transfer.
`3`, `checksum`	Determine whether or not to transfer based on checksums of file contents. If source and destination contents differ, as determined by a checksum of their contents, do the transfer.

Examples

See the submit_transfer documentation for example usage.

External Documentation

See the Task document definition and Transfer specific fields in the REST documentation for more details on Transfer Task documents.

Methods

add_filter_rule()
add_item()
add_symlink_item()
iter_items()

add_filter_rule(name, *, method='exclude', type=None)[source]#

Add a filter rule to the transfer document.

These rules specify which items are or are not included when recursively transferring directories. Each item that is found during recursive directory traversal is matched against these rules in the order they are listed. The method of the first filter rule that matches an item is applied (either “include” or “exclude”), and filter rule matching stops. If no rules match, the item is included in the transfer. Notably, this makes “include” filter rules only useful when overriding more general “exclude” filter rules later in the list.

Parameters:

name (str) – A pattern to match against item names. Wildcards are supported, as are character groups: * matches everything, ? matches any single character, [] matches any single character within the brackets, and [!] matches any single character not within the brackets.
method (Literal['include', 'exclude']) – The method to use for filtering. If “exclude” (the default) items matching this rule will not be included in the transfer. If “include” items matching this rule will be included in the transfer.
type (None | Literal['file', 'dir']) – The types of items on which to apply this filter rule. Either "file" or "dir". If unspecified, the rule applies to both. Note that if a "dir" is excluded then all items within it will also be excluded regardless if they would have matched any include rules.

Example Usage:

>>> tdata = TransferData(...)
>>> tdata.add_filter_rule(method="exclude", "*.tgz", type="file")
>>> tdata.add_filter_rule(method="exclude", "*.tar.gz", type="file")

tdata now describes a transfer which will skip any gzipped tar files with the extensions .tgz or .tar.gz

>>> tdata = TransferData(...)
>>> tdata.add_filter_rule(method="include", "*.txt", type="file")
>>> tdata.add_filter_rule(method="exclude", "*", type="file")

tdata now describes a transfer which will only transfer files with the .txt extension.

add_item(source_path, destination_path, *, recursive=None, external_checksum=None, checksum_algorithm=None, additional_fields=None)[source]#

Add a file or directory to be transferred. If the item is a symlink to a file or directory, the file or directory at the target of the symlink will be transferred.

Appends a transfer_item document to the DATA key of the transfer document.

Note

The full path to the destination file must be provided for file items. Parent directories of files are not allowed. See task submission documentation for more details.

Parameters:

source_path (str) – Path to the source directory or file to be transferred
destination_path (str) – Path to the destination directory or file will be transferred to
recursive (bool | None) – Set to True if the target at source path is a directory
external_checksum (str | None) – A checksum to verify both source file and destination file integrity. The checksum will be verified after the data transfer and a failure will cause the entire task to fail. Cannot be used with directories. Assumed to be an MD5 checksum unless checksum_algorithm is also given.
checksum_algorithm (str | None) – Specifies the checksum algorithm to be used when verify_checksum is True, sync_level is “checksum” or 3, or an external_checksum is given.
additional_fields (dict[str, Any] | None) – additional fields to be added to the transfer item

add_symlink_item(source_path, destination_path)[source]#

Add a symlink to be transferred as a symlink rather than as the target of the symlink.

Appends a transfer_symlink_item document to the DATA key of the transfer document.

Parameters:

source_path (str) – Path to the source symlink
destination_path (str) – Path to which the source symlink will be transferred

iter_items()[source]#

An iterator of items created by add_item.

Each item takes the form of a dictionary.

Return type:: Iterator[dict[str, Any]]

class globus_sdk.DeleteData(transfer_client=None, endpoint=None, *, label=None, submission_id=None, recursive=False, ignore_missing=False, interpret_globs=False, deadline=None, skip_activation_check=None, notify_on_succeeded=True, notify_on_failed=True, notify_on_inactive=True, local_user=None, additional_fields=None)[source]#

Bases: PayloadWrapper

Convenience class for constructing a delete document, to use as the data parameter to submit_delete.

At least one item must be added using add_item.

Parameters:

transfer_client (globus_sdk.TransferClient | None) – A TransferClient instance which will be used to get a submission ID if one is not supplied. Should be the same instance that is used to submit the deletion.
endpoint (UUIDLike | None) – The endpoint ID which is targeted by this deletion Task
label (str | None) – A string label for the Task
submission_id (UUIDLike | None) – A submission ID value fetched via get_submission_id. Defaults to using transfer_client.get_submission_id if a transfer_client is provided
recursive (bool) – Recursively delete subdirectories on the target endpoint [default: False]
ignore_missing (bool) – Ignore nonexistent files and directories instead of treating them as errors. [default: False]
interpret_globs (bool) – Enable expansion of \*?[] characters in the last component of paths, unless they are escaped with a preceding backslash, \\ [default: False]
deadline (str | datetime.datetime | None) – An ISO-8601 timestamp (as a string) or a datetime object which defines a deadline for the deletion. At the deadline, even if the data deletion is not complete, the job will be canceled. We recommend ensuring that the timestamp is in UTC to avoid confusion and ambiguity. Examples of ISO-8601 timestamps include 2017-10-12 09:30Z, 2017-10-12 12:33:54+00:00, and 2017-10-12
skip_activation_check (bool | None) – When true, allow submission even if the endpoint isn’t currently activated
notify_on_succeeded (bool) – Send a notification email when the delete task completes with a status of SUCCEEDED. [default: True]
notify_on_failed (bool) – Send a notification email when the delete task completes with a status of FAILED. [default: True]
notify_on_inactive (bool) – Send a notification email when the delete task changes status to INACTIVE. e.g. From credentials expiring. [default: True]
local_user (str | None) – Optional value passed to identity mapping specifying which local user account to map to. Only usable with Globus Connect Server v5 mapped collections.
additional_fields (dict[str, t.Any] | None) – additional fields to be added to the delete document. Mostly intended for internal use

Examples

See the submit_delete documentation for example usage.

External Documentation

See the Task document definition and Delete specific fields in the REST documentation for more details on Delete Task documents.

Methods

add_filter_rule()
add_item()
add_symlink_item()
iter_items()

add_item(path, *, additional_fields=None)[source]#

Add a file or directory or symlink to be deleted. If any of the paths are directories, recursive must be set True on the top level DeleteData. Symlinks will never be followed, only deleted.

Appends a delete_item document to the DATA key of the delete document.

Parameters:

path (str) – Path to the directory or file to be deleted
additional_fields (dict[str, Any] | None) – additional fields to be added to the delete item

iter_items()[source]#

An iterator of items created by add_item.

Each item takes the form of a dictionary.

Return type:: Iterator[dict[str, Any]]

Client Errors#

When an error occurs, a TransferClient will raise this specialized type of error, rather than a generic GlobusAPIError.

class globus_sdk.TransferAPIError(r, *args, **kwargs)[source]#

Bases: GlobusAPIError

Error class for the Transfer API client.

Transfer Responses#

class globus_sdk.ActivationRequirementsResponse(*args, **kwargs)[source]#

Bases: GlobusHTTPResponse

Response class for Activation Requirements responses.

All Activation Requirements documents refer to a specific Endpoint, from whence they were acquired. References to “the Endpoint” implicitly refer to that originating Endpoint, and not to some other Endpoint.

External Documentation

See Activation Requirements Document in the API documentation for details.

active_until(time_seconds, relative_time=True)[source]#

Check if the Endpoint will be active until some time in the future, given as an integer number of seconds. When relative_time=False, the time_seconds is interpreted as a POSIX timestamp.

This supports queries using both relative and absolute timestamps to better support a wide range of use cases. For example, if I have a task that I know will typically take N seconds, and I want an M second safety margin:

>>> num_secs_allowed = N + M
>>> tc = TransferClient(...)
>>> reqs_doc = tc.endpoint_get_activation_requirements(...)
>>> if not reqs_doc.active_until(num_secs_allowed):
>>>     raise Exception("Endpoint won't be active long enough")
>>> ...

or, alternatively, if I know that the endpoint must be active until October 18th, 2016 for my tasks to complete:

>>> oct18_2016 = 1476803436
>>> tc = TransferClient(...)
>>> reqs_doc = tc.endpoint_get_activation_requirements(...)
>>> if not reqs_doc.active_until(oct18_2016, relative_time=False):
>>>     raise Exception("Endpoint won't be active long enough")
>>> ...

Parameters:

time_seconds (int) – Number of seconds into the future.
relative_time (bool) – Defaults to True. When False, time_seconds is treated as a POSIX timestamp (i.e. seconds since epoch as an integer) instead of its ordinary behavior.

Returns:

True if the Endpoint will be active until the deadline, False otherwise

Return type:

bool

property always_activated: bool#: Returns True if the endpoint activation never expires (e.g. shared endpoints, globus connect personal endpoints).

property supports_auto_activation: bool#

Check if the document lists Auto-Activation as an available type of activation. Typically good to use when you need to catch endpoints that require web activation before proceeding.

>>> endpoint_id = "..."
>>> tc = TransferClient(...)
>>> reqs_doc = tc.endpoint_get_activation_requirements(endpoint_id)
>>> if not reqs_doc.supports_auto_activation:
>>>     # use `from __future__ import print_function` in py2
>>>     print(("This endpoint requires web activation. "
>>>            "Please login and activate the endpoint here:\n"
>>>            "https://app.globus.org/file-manager?origin_id={}")
>>>           .format(endpoint_id), file=sys.stderr)
>>>     # py3 calls it `input()` in py2, use `raw_input()`
>>>     input("Please Hit Enter When You Are Done")

property supports_web_activation: bool#

Check if the document lists known types of activation that can be done through the web. If this returns False, it means that the endpoint is of a highly unusual type, and you should directly inspect the response’s data attribute to see what is required. Sending users to the web page for activation is also a fairly safe action to take. Note that ActivationRequirementsResponse.supports_auto_activation directly implies ActivationRequirementsResponse.supports_web_activation, so these are not exclusive.

For example,

>>> tc = TransferClient(...)
>>> reqs_doc = tc.endpoint_get_activation_requirements(...)
>>> if not reqs_doc.supports_web_activation:
>>>     # use `from __future__ import print_function` in py2
>>>     print("Highly unusual endpoint. " +
>>>           "Cannot webactivate. Raw doc: " +
>>>           str(reqs_doc), file=sys.stderr)
>>>     print("Sending user to web anyway, just in case.",
>>>           file=sys.stderr)
>>> ...

class globus_sdk.IterableTransferResponse(response, client=None, *, iter_key=None)[source]#

Bases: IterableResponse

Response class for non-paged list oriented resources. Allows top level fields to be accessed normally via standard item access, and also provides a convenient way to iterate over the sub-item list in a specified key:

>>> print("Path:", r["path"])
>>> # Equivalent to: for item in r["DATA"]
>>> for item in r:
>>>     print(item["name"], item["type"])

Globus Transfer#

Client#

Helper Objects#

Client Errors#

Transfer Responses#