pooch.Pooch

class pooch.Pooch(path, base_url, registry=None, urls=None, retry_if_failed=0, allow_updates=True)[source]

Manager for a local data storage that can fetch from a remote source.

Avoid creating Pooch instances directly. Use pooch.create instead.

Parameters

  • path (str) – The path to the local data storage folder. The path must exist in the file system.

  • base_url (str) – Base URL for the remote data source. All requests will be made relative to this URL.

  • registry (dict or None) – A record of the files that are managed by this good boy. Keys should be the file names and the values should be their hashes. Only files in the registry can be fetched from the local storage. Files in subdirectories of path must use Unix-style separators ('/') even on Windows.

  • urls (dict or None) – Custom URLs for downloading individual files in the registry. A dictionary with the file names as keys and the custom URLs as values. Not all files in registry need an entry in urls. If a file has an entry in urls, the base_url will be ignored when downloading it in favor of urls[fname].

  • retry_if_failed (int) – Retry a file download the specified number of times if it fails because of a bad connection or a hash mismatch. By default, downloads are only attempted once (retry_if_failed=0). Initially, will wait for 1s between retries and then increase the wait time by 1s with each retry until a maximum of 10s.

  • allow_updates (bool) – Whether existing files in local storage that have a hash mismatch with the registry are allowed to update from the remote URL. If False, any mismatch with hashes in the registry will result in an error. Defaults to True.

Methods Summary

Pooch.fetch(fname[, processor, downloader, …])

Get the absolute path to a file in the local storage.

Pooch.get_url(fname)

Get the full URL to download a file in the registry.

Pooch.is_available(fname)

Check availability of a remote file without downloading it.

Pooch.load_registry(fname)

Load entries from a file and add them to the registry.
Pooch.fetch(fname, processor=None, downloader=None, progressbar=False)[source]

Get the absolute path to a file in the local storage.

If it’s not in the local storage, it will be downloaded. If the hash of the file in local storage doesn’t match the one in the registry, will download a new copy of the file. This is considered a sign that the file was updated in the remote storage. If the hash of the downloaded file still doesn’t match the one in the registry, will raise an exception to warn of possible file corruption.

Post-processing actions sometimes need to be taken on downloaded files (unzipping, conversion to a more efficient format, etc). If these actions are time or memory consuming, it would be best to do this only once right after the file is downloaded. Use the processor argument to specify a function that is executed after the download to perform these actions. See Processors: Post-download actions for details.

Custom file downloaders can be provided through the downloader argument. By default, Pooch will determine the download protocol from the URL in the registry. If the server for a given file requires authentication (username and password), use a downloader that support these features. Downloaders can also be used to print custom messages (like a progress bar), etc. See Downloaders: Customizing the download for details.

Parameters

  • fname (str) – The file name (relative to the base_url of the remote data storage) to fetch from the local storage.

  • processor (None or callable) – If not None, then a function (or callable object) that will be called before returning the full path and after the file has been downloaded. See Processors: Post-download actions for details.

  • downloader (None or callable) – If not None, then a function (or callable object) that will be called to download a given URL to a provided local file name. See Downloaders: Customizing the download for details.

  • progressbar (bool or an arbitrary progress bar object) – If True, will print a progress bar of the download to standard error (stderr). Requires tqdm to be installed. Alternatively, an arbitrary progress bar object can be passed. See Using custom progress bars for details.

Returns

full_path (str) – The absolute path (including the file name) of the file in the local storage.

Pooch.get_url(fname)[source]

Get the full URL to download a file in the registry.

Parameters

fname (str) – The file name (relative to the base_url of the remote data storage) to fetch from the local storage.

Pooch.is_available(fname)[source]

Check availability of a remote file without downloading it.

Use this method when working with large files to check if they are available for download.

Parameters

fname (str) – The file name (relative to the base_url of the remote data storage) to fetch from the local storage.

Returns

status (bool) – True if the file is available for download. False otherwise.

Pooch.load_registry(fname)[source]

Load entries from a file and add them to the registry.

Use this if you are managing many files.

Each line of the file should have file name and its hash separated by a space. Hash can specify checksum algorithm using “alg:hash” format. In case no algorithm is provided, SHA256 is used by default. Only one file per line is allowed. Custom download URLs for individual files can be specified as a third element on the line. Line comments can be added and must be prepended with #.

Parameters

fname (str | fileobj) – Path (or open file object) to the registry file.

pooch.create pooch.retrieve