S3-compatible Destination
This page describes the S3 storage destination
The Simple Storage Service, S3, was originally described, developed and offered by Amazon via AWS. Since then, numerous other providers have adopted the protocol and offer S3-compatible services. While these services are mostly compatible with the core S3 protocol, a number of additional AWS-specific settings are usually not supported and will be ignored.
This page deals with S3 in general, for a specific setup on AWS S3, refer to the AWS specific page.
When storing data in S3, the storage is divided into a top-level "folder" called a "bucket", and each bucket has "objects", similar to files. For most providers, an object name with /characters will be interpreted as subfolders in some way.
In the original S3 specification, the bucket name was used as part of the hostname, causing some issues with bucket names that are not valid hostnames, and some delays for new buckets caused by DNS update speeds. Newer solutions use a single shared hostname and provide the bucket name as a parameter.
For AWS S3, and most other providers, the bucket name is a global name, shared across all users. This means that simple names, such as backup or data will likely be taken, and attempts to use these will cause permission errors. For AWS, the recommendation is to use a guid in the bucket name to make it unique. The Duplicati UI will recommend prefixing the account id to the bucket name, to make it unique.
To use S3 as the storage destination, us a format such as:
Note that the default for S3 is to use unencrypted connections. The connections are secured with signatures, but all data transfered can be captured through the network. If the provider supports SSL/TLS, which most do, make sure to add --use-ssl=trueto also encrypt the connection.
Make sure you consult the provider documentation to get the server name you need for the bucket region. If you are using AWS, see the AWS S3 description.
Choosing the client
The S3 storage destination can either use the AWS S3 library or Minio library, and you can choose the library to use with --s3-client=minio.
Generally, both libraries will work with most providers, but the AWS library has some defaults that may not be compatible with other providers. While you can configure the settings, it may be simpler to use Minio with the default settings.
Creating the bucket
Since the bucket defines the place where data is stored, a bucket needs to be created before it can be used. All providers will offer a way to do this through their UI, and allows you to set various options, such as which geographical region the bucket is located in.
If you use Duplicati to create the bucket, you can also set the option --s3-location-contraintto provide the desired location. Support for this, and available regions, depends on the provider.
Storage class
With S3 it is also possible to set the storage class which is sometimes used to fine-tune the cost/performance/durability of the files. The storage class is set with --s3-storage-class, but the possible settings depends on the provider.
Last updated