Digital Ocean Object Storage | Spaces S3cmd Guide

Digital Ocean Object Storage | Spaces

Digital Ocean has started a new services called ‘Object Storage’ which provides users with a means to store their data.

In the Control Panel, there is a new tab for “Spaces”. This enables users to set up and manage data stored on the service.

Early access members are entitled to use Spaces for free till October 31, 2017. There is a limit of 1 TB of stored data.

The pricing of Spaces after that date is not known at present.

At present the Object Storage is only available at the NYC3 datacenter of Digital Ocean.

How To Configure s3cmd 2.x To Manage DigitalOcean Spaces

This guide will show how to install, configure, and manage DigitalOcean Spaces with the command-line tool s3cmd.

Prerequisites

To use this s3cmd, the following are required:

• A working installation of s3cmd version 2.0.0+. Note that package managers may be out of date. To get this version, visit the s3cmd Download page.

• An access key pair for your Spaces. To generate these, visit the API page in the DigitalOcean Control Panel. To learn more about access keys, see the Introduction to DigitalOcean Spaces access key section.

When you have completed the prerequisites, you’re ready to follow along.

Verify the Version

We’ll begin by confirming our version of s3cmd: with the following command:

s3cmd --version

Output:

s3cmd version 2.0.0+

By verifying that we’re running the same version, we can expect that the output of our commands will match the diretions that follow.

Choose the Configuration File

By default, s3cmd stores its configuration file, .s3cfg, in the home directory of the user who runs the configuration command. This is a plain text file of key/value pairs which can be edited directly once it has been created. You can choose between setting up DigitalOcean as the default configuration or creating a different configuration file:

Option 1: Make DigitalOcean the default connection

If DigitalOcean is the main or only provider you’ll connect to with s3cmd, configure it in the default ~/.s3cfg file with the following command:

s3cmd --configure

By choosing this option, you won’t have to specify the configuration file each time you run a command.

Option 2: Create an explicit configuration file for DigitalOcean

If you’re already using s3cmd with another service, you may want to create an alternate configuration file, which you can do by adding the -c flag and supplying a filename. The configuration file will be created in the directory where you issue the command, so specify the path if you want it created elsewhere.

In this example, we’ll place a config file named nyc3in our home directory:

s3cmd --configure -c ~/nyc3

Important: To use this configuration file, it must be explicitly provided at the end of each command by appending -c ~/nyc3.

Configure s3cmd

Whether you use the default configuration file or specify your own, when you issue the configuration command, it will launch an interactive multi-step configuration script. In this section, we’ll go through each of the steps in the configuration process. The places where you should substitute your specific information are surrounded by angle brackets < ... >.

Enter Access Keys

The script begins by asking for an Access Key and Secret Key. If you don’t already have keys, you can generate a set for s3cmd by visiting the Control Panel’s API page.

We’ll enter our keys, then accept US for the Default Region since the region information isn’t relevant to DigitalOcean.

Enter new values or accept defaults in brackets with Enter.

Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key []:
Secret Key []:
Default Region [US]:
Note: If you prefer, you can use the environment variables AWS_ACCESS_KEY_ID AWS_ACCESS_KEY_ID to store a set of keys.

Set the Endpoint

Next, we’ll enter the early access DigitalOcean endpoint, nyc3.digitaloceanspaces.com

Use “s3.amazonaws.com” for S3 Endpoint and not modify it to the target Amazon S3.

S3 Endpoint [s3.amazonaws.com]: nyc3.digitaloceanspaces.com

Since Spaces supports DNS-based buckets, at the next prompt, we’ll supply the bucket value in the required format:

%(bucket)s.nyc3.digitaloceanspaces.com

Use “%(bucket)s.s3.amazonaws.com” to the target Amazon S3. “%(bucket)s” and “%(location)s” vars

can be used if the target S3 system supports dns based buckets.

DNS-style bucket+hostname:port template for accessing a bucket []: %(bucket)s.nyc3.digitaloceanspaces.com

Optional: Set an encryption password

Next, we’re asked to supply an encryption password. Unlike HTTPS, which protects file only while in transit, GPG encryption prevents others from reading files while they are stored on DigitalOcean as well as in transit. Setting a password now won’t cause objects to be automatically encrypted; that’s done by adding a -e flag to the put command when you copy the object to a Space.

We’ll enter a password so it will be available in the event we want to use encryption.:

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3

Encryption password:
Path to GPG program [/usr/bin/gpg]:
Connect via HTTPS

Next, we’re prompted to connect via HTTPS, which protects data from being read while it is in transit. DigitalOcean Spaces do not support unencrypted transfer, so we’ll press ENTER to accept the default, YES:

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]:
Optional: Set a Proxy Server

If your network requires you to use an HTTP Proxy server, enter its IP address or domain name without the protocol, e.g. 203.0.113.1 or proxy.example.com Since we aren’t using an HTTP Proxy server, we’ll leave this question blank and press ENTER:

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can’t connect to S3 directly
HTTP Proxy server name:
Confirm and test settings

After the prompt for the HTTP Proxy server name, the configuration script presents a summary of the values it will use, followed by the opportunity to test them:

New settings:

Access Key: EXAMPLES7UQOTHDTF3GK4
Secret Key: b8e1ec97b97bff326955375c5example
Default Region: US
S3 Endpoint: nyc3.digitaloceanspaces.com
DNS-style bucket+hostname:port template for accessing a bucket: %(bucket)s.n
yc3.digitaloceanspaces.com
Encryption password: secure_password
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: True
HTTP Proxy server name:
HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n]

Save the settings

When the test completes successfully, enter Y to save the settings:

Please wait, attempting to list all buckets…
Success. Your access key and secret key worked fine 🙂

Now verifying that encryption works…
Success. Encryption and decryption worked fine 🙂

Save settings? [y/N]

If the test fails or you choose N you’ll have the opportunity to retry the configuration. Once you save the configuration, you’ll receive confirmation of its location:

Configuration saved to ‘/home/sammy/nyc3’

The test step confirms that we can connect to our DigitalOcean Spaces.

Conclusion

In this tutorial we’ve configured and tested the s3cmd settings.

• For a guide to some of the most common s3cmd tasks, see the How to Manage DigitalOcean Spaces with s3cmd.
• For a complete listing of options and commands, reference the s3cmd usage guide or access the help file from the command line with s3cmd –help.

How to Manage DigitalOcean Spaces with s3cmd

A Cheat Sheet

Introduction

s3cmd is a popular cross-platform command-line tool for managing S3 and S3-compatible object stores. This guide provides a quick reference to commands that are useful for managing DigitalOcean Spaces.

How to Use this Guide:

This guide is in cheat sheet format with self-contained command-line snippets
Jump to any section that is relevant to the task you are trying to complete
Important: If you followed How To Configure s3cmd 2.x To Manage DigitalOcean Spaces and chose to create a separate profile, remember to append -c ~/nyc3 or whatever you named your specific profile to the end of each command.

Create

Create one or more Spaces: mb

Use the command mb, short for “make bucket”, to create a new space:

s3cmd mb s3://spacename s3://secondspace

Create Directories

Object stores use a flat namespace and don’t have directories. What look like directories in the path of an object are actually part of the object’s file name. Graphical interfaces mimic the more familiar experience of directories for their users by creating an empty file for each element that doesn’t already exist.

From the command line, there’s no need to create empty files. Instead, type the path where you want the file to be as part of the put command.

List

List Spaces

Get a list of all the existing Spaces with ls:

s3cmd ls

List Contents of one or more specific Spaces

List all the contents of a one or more specific Spaces, recursively:

s3cmd ls s3://spacename s3://secondspace

List Contents of all Spaces List all the content in every Space:

s3cmd la --recursive

Copy

Copy Objects into a Space: put

Use the put command to copy files from your local machine to a Space. In all of these commands, you must include the trailing slash.

Put one file

When you include the trailing slash, as in the example below, the original file name will be appended. If you omit the slash, then the file will be copied to the space with the new name, path.

s3cmd put file.txt s3://spacename/path/

Put multiple files

This example shows multiple specific files getting transferred to a Space:

s3cmd put file1.txt file2.txt path/to/file3.txt s3://spacename/path/

Put all the files in your current working directory

Using the * with the put command will copy everything in the current working directory, recursively, into your Space:

s3cmd put * s3://spacename/path/ --recursive

Put a file under a new name

You can change the name of a file at the same time you put it in a Space by typing the new name at the end of the path as follows:

s3cmd put file.txt s3://spacename/newname.txt

Copy Files to your Local Machine: get

The command get copies files from a Space to your local machine.

Get one file

s3cmd get s3://spacename/path/to/file.txt

Get all the files in directory

To get multiple files, the s3 address must end with a trailing slash, and the command requires the –recursive flag.

s3cmd get s3://spacename/path/ --recursive

Get a file and save it under a new name

Like the put command, the command get command allows you to give the file a different name.

s3cmd get s3://spacename/file.txt newfilename.txt

Set Permissions

Currently the setacl commands are supported by DigitalOcean Spaces.

Set directory listings on a Space

Note that s3cmd only provides output when the command you issue changes tha access. For example, when you change the ACL from private to public, you’ll see output like s3://spacename/: ACL set to Public. If the ACL is already public, s3cmd will return silently to the command prompt.

Enable directory listings

s3cmd setacl s3://spacename/ --acl-public

Disable directory listings

s3cmd setacl s3://spacename/ --acl-private

Set permissions on file

Using the setacl command, files can be made private so that only someone connecting with a valid key pair will be able to read the file, or public so that anyone can read the file with either an S3 compatible client or via HTTPS.

Note that s3cmd only provides output when the command you issue changes tha access. For example, when you change the ACL from private to public, you’ll see output like s3://spacename/test.txt: ACL set to Public [1 of 1]. If the ACL is already public, s3cmd will return silently to the command prompt.

Make a file public

s3cmd setacl s3://spacename/file.txt --acl-public

Make all the files at a path public, recursively Use the –recursive flag to make multiple files public recursively:

s3cmd setacl s3://spacename/path/to/files/ --acl-public --recursive

Make a file private

s3cmd setacl s3://spacename/file.txt --acl-private

Make all the files at a path private, recursively Use the –recursive flag to make multiple files private recursively:

s3cmd setacl s3://spacename/path/to/files/ --acl-private --recursive

Delete

The s3cmd commands del and rm are identical and can be used interchangeably.

Delete a Space

Use rb short for “remove bucket” to delete an empty Space. If you wish to remove the Space and all its contents, append –recursive. This will permanently remove all the files.

s3cmd rb s3://spacename

Delete a file

s3cmd rm s3://spacename/name/of/file

Delete all files in a Space

Use rm or del with both the –recursive and –force flags to remove all the files in a Space but not the Space itself.

s3cmd rm s3://spacename/ --recursive --force

Encrypt a file

Adding the -e or –encrypt flag when you put a file in a Space with s3cmd will encrypt the file to protect if from being read on the server or in transit. If you download the file using s3cmd and the same configuration file, s3cmd will automatically use the password to decrypt it. Other users would need to use gpg -d file.txt to decrypt it and would have to enter the password you supplied. s3cmd allows you to use only one password, so it’s suitable for you and for other users with full administrative access. You can learn more about gpg’s symmetric encryption in The GNU Privacy Handbook:

s3cmd put s3://path/to/file.txt -e

Conclusion

This guide covered some of the common commands used to manage Spaces with s3cmd version 2.0.0+. There are many other flags and commands that can be used in combination. For a comprehensive guide to what’s available, see the s3cmd usage guide.

Read the official guide at Digital Ocean

DigitalOcean Spaces Object Storage

Leave a Reply

Your email address will not be published. Required fields are marked *