Bulk Operations

The Bulk Operations middleware enables uploading and deleting many objects with just one request.

Configuration

On the Controller go to the Clusters page, click on the Manage button next to your cluster. On the Manage Cluster page, click on the Middleware tab. Click the Bulk Operations link.

Check the box next to Enabled.

../../_images/bulk.png

Once the change has been saved and deployed to the cluster, bulk operations can be performed.

Note

The swift command line tool does not support Bulk Operations. Please use curl.

Uploading Archives

You will need the AUTH_TOKEN and STORAGE_URL, more information on those can be found on the Authentication and Authorization page.

Requests must be of type PUT, and have query string ?extract-archive=<format>. Supported formats are tar, tar.gz, and tar.bz2. A container name may be specified, or derived from the folders within the archive.

The status-code of the reply will always be 200 OK, and actual results will be included in the body of the response. The response format defaults to text/plain but the Accept header can be used to specify application/json, application/xml, or text/xml.

curl -i -H "X-Auth-Token: <token>" [-H "Accept: <response-format>"] -X PUT <storage_url>/[<container>]/?extract-archive=<format> --data-binary @<file>

curl -i -H "X-Auth-Token: AUTH_tk8281adc254804c42b551d2f519e480e3" -X PUT \
    http://swift.example.com/v1/AUTH_bob//?extract-archive=tar.gz \
    --data-binary @example.tar.gz
HTTP/1.1 200 OK
Content-Type: text/plain
X-Trans-Id: txbe80d31baaa9467981217-0052094e4d
Date: Mon, 12 Aug 2013 21:06:21 GMT
Transfer-Encoding: chunked

Number Files Created: 4
Response Body:
Response Status: 201 Created
Errors:

curl -i -H "X-Auth-Token: AUTH_tk8281adc254804c42b551d2f519e480e3" \
    -H "Accept: application/json" -X PUT \
    http://swift.example.com/v1/AUTH_bob/archive2/?extract-archive=tar \
    --data-binary @example.tar
HTTP/1.1 200 OK
Content-Type: application/json
X-Trans-Id: tx2ccc8a5235174db788716-0052094ed8
Date: Mon, 12 Aug 2013 21:08:40 GMT
Transfer-Encoding: chunked

{"Number Files Created": 5, "Response Status": "201 Created", "Errors": [], "Response Body": ""}

Bulk Delete

Requests must be of type DELETE, and have an Content-Type header of text/plain. The data is a new-line separated list of url-encoded objects or containers to be deleted. Containers being deleted must already be empty. Use a query string of ?bulk-delete. As above, the response format defaults to text/plain but the Accept header can be used to specify application/json, application/xml, or text/xml. Also as above, the status-code of the reply will always be 200 OK, and actual results will be included in the body of the response.

Create a file containing the objects and containers to be deleted.

alpha/one.txt
alpha/two.txt
alpha
beta/three.txt
beta/four.txt
beta

curl -i -H "X-Auth-Token: <token>" -H "Content-Type: text/plain" [-H "Accept: <response-format>"] <storage_url>/?bulk-delete -X DELETE --data-binary @<list-file>

curl -i -H "X-Auth-Token: AUTH_tkd17e7282441a4bde9e09094f02a36108" \
    -H "Accept: application/json" -H "Content-Type: text/plain" \
    -X DELETE http://swift.example.com/v1/AUTH_bob/?bulk-delete \
    --data-binary @bulk-delete-list.txt
HTTP/1.1 200 OK
Content-Type: application/json
X-Trans-Id: txd7e5e895d61f49d19f22f-0052096a31
Date: Mon, 12 Aug 2013 23:05:21 GMT
Transfer-Encoding: chunked

{"Number Not Found": 0, "Response Status": "200 OK", "Errors": [], "Number Deleted": 6, "Response Body": ""}