SwiftStack Metadata Overview

SwiftStack Metadata

All user resources in SwiftStack (i.e. account, container, objects) can have user metadata associated with them.

Metadata is a key:value pair that provides information about a resource. It can describe the data contained in the resource but is not the data in the resource itself.

User Metadata

User metadata takes the form of X-<type>-Meta-<key>: <value>, where <type> depends on the resources type (i.e. Account, Container, Object) and <key> and <value> are set by the client.

User metadata should generally be reserved for use by the client or client applications. A perfect example use-case for user metadata is python-swiftclient’s X-Object-Meta-Mtime which it stores on an object it uploads to implement its --changed option which will only upload files that have changed since the last upload.

User metadata that is stored by a PUT or POST request to a container or account resource persists until it is explicitly removed by a subsequent PUT or POST request that includes a header X-<type>-Meta-<key> with no value or a header X-Remove-<type>-Meta-<key>: <ignored-value>. In the latter case the <ignored-value> is not stored. All user metadata stored with an account or container resource is deleted when the account or container is deleted.

User metadata that is stored with an object resource has a different semantic; object user metadata persists until any subsequent PUT or POST request is made to the same object, at which point all user metadata stored with that object is deleted en-masse and replaced with any user metadata included with the PUT or POST request. As a result, it is not possible to update a subset of the user metadata items stored with an object while leaving some items unchanged.

System Metadata

System metadata takes the form of X-<type>-Sysmeta-<key>: <value>, where <type> depends on the resources type (i.e. Account, Container, Object) and <key> and <value> are set by trusted code running in a SwiftStack WSGI Server.

All headers on client requests in the form of X-<type>-Sysmeta-<key> will be dropped from the request before being processed by any middleware. All headers on responses from back-end systems in the form of X-<type>-Sysmeta-<key> will be removed after all middlewares have processed the response but before the response is sent to the client.

System metadata provides a means to store potentially private custom metadata with associated SwiftStack resources in a safe and secure fashion without actually having to plumb custom metadata through the core swift servers. The incoming filtering ensures that the namespace can not be modified directly by client requests, and the outgoing filter ensures that removing middleware that uses a specific system metadata key renders it benign. New middleware should take advantage of system metadata.

System metadata may be set on accounts and containers by including headers with a PUT or POST request. Where a header name matches the name of an existing item of system metadata, the value of the existing item will be updated. Otherwise existing items are preserved. A system metadata header with an empty value will cause any existing item with the same name to be deleted.

System metadata may be set on objects using only PUT requests. All items of existing system metadata will be deleted and replaced en-masse by any system metadata headers included with the PUT request. System metadata is neither updated nor deleted by a POST request: updating individual items of system metadata with a POST request is not yet supported in the same way that updating individual items of user metadata is not supported.

Modifying Metadata

There are three common ways of working with metadata in SwiftStack.

  • The SwiftStack Client

  • The python-swiftclient

  • Traditional curl commands

    • Show object metadata
    curl <cluster_url>/v1/<account_name>/<container_name>/<object_name> \
--head -H "X-Auth-Token: <AUTH_Token>"
    • Update object metadata
    curl <cluster_url>/v1/<account_name>/<container_name>/<object_name> \
-i -XCOPY -H "X-Auth-Token: <Auth_Token>" -H "X-Object-Meta-<meta_key>: \
<meta_value>" -H "Destination: <container_name>/<object_name>

    Note

    If the Destination header is the same as the original, the object is overwritten with the updated metadata.