1space File Connector

The 1space File Connector extends the capabilities of 1space to access data stored in Filesystems. It can be deployed to sync data from the SwiftStack cluster to a NFS/SMB file share or to migrate data from the NFS/SMB file share to the SwiftStack cluster while also proving a single object namespace between data stored on the SwiftStack cluster and the file share.

The 1space File Connector is distributed as a docker image and requires the ability to run docker containers. Once the File Connector container has been deployed a sync or migration mappings can be created to pull or push data to your SwiftStack cluster.

Features

  1. You can run as many 1space File Connector container instances as you want--they share nothing, and scale horizontally.
  2. Every 1space File Connector docker container will map one filesystem directory to one Swift container.
  3. Each File Connector will also have its own 1space cloud credentials.
  4. On a Migration, 1space will pull large files and store them automatically as SLOs.
  5. On a Sync mapping, 1space will write SLOs or DLOs as one single large file on the filesystem directory.
  6. By default, File Connector will store file metadata in a hidden directory in the same directory a file is stored. This metadata is useful to 1space to determine data integrity as data is copied to/from file share. The File Connector can be configured to write the metadata directory in a separate file share, see below for details.
  7. The Docker container image for the 1space File Connector is publically available from Docker Hub as swiftstack/file-connector:TAG, where TAG is a release tag.

Limitations

  1. There’s no ACL configuration control on the file share.
  2. 1space Cloud Connector only speaks HTTP; if TLS (HTTPS) is desired, a TLS-terminator must be deployed in front of cloud-connector.
  3. Any desired load-balancing must be deployed in front of 1space File Connector.

Configuration Example

The following example shows how to configure and run 1space File Connector.

Run the 1space File Connector

You can run the container on any host that is accessible from your SwiftStack cluster. To run a file-connector container, you must supply the container with a path to the filesystem directory you wish to migrate. Make sure to use a full path when bind mounting. The service will bind to port 8083 inside the container. You must map this port to an open port on your host. By default, the docker container will need write/read permissions to the filesystem directory. Write permissions are needed to store metadata about the files.

Here is a template docker run command to run the file connector in the background exposed on port 8083 of the container host; substitute the parts in <>:

docker run -d -v </mnt/path/to/share>:/srv/fileconnector/<share_name> \
    -e FILE_CONNECTOR_SECRET=<your_secret_key> -p 8083:8083 \
    swiftstack/file-connector

Once the container is running, you can create Credentials for it. The information needed will be the container endpoint (e.g., http://ip:port/auth/v1.0) and the secret key passed in to FILE_CONNECTOR_SECRET.

Create a 1space Cloud Credential

Before adding a mapping, a set of credentials must be created. To setup credentials, navigate to the 1space tab in the manage cluster interface. Follow these steps to configure specific providers:

  1. Click on Credentials
  2. Select File Connector as the provider.
  3. Add credentials, by specifying a friendly Label, Secret and Endpoint. The Secret must be the same passed in to FILE_CONNECTOR_SECRET when running the container. The Endpoint URL must be similar to a Swift auth URL, such as http://swift.com:8083/auth/v1.0. Make sure to match the port numbers with what was used when running the container.

Create a 1space Sync or Migration Mapping

Proceed to configure a 1space sync or migration mapping similar to how it was detailed in the documentation above. When configuring data migration, add the <share_name> in the Source Bucket field and if configuring a sync mapping write the <share_name> in the Remote Bucket field.

Read-only share configuration

If you wish to maintain your data share read-only, use the USE_FC_METADATA_PATH flag and bind mount another host path to /fc_metadata where metadata will now be stored. Use the template below to run the container.

docker run -d -v </mnt/path/to/share>:/srv/fileconnector/<share_name> \
    -v </mnt/path/to/metadata_dir>:/fc_metadata \
    -e FILE_CONNECTOR_SECRET=<your_secret_key> \
    -e USE_FC_METADATA_PATH=true \
    -p 8083:8083 swiftstack/file-connector

Running Multiple File Connectors on the Same Host

When running multiple File Connectors on the same host, make sure to use different host port numbers for each container instance. See example below:

docker run -d -v /mnt/findata:/srv/fileconnector/findata \
    -e FILE_CONNECTOR_SECRET=password -p 8083:8083 \
    swiftstack/file-connector

docker run -d -v /mnt/marketing:/srv/fileconnector/marketing \
    -e FILE_CONNECTOR_SECRET=password2 -p 8084:8083 \
    swiftstack/file-connector