Access other storage backends via the S3 API
Clone or download

README.md

S3Proxy

Github All Releases Docker Pulls Maven Central Twitter Follow

S3Proxy implements the S3 API and proxies requests, enabling several use cases:

  • translation from S3 to Backblaze B2, EMC Atmos, Google Cloud, Microsoft Azure, and OpenStack Swift
  • testing without Amazon by using the local filesystem
  • extension via middlewares
  • embedding into Java applications

Usage with Docker

Docker Hub hosts a Docker image and has instructions on how to run it.

Usage without Docker

Users can download releases from GitHub. Developers can build the project by running mvn package which produces a binary at target/s3proxy. S3Proxy requires Java 7 to run.

Configure S3Proxy via a properties file. An example using the local file system as the storage backend with anonymous access:

s3proxy.authorization=none
s3proxy.endpoint=http://127.0.0.1:8080
jclouds.provider=filesystem
jclouds.filesystem.basedir=/tmp/s3proxy

First create the filesystem basedir:

mkdir /tmp/s3proxy

Next run S3Proxy. Linux and Mac OS X users can run the executable jar:

chmod +x s3proxy
s3proxy --properties s3proxy.conf

Windows users must explicitly invoke java:

java -jar s3proxy --properties s3proxy.conf

Finally test by creating a bucket then listing all the buckets:

$ curl --request PUT http://localhost:8080/testbucket

$ curl http://localhost:8080/
<?xml version="1.0" ?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>75aa57f09aa0c8caeab4f8c24e99d10f8e7faeebf76c078efc7c6caea54ba06a</ID><DisplayName>CustomersName@amazon.com</DisplayName></Owner><Buckets><Bucket><Name>testbucket</Name><CreationDate>2015-08-05T22:16:24.000Z</CreationDate></Bucket></Buckets></ListAllMyBucketsResult>

Usage with Java

Maven Central hosts S3Proxy artifacts and the wiki has instructions on Java use.

Supported storage backends

  • atmos
  • aws-s3 (Amazon-only)
  • azureblob
  • b2
  • filesystem (on-disk storage)
  • google-cloud-storage
  • openstack-swift
  • rackspace-cloudfiles-uk and rackspace-cloudfiles-us
  • s3 (all implementations)
  • transient (in-memory storage)

See the wiki for examples of configurations.

Middlewares

S3Proxy can modify its behavior based on middlewares:

Limitations

S3Proxy has broad compatibility with the S3 API, however, it does not support:

  • ACLs other than private and public-read
  • BitTorrent hosting
  • bucket logging
  • cross-origin resource sharing, see #142
  • hosting static websites
  • list objects v2, see #168
  • object server-side encryption
  • object tagging
  • object versioning, see #74
  • POST upload policies, see #73
  • requester pays buckets
  • select object content

S3Proxy emulates the following operations:

  • copy multi-part objects, see #76

The wiki collects compatibility notes for specific storage backends.

Support

GitHub issues provide community support. Andrew Gaul provides commercial support via andrew@gaul.org.

References

  • Apache jclouds provides storage backend support for S3Proxy
  • Ceph s3-tests help maintain and improve compatibility with the S3 API
  • fake-s3, gofakes3, S3 ninja, and s3rver provide functionality similar to S3Proxy when using the filesystem backend
  • GlacierProxy and SwiftProxy provide similar functionality for the Amazon Glacier and OpenStack Swift APIs
  • minio and Zenko provide similar multi-cloud functionality
  • s3mock mocks the S3 API for Java/Scala projects
  • sbt-s3 runs S3Proxy via the Scala Build Tool
  • swift3 provides an S3 middleware for OpenStack Swift

License

Copyright (C) 2014-2018 Andrew Gaul

Licensed under the Apache License, Version 2.0