1 02 2016

Openshift registry on Amazon S3

gaelL | aws, devops, docker, linux, openshift3, redhat, s3, troubleshooting | 3

Few weeks ago, I was working with Openshift on AWS. For this reason I was looking for a cool feature : Use an AWS s3 bucket as backend for Openshift registry.

This feature works but it’s not very documented yet. In this post we are going to share with you the issues we encountered and how we did to make it working.

All the following steps have been tested on Openshift v3.1.0.4.

Create the registry

The first step is to create a dedicated IAM for the S3 and apply this kind of policy on your bucket:

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Action":"s3:*",
         "arn:aws:s3:::mybucket-docker-registry"
      }
   ]
}

S3 policy

Once the policy is set, it’s time to create the registry. We have to start with a basic one, after that we will apply the s3 config.

oadm registry --create --service-account=registry \
--config=/etc/origin/master/admin.kubeconfig \
--credentials=/etc/origin/master/openshift-registry.kubeconfig \
--selector="region=infra"

Create registry

Openshift provide an Ansible playbook to configuration the registry with s3 backend. Let’s use it.

Currently the ansible playbook create the S3 bucket. So if you have a restricted policy on bucket creation for your IAM like us or if your bucket is already created, you will have to edit the playbook and remove the create part.

Be careful, the region and bucket name are “hardcoded” and the auth part in the registry config is missing. Some pull requests are in progress to fix these issues :

This playbook is in progress. Please check if some lines are changed or edit the playbook before run.

export S3_ACCESS_KEY_ID=...
export S3_SECRET_ACCESS_KEY=...
ansible-playbook playbooks/adhoc/s3_registry/s3_registry.yml \
-e clusterid="openpaas-blue-docker-registry" \
-e aws_bucket="openpaas-blue-docker-registry" -v

ansible run

Test the registry on S3

For this part the best is to build a new app to test the full stack. In our case we only want to do some quick tests focused on the registry only.

Get the service IP of the registry:

oc get svc
NAME CLUSTER_IP EXTERNAL_IP PORT(S) SELECTOR AGE
docker-registry 172.30.234.98 <none> 5000/TCP docker-registry=default 44m

oc get svc

The First dummy test is to curl the registry to know if the container listen at least on the expected port :

# curl -k <registry_ip>:5000/healthz
[root@ip-172-31-28-54 ~]# curl -k 172.30.56.252:5000/healthz
{}

curl

It’s time to do a more representative test. Manually push an image in the registry. (https://docs.openshift.com/enterprise/3.0/install_config/install/docker_registry.html#access)

Start to get a valid token from one of your Openshift users

oc login
...
oc whoami -t
>>><token>

get token

Pull a small image for the test. For example the busybox image. Tag this image on our registry and push it :

docker pull busybox
docker tag busybox 172.30.234.98:5000/florian/myimage
# login to the registry and push the image
docker login -u florian -e foo@foo.com -p "<token>" 172.30.234.98:5000
docker push 172.30.234.98:5000/florian/myimage
9e77fef7a1c9: Pushing [==================================================>] 1.312 MB
9e77fef7a1c9: Pushed
latest: digest: sha256:33aac74e7f5782778e82d825ae4bfbe346aec2a3467fbd2a98a5091bce9fe46f size: 3214

docker push

If your configuration is ok, you should be able to find new directory and data in the s3 bucket

s3cmd ls s3://openpaas-blue-docker-registry/registry/docker/registry/v2/repositories/florian/myimage/_layers/sha256/
DIR s3://openpaas-blue-docker-registry/registry/docker/registry/v2/repositories/florian/myimage/_layers/sha256/2c84284818d186d88a16ac7fa731d4b71ba69ecfe11b4ce00413366833cb2403/
DIR s3://openpaas-blue-docker-registry/registry/docker/registry/v2/repositories/florian/myimage/_layers/sha256/5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef/

s3cmd ls

Troubleshooting

The first error we faced was a GO stack trace :

301 response missing Location header

It was due to a to a miss config of the s3 bucket region. The output message is not very user friendly. If you have some stack trace start to verify s3 bucket name, region and credentials.

https://github.com/mitchellh/goamz/issues/51

tips : To ensure the S3 registry configuration applied to the registry is correct :

Ansible generate config file from this template (https://github.com/openshift/openshift-ansible/blob/master/playbooks/adhoc/s3_registry/s3_registry.j2). You can find this generated file in /root/config.yml during the execution of the playbook but also directly inside the registry container.

Just go into the registry container and verify the config file given at the docker registry daemon.

# oc rsh <registry container name>
ps faux | grep docker
1000000+ 1 0.0 0.3 441380 26944 ? Ssl 08:28 0:00 /usr/bin/dockerregistry /etc/registryconfig/config.yml

oc rsh

Example of our config file :

cat /etc/registryconfig/config.yml
version: 0.1
log:
  level: debug
http:
  addr: :5000
storage:
  cache:
    layerinfo: inmemory
  s3:
    accesskey: ...
    secretkey: ...
    region: eu-west-1
    bucket: openpaas-blue-docker-registry
    encrypt: true
    secure: true
    v4auth: true
    rootdirectory: /registry
auth:
  openshift:
    realm: openshift
middleware:
  repository:
    - name: openshift

registry configuration

That’s it. Hoping this post can help someone.

gaelL

aws, docker, openshift3, s3, troubleshooting

Openshift 3 – Create my custom router/haproxy Continuous integration for Ansible

3 thoughts on “Openshift registry on Amazon S3”

Dean Peterson says:

February 26, 2016 at 18:12

I am trying to follow your tutorial here: https://readme.fr/openshift-registry-on-amazon-s3/

What/where do I get my clusterid? I know there is an ansible hosts file but nowhere in ansible’s or openshift 3’s documentation do they mention clusterid or how to get it.

Reply
- gaelL says:
  
  February 29, 2016 at 16:12
  
  Hello,
  
  The clusterid is related to ec2 instance tag.
  
  The hosts syntax use tags on aws instances. This feature is provided by ansible dynamic dynamic ec2 (http://docs.ansible.com/ansible/intro_dynamic_inventory.html#example-aws-ec2-external-inventory-script).
  
  See the ansible openshift inventory here for aws openshift cluster : https://github.com/openshift/openshift-ansible/tree/master/inventory/aws/hosts
  
  If you haven’t the clusterid tag on your ec2 instance, you need to edit this playbook.
  
  BTW, All tasks in this playbook can run only on one of the master. This playbook is pretty short and simple if you want to run manually each commands to test it yourself.
  
  Reply
AD says:

March 1, 2017 at 17:57

Hello,

Thanks for this nice article, it really clarifies a lot already about the usage of S3 bucket with OC docker registry:

I’m unfortunately have issues to get this to work properly: first of all this the

content of my config.yml file:

version: 0.1
log:
level: debug
http:
addr: :5000
storage:
cache:
layerinfo: inmemory
s3:
accesskey: awsaccesskey
secretkey: awssecretkey
region: eu-central-1
bucket: my-docker-registry-int
encrypt: true
secure: true
v4auth: true
rootdirectory: /registry
auth:
openshift:
realm: openshift
middleware:
repository:
– name: openshift

I didn’t use the Ansible playbooks, but run the OC commands manually instead, so

when running the docker push command i have the following :

38ac8d0f5bb3: Retrying in 5 seconds
38ac8d0f5bb3: Retrying in 1 second
received unexpected HTTP status: 500 Internal Server Error

checking the oc logs dc/docker-registry logs i see the following :

time=”2017-03-01T15:27:50.691756871Z” level=debug msg=”authorizing request” go.version=go1.7.4 http.request.host=”:5000″ http.request.id=a7bab7f5-aed5-4d66-b6d8-8438a46dd614 http.request.method=POST http.request.remoteaddr=”:36880″ http.request.uri=”/v2/admin/myimage/blobs/uploads/” http.request.useragent=”docker/1.12.5 go/go1.7.4 kernel/3.10.0-514.6.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.5 \\(linux\\))” instance.id=3f168da6-2761-4a86-9f89-43a2aa655a2a vars.name=”admin/myimage”
time=”2017-03-01T15:27:50.691879398Z” level=debug msg=”Origin auth: checking for access to repository:admin/myimage:pull” go.version=go1.7.4 http.request.host=”:5000″ http.request.id=a7bab7f5-aed5-4d66-b6d8-8438a46dd614 http.request.method=POST http.request.remoteaddr=”:36880″ http.request.uri=”/v2/admin/myimage/blobs/uploads/” http.request.useragent=”docker/1.12.5 go/go1.7.4 kernel/3.10.0-514.6.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.5 \\(linux\\))” instance.id=3f168da6-2761-4a86-9f89-43a2aa655a2a vars.name=”admin/myimage”
time=”2017-03-01T15:27:50.694995684Z” level=debug msg=”Origin auth: checking for access to repository:admin/myimage:push” go.version=go1.7.4 http.request.host=”172.30.22.114:5000″ http.request.id=a7bab7f5-aed5-4d66-b6d8-8438a46dd614 http.request.method=POST http.request.remoteaddr=”:36880″ http.request.uri=”/v2/admin/myimage/blobs/uploads/” http.request.useragent=”docker/1.12.5 go/go1.7.4 kernel/3.10.0-514.6.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.5 \\(linux\\))” instance.id=3f168da6-2761-4a86-9f89-43a2aa655a2a vars.name=”admin/myimage”
time=”2017-03-01T15:27:50.697532384Z” level=debug msg=”(*linkedBlobStore).Writer” go.version=go1.7.4 http.request.host=”:5000″ http.request.id=a7bab7f5-aed5-4d66-b6d8-8438a46dd614 http.request.method=POST http.request.remoteaddr=”:36880″ http.request.uri=”/v2/admin/myimage/blobs/uploads/” http.request.useragent=”docker/1.12.5 go/go1.7.4 kernel/3.10.0-514.6.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.5 \\(linux\\))” instance.id=3f168da6-2761-4a86-9f89-43a2aa655a2a vars.name=”admin/myimage”
time=”2017-03-01T15:27:50.716044855Z” level=debug msg=”s3aws.PutContent(\”/docker/registry/v2/repositories/admin/myimage/_uploads/d1643410-e510-4086-a544-fa5ca35b9a15/startedat\”)” go.version=go1.7.4 http.request.host=”:5000″ http.request.id=a7bab7f5-aed5-4d66-b6d8-8438a46dd614 http.request.method=POST http.request.remoteaddr=”10.128.0.1:36880″ http.request.uri=”/v2/admin/myimage/blobs/uploads/” http.request.useragent=”docker/1.12.5 go/go1.7.4 kernel/3.10.0-514.6.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.5 \\(linux\\))” instance.id=3f168da6-2761-4a86-9f89-43a2aa655a2a trace.duration=18.430328ms trace.file=”/builddir/build/BUILD/atomic-openshift-git-0.690b5d4/_output/local/go/src/github.com/openshift/origin/vendor/github.com/docker/distribution/registry/storage/driver/base/base.go” trace.func=”github.com/openshift/origin/vendor/github.com/docker/distribution/registry/storage/driver/base.(*Base).PutContent” trace.id=75204cd9-6000-4366-9c6a-4718e8830477 trace.line=95 vars.name=”admin/myimage”
time=”2017-03-01T15:27:50.716146688Z” level=error msg=”response completed with error” err.code=unknown err.detail=”s3aws: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.\n\tstatus code: 403, request id: ” err.message=”unknown error” go.version=go1.7.4 http.request.host=”:5000″ http.request.id=a7bab7f5-aed5-4d66-b6d8-8438a46dd614 http.request.method=POST http.request.remoteaddr=”:36880″ http.request.uri=”/v2/admin/myimage/blobs/uploads/” http.request.useragent=”docker/1.12.5 go/go1.7.4 kernel/3.10.0-514.6.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.5 \\(linux\\))” http.response.contenttype=”application/json; charset=utf-8″ http.response.duration=25.603727ms http.response.status=500 http.response.written=117 instance.id=3f168da6-2761-4a86-9f89-43a2aa655a2a vars.name=”admin/myimage”
10.128.0.1 – – [01/Mar/2017:15:27:50 +0000] “POST /v2/admin/myimage/blobs/uploads/ HTTP/1.1” 500 117 “” “docker/1.12.5 go/go1.7.4 kernel/3.10.0-514.6.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.5 \\(linux\\))”
10.129.0.1 – – [01/Mar/2017:15:27:50 +0000] “GET /healthz HTTP/1.1” 200 0 “” “Go-http-client/1.1”
10.129.0.1 – – [01/Mar/2017:15:27:50 +0000] “GET /healthz HTTP/1.1” 200 0 “” “Go-http-client/1.1”
10.129.0.1 – – [01/Mar/2017:15:28:00 +0000] “GET /healthz HTTP/1.1” 200 0 “” “Go-http-client/1.1”
10.129.0.1 – – [01/Mar/2017:15:28:00 +0000] “GET /healthz HTTP/1.1” 200 0 “” “Go-http-client/1.1”
10.129.0.1 – – [01/Mar/2017:15:28:10 +0000] “GET /healthz HTTP/1.1” 200 0 “” “Go-http-client/1.1”
10.129.0.1 – – [01/Mar/2017:15:28:10 +0000] “GET /healthz HTTP/1.1” 200 0 “” “Go-http-client/1.1”
10.129.0.1 – – [01/Mar/2017:15:28:20 +0000] “GET /healthz HTTP/1.1” 200 0 “” “Go-http-client/1.1”
10.129.0.1 – – [01/Mar/2017:15:28:20 +0000] “GET /healthz HTTP/1.1” 200 0 “” “Go-http-client/1.1”
10.129.0.1 – – [01/Mar/2017:15:28:30 +0000] “GET /healthz HTTP/1.1” 200 0 “” “Go-http-client/1.1”
10.129.0.1 – – [01/Mar/2017:15:28:30 +0000] “GET /healthz HTTP/1.1” 200 0 “” “Go-http-client/1.1”
10.129.0.1 – – [01/Mar/2017:15:28:40 +0000] “GET /healthz HTTP/1.1” 200 0 “” “Go-http-client/1.1”
10.129.0.1 – – [01/Mar/2017:15:28:40 +0000] “GET /healthz HTTP/1.1” 200 0 “” “Go-http-client/1.1”
10.129.0.1 – – [01/Mar/2017:15:28:50 +0000] “GET /healthz HTTP/1.1” 200 0 “” “Go-http-client/1.1”

Help or hints to solve the issue will be very appreciated.

Thanks in Advance

AD

Reply

Openshift registry on Amazon S3

Create the registry

Test the registry on S3

Troubleshooting

3 thoughts on “Openshift registry on Amazon S3”

Leave a Reply Cancel reply