创建PV

关于PV的创建参考 《K8S笔记之持久化存储》

 1apiVersion: v1
 2kind: PersistentVolumeClaim
 3metadata:
 4  name: docker-claim
 5spec:
 6  accessModes:
 7    - ReadWriteOnce
 8  resources:
 9    requests:
10      storage: 80Gi
11  storageClassName: local-volume

搭建仓库

Docker 私有化仓库的搭建可参考官方文档, 并且将相关的配置转化为K8S所需要的yaml文件

 1---
 2apiVersion: v1
 3kind: ReplicationController
 4metadata:
 5  name: registry
 6  namespace: default
 7  labels:
 8    k8s-app: registry
 9spec:
10  replicas: 1
11  selector:
12    k8s-app: registry
13  template:
14    metadata:
15      labels:
16        k8s-app: registry
17    spec:
18      containers:
19        - name: registry
20          image: registry:2
21          ports:
22            - containerPort: 5000
23          env:
24            - name: REGISTRY_HTTP_ADDR
25              value: :5000
26            - name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
27              value: /var/lib/registry
28            - name: REGISTRY_HTTP_TLS_CERTIFICATE
29              value: /certs/tls.crt
30            - name: REGISTRY_HTTP_TLS_KEY
31              value: /certs/tls.key
32            - name: REGISTRY_AUTH
33              value: htpasswd
34            - name: REGISTRY_AUTH_HTPASSWD_REALM
35              value: Registry Realm
36            - name: REGISTRY_AUTH_HTPASSWD_PATH
37              value: /auth/htpasswd
38          volumeMounts:
39            - name: registry-storage
40              mountPath: /var/lib/registry
41              subPath: registry
42            - name: registry-secret
43              mountPath: /certs
44            - name: registry-passwd
45              mountPath: /auth
46      volumes:
47        - name: registry-storage
48          persistentVolumeClaim:
49            claimName: docker-claim
50        - name: registry-secret
51          secret:
52            secretName: registry-tls-secret
53        - name: registry-passwd
54          secret:
55            secretName: registry-passwd-secret

需要注意的几个点

命名空间

这里我指定的命名空间为 default， 包括前面的PV以及后续所创建的 service, secret, ingress等服务也需要指定相同的命名空间

https验证

https验证所需要的ca文件命名为tls.crt, tls.key是为了后面可以方便的配置 ingress

• 创建ca证书与私钥

   1# openssl req -newkey rsa:4096 -nodes -sha256 -keyout certs/tls.key -x509 -days 365 -out certs/tls.crt
   2Generating a 4096 bit RSA private key
   3.......................................................................................................++
   4................................................++
   5writing new private key to 'certs/tls.key'
   6-----
   7You are about to be asked to enter information that will be incorporated
   8into your certificate request.
   9What you are about to enter is what is called a Distinguished Name or a DN.
  10There are quite a few fields but you can leave some blank
  11For some fields there will be a default value,
  12If you enter '.', the field will be left blank.
  13-----
  14Country Name (2 letter code) [XX]:CN
  15State or Province Name (full name) []:sichuan
  16Locality Name (eg, city) [Default City]:chengdu
  17Organization Name (eg, company) [Default Company Ltd]:
  18Organizational Unit Name (eg, section) []:software
  19Common Name (eg, your name or your server's hostname) []:registry.k8s.cloud
  20Email Address []:[email protected]

  需要注意的是 Common Name 字段, 后续配置 ingress 的 host 要与该字段保持一致

• 创建 secret

  1kubectl --namespace=default create secret tls registry-tls-secret --cert ./certs/tls.crt --key ./certs/tls.key

  该密钥创建后会生成 data 下的两个文件，一个 tls.crt, 一个 tls.key, 这两个名字总是固定的，与执行命令中的文件名无关

用户名密码验证

Docker仓库默认未开启账号验证，需要添加三个环境变量

1- name: REGISTRY_AUTH
2  value: htpasswd
3- name: REGISTRY_AUTH_HTPASSWD_REALM
4  value: Registry Realm
5- name: REGISTRY_AUTH_HTPASSWD_PATH
6  value: /auth/htpasswd

其中 /auth/htpasswd 将会被挂载在 secret 上

• 生成密码文件

  1docker run --entrypoint htpasswd registry:2 -Bbn username password > certs/htpasswd

  或者

  1yum install httpd
  2htpasswd -Bbn user passwd > certs/passwd

• 创建 secret

  1kubectl --namespace=default create secret generic registry-passwd-secret --from-file=htpasswd=./certs/passwd

创建service

 1---
 2apiVersion: v1
 3kind: Service
 4metadata:
 5  name: registry
 6  namespace: default
 7  labels:
 8    k8s-app: registry
 9    kubernetes.io/name: "KubeRegistry"
10spec:
11  selector:
12    k8s-app: registry
13  ports:
14    - port: 5000
15      protocol: TCP

注: 我也不知道 label 中的 kubernetes.io/name 有什么作用

创建ingress

由于该私有仓库需要对外服务, 我采用 ingress 方式

 1---
 2kind: Ingress
 3apiVersion: extensions/v1beta1
 4metadata:
 5  name: ingress.registry
 6  namespace: default
 7  annotations:
 8    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
 9    nginx.ingress.kubernetes.io/ingress.class: nginx
10    nginx.ingress.kubernetes.io/rewrite-target: /
11    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
12spec:
13  tls:
14    - hosts:
15        - registry.k8s.cloud
16      secretName: registry-tls-secret
17  rules:
18    - host: registry.k8s.cloud
19      http:
20        paths:
21          - path: /
22            backend:
23              serviceName: registry
24              servicePort: 5000

• 命名空间必须与之前保持一致

• host 必须与之前创建证书时的 CN 保持一致

测试Docker仓库

修改host

由于 registry.k8s.cloud 并不是一个真实的域名，所以采用修改 /etc/hosts 方式进行测试和访问

1echo "10.8.235.202 registry.k8s.cloud" >> /etc/hosts

更新Docker证书

1mkdir -p /etc/docker/certs.d/registry.k8s.cloud
2cp certs/tls.crt /etc/docker/certs.d/registry.k8s.cloud/ca.crt

如果是不同机器，复制 certs/tls.crt 的内容到另一台机器上即可

注：不需要重启Docker

登陆仓库

1docker login registry.k8s.cloud

输入之前创建时用的用户名，密码, 一般就可以登陆成功

1# docker login registry.k8s.cloud
2Username: xxxx
3Password: 
4WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
5Configure a credential helper to remove this warning. See
6https://docs.docker.com/engine/reference/commandline/login/#credentials-store
7
8Login Succeeded

测试上传与下载

准备任意 Dockerfile

• 构建本地镜像

  1docker build -t registry.k8s.cloud/mytest .

• 上传镜像

   1# docker push registry.k8s.cloud/mytest:latest
   2The push refers to repository [registry.k8s.cloud/mytest]
   31357220cb89: Layer already exists
   4a965e1c532c: Pushed
   5beaee123df5: Layer already exists
   62568a15e513: Layer already exists
   7badc4d6e828: Pushed
   83c730fdba93: Pushed
   99d9e1e59db0: Pushed
  109dfa0a0da3b: Pushed
  11latest: digest: sha256:c20c32ba413e6555159aa0722a866a576481fe4a177116c4919xxxxxxx2a5a size: 1995

• 拉取镜像 当镜像上传成功，可在另一台机器上测试下载

   1# docker pull registry.k8s.cloud/mytest:latest
   2latest: Pulling from mytest
   3605cebd3f31: Pull complete
   4d08402297b9: Pull complete
   568c3b3bcbd0: Pull complete
   69ee925a87cc: Pull complete
   7ebb13422b6b: Pull complete
   859c8302ac13: Pull complete
   97149c344f10: Pull complete
  107d8abdd4c32: Pull complete
  11Digest: sha256:c20c32ba1e6555159aa0722a866a576481fe4a177116419ade0708652a5a
  12Status: Downloaded newer image for registry.k8s.cloud/mytest:latest
  13registry.k8s.cloud/mytest:latest

FAQ

x509: certificate is valid for ingress.local, not registry.k8s.cloud

1Error response from daemon: Get https://registry.k8s.cloud/v2/: x509: certificate is valid for ingress.local, not registry.k8s.cloud

这一般是 ingress tls 的配置问题, 比如创建 secret 时采用 generic 而不是 tls

x509: certificate signed by unknown authority

1Error response from daemon: Get https://registry.k8s.cloud/v2/: x509: certificate signed by unknown authority

这是未更新Docker证书导致的, 需要更新

1/etc/docker/certs.d/registry.k8s.cloud/ca.crt

413 Request Entity Too Large

1error parsing HTTP 413 response body: invalid character '<' looking for beginning of value: "<html>\r\n<head><title>413 Request Entity Too Large</title></head>\r\n<body>\r\n<center><h1>413 Request Entity Too Large</h1></center>\r\n<hr><center>nginx/1.17.7</center>\r\n</body>\r\n</html>\r\n"

当上传镜像时遇到413, 需要更新 ingress 配置

1nginx.ingress.kubernetes.io/proxy-body-size: 1024m

参考资料

• https://docs.docker.com/registry/deploying/

• http://www.huamo.online/2017/06/07/Kubernetes%E6%90%AD%E5%BB%BATLS%E7%A7%81%E6%9C%89docker%E4%BB%93%E5%BA%93/