K8S笔记之私有仓库搭建

2020年02月20日周四 | By honmaple | In Tech | 0f linux k8s

创建PV
搭建仓库
创建service
创建ingress
测试Docker仓库
FAQ
参考资料

创建PV

关于PV的创建参考《K8S笔记之持久化存储》

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: docker-claim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 80Gi
  storageClassName: local-volume

搭建仓库

Docker 私有化仓库的搭建可参考官方文档, 并且将相关的配置转化为K8S所需要的yaml文件

---
apiVersion: v1
kind: ReplicationController
metadata:
  name: registry
  namespace: default
  labels:
    k8s-app: registry
spec:
  replicas: 1
  selector:
    k8s-app: registry
  template:
    metadata:
      labels:
        k8s-app: registry
    spec:
      containers:
        - name: registry
          image: registry:2
          ports:
            - containerPort: 5000
          env:
            - name: REGISTRY_HTTP_ADDR
              value: :5000
            - name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
              value: /var/lib/registry
            - name: REGISTRY_HTTP_TLS_CERTIFICATE
              value: /certs/tls.crt
            - name: REGISTRY_HTTP_TLS_KEY
              value: /certs/tls.key
            - name: REGISTRY_AUTH
              value: htpasswd
            - name: REGISTRY_AUTH_HTPASSWD_REALM
              value: Registry Realm
            - name: REGISTRY_AUTH_HTPASSWD_PATH
              value: /auth/htpasswd
          volumeMounts:
            - name: registry-storage
              mountPath: /var/lib/registry
              subPath: registry
            - name: registry-secret
              mountPath: /certs
            - name: registry-passwd
              mountPath: /auth
      volumes:
        - name: registry-storage
          persistentVolumeClaim:
            claimName: docker-claim
        - name: registry-secret
          secret:
            secretName: registry-tls-secret
        - name: registry-passwd
          secret:
            secretName: registry-passwd-secret

需要注意的几个点

命名空间

这里我指定的命名空间为 default，包括前面的PV以及后续所创建的 service, secret, ingress等服务也需要指定相同的命名空间

https验证

https验证所需要的ca文件命名为tls.crt, tls.key是为了后面可以方便的配置 ingress

创建ca证书与私钥

# openssl req -newkey rsa:4096 -nodes -sha256 -keyout certs/tls.key -x509 -days 365 -out certs/tls.crt
Generating a 4096 bit RSA private key
.......................................................................................................++
................................................++
writing new private key to 'certs/tls.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:sichuan
Locality Name (eg, city) [Default City]:chengdu
Organization Name (eg, company) [Default Company Ltd]:
Organizational Unit Name (eg, section) []:software
Common Name (eg, your name or your server's hostname) []:registry.k8s.cloud
Email Address []:[email protected]

需要注意的是 Common Name 字段, 后续配置 ingress 的 host 要与该字段保持一致

创建 secret
```
kubectl --namespace=default create secret tls registry-tls-secret --cert ./certs/tls.crt --key ./certs/tls.key
```
该密钥创建后会生成 data 下的两个文件，一个 tls.crt, 一个 tls.key, 这两个名字总是固定的，与执行命令中的文件名无关

用户名密码验证

Docker仓库默认未开启账号验证，需要添加三个环境变量

- name: REGISTRY_AUTH
  value: htpasswd
- name: REGISTRY_AUTH_HTPASSWD_REALM
  value: Registry Realm
- name: REGISTRY_AUTH_HTPASSWD_PATH
  value: /auth/htpasswd

其中 /auth/htpasswd 将会被挂载在 secret 上

生成密码文件

docker run --entrypoint htpasswd registry:2 -Bbn username password > certs/htpasswd

或者

yum install httpd
htpasswd -Bbn user passwd > certs/passwd

创建 secret

kubectl --namespace=default create secret generic registry-passwd-secret --from-file=htpasswd=./certs/passwd

创建service

---
apiVersion: v1
kind: Service
metadata:
  name: registry
  namespace: default
  labels:
    k8s-app: registry
    kubernetes.io/name: "KubeRegistry"
spec:
  selector:
    k8s-app: registry
  ports:
    - port: 5000
      protocol: TCP

注: 我也不知道 label 中的 kubernetes.io/name 有什么作用

创建ingress

由于该私有仓库需要对外服务, 我采用 ingress 方式

---
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  name: ingress.registry
  namespace: default
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
spec:
  tls:
    - hosts:
        - registry.k8s.cloud
      secretName: registry-tls-secret
  rules:
    - host: registry.k8s.cloud
      http:
        paths:
          - path: /
            backend:
              serviceName: registry
              servicePort: 5000

命名空间必须与之前保持一致
host 必须与之前创建证书时的 CN 保持一致

测试Docker仓库

修改host

由于 registry.k8s.cloud 并不是一个真实的域名，所以采用修改 /etc/hosts 方式进行测试和访问

echo "10.8.235.202 registry.k8s.cloud" >> /etc/hosts

更新Docker证书

mkdir -p /etc/docker/certs.d/registry.k8s.cloud
cp certs/tls.crt /etc/docker/certs.d/registry.k8s.cloud/ca.crt

如果是不同机器，复制 certs/tls.crt 的内容到另一台机器上即可

注：不需要重启Docker

登陆仓库

docker login registry.k8s.cloud

输入之前创建时用的用户名，密码, 一般就可以登陆成功

# docker login registry.k8s.cloud
Username: xxxx
Password: 
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

测试上传与下载

准备任意 Dockerfile

构建本地镜像

docker build -t registry.k8s.cloud/mytest .

上传镜像

# docker push registry.k8s.cloud/mytest:latest
The push refers to repository [registry.k8s.cloud/mytest]
1357220cb89: Layer already exists
a965e1c532c: Pushed
beaee123df5: Layer already exists
2568a15e513: Layer already exists
badc4d6e828: Pushed
3c730fdba93: Pushed
9d9e1e59db0: Pushed
9dfa0a0da3b: Pushed
latest: digest: sha256:c20c32ba413e6555159aa0722a866a576481fe4a177116c4919xxxxxxx2a5a size: 1995

拉取镜像当镜像上传成功，可在另一台机器上测试下载

# docker pull registry.k8s.cloud/mytest:latest
latest: Pulling from mytest
605cebd3f31: Pull complete
d08402297b9: Pull complete
68c3b3bcbd0: Pull complete
9ee925a87cc: Pull complete
ebb13422b6b: Pull complete
59c8302ac13: Pull complete
7149c344f10: Pull complete
7d8abdd4c32: Pull complete
Digest: sha256:c20c32ba1e6555159aa0722a866a576481fe4a177116419ade0708652a5a
Status: Downloaded newer image for registry.k8s.cloud/mytest:latest
registry.k8s.cloud/mytest:latest

FAQ

x509: certificate is valid for ingress.local, not registry.k8s.cloud

Error response from daemon: Get https://registry.k8s.cloud/v2/: x509: certificate is valid for ingress.local, not registry.k8s.cloud

这一般是 ingress tls 的配置问题, 比如创建 secret 时采用 generic 而不是 tls

x509: certificate signed by unknown authority

Error response from daemon: Get https://registry.k8s.cloud/v2/: x509: certificate signed by unknown authority

这是未更新Docker证书导致的, 需要更新

/etc/docker/certs.d/registry.k8s.cloud/ca.crt

413 Request Entity Too Large

error parsing HTTP 413 response body: invalid character '<' looking for beginning of value: "<html>\r\n<head><title>413 Request Entity Too Large</title></head>\r\n<body>\r\n<center><h1>413 Request Entity Too Large</h1></center>\r\n<hr><center>nginx/1.17.7</center>\r\n</body>\r\n</html>\r\n"

当上传镜像时遇到413, 需要更新 ingress 配置

nginx.ingress.kubernetes.io/proxy-body-size: 1024m

参考资料

作者: honmaple

链接: https://honmaple.me/articles/2020/02/K8S笔记之私有仓库搭建.html

版权:

知识共享署名-非商业性使用-相同方式共享4.0国际许可协议

加载评论