K8S笔记之私有仓库搭建


创建PV

关于PV的创建参考 《K8S笔记之持久化存储》

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: docker-claim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 80Gi
  storageClassName: local-volume

搭建仓库

Docker 私有化仓库的搭建可参考官方文档, 并且将相关的配置转化为K8S所需要的yaml文件

---
apiVersion: v1
kind: ReplicationController
metadata:
  name: registry
  namespace: default
  labels:
    k8s-app: registry
spec:
  replicas: 1
  selector:
    k8s-app: registry
  template:
    metadata:
      labels:
        k8s-app: registry
    spec:
      containers:
        - name: registry
          image: registry:2
          ports:
            - containerPort: 5000
          env:
            - name: REGISTRY_HTTP_ADDR
              value: :5000
            - name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
              value: /var/lib/registry
            - name: REGISTRY_HTTP_TLS_CERTIFICATE
              value: /certs/tls.crt
            - name: REGISTRY_HTTP_TLS_KEY
              value: /certs/tls.key
            - name: REGISTRY_AUTH
              value: htpasswd
            - name: REGISTRY_AUTH_HTPASSWD_REALM
              value: Registry Realm
            - name: REGISTRY_AUTH_HTPASSWD_PATH
              value: /auth/htpasswd
          volumeMounts:
            - name: registry-storage
              mountPath: /var/lib/registry
              subPath: registry
            - name: registry-secret
              mountPath: /certs
            - name: registry-passwd
              mountPath: /auth
      volumes:
        - name: registry-storage
          persistentVolumeClaim:
            claimName: docker-claim
        - name: registry-secret
          secret:
            secretName: registry-tls-secret
        - name: registry-passwd
          secret:
            secretName: registry-passwd-secret

需要注意的几个点

命名空间

这里我指定的命名空间为 default, 包括前面的PV以及后续所创建的 service, secret, ingress等服务也需要指定相同的命名空间

https验证

https验证所需要的ca文件命名为tls.crt, tls.key是为了后面可以方便的配置 ingress

  • 创建ca证书与私钥

    # openssl req -newkey rsa:4096 -nodes -sha256 -keyout certs/tls.key -x509 -days 365 -out certs/tls.crt
    Generating a 4096 bit RSA private key
    .......................................................................................................++
    ................................................++
    writing new private key to 'certs/tls.key'
    -----
    You are about to be asked to enter information that will be incorporated
    into your certificate request.
    What you are about to enter is what is called a Distinguished Name or a DN.
    There are quite a few fields but you can leave some blank
    For some fields there will be a default value,
    If you enter '.', the field will be left blank.
    -----
    Country Name (2 letter code) [XX]:CN
    State or Province Name (full name) []:sichuan
    Locality Name (eg, city) [Default City]:chengdu
    Organization Name (eg, company) [Default Company Ltd]:
    Organizational Unit Name (eg, section) []:software
    Common Name (eg, your name or your server's hostname) []:registry.k8s.cloud
    Email Address []:registry@xxx.com
    

    需要注意的是 Common Name 字段, 后续配置 ingresshost 要与该字段保持一致

  • 创建 secret

    kubectl --namespace=default create secret tls registry-tls-secret --cert ./certs/tls.crt --key ./certs/tls.key
    

    该密钥创建后会生成 data 下的两个文件,一个 tls.crt, 一个 tls.key, 这两个名字总是固定的,与执行命令中的文件名无关

用户名密码验证

Docker仓库默认未开启账号验证,需要添加三个环境变量

- name: REGISTRY_AUTH
  value: htpasswd
- name: REGISTRY_AUTH_HTPASSWD_REALM
  value: Registry Realm
- name: REGISTRY_AUTH_HTPASSWD_PATH
  value: /auth/htpasswd

其中 /auth/htpasswd 将会被挂载在 secret

  • 生成密码文件

    docker run --entrypoint htpasswd registry:2 -Bbn username password > certs/htpasswd
    

    或者

    yum install httpd
    htpasswd -Bbn user passwd > certs/passwd
    
  • 创建 secret

    kubectl --namespace=default create secret generic registry-passwd-secret --from-file=htpasswd=./certs/passwd
    

创建service

---
apiVersion: v1
kind: Service
metadata:
  name: registry
  namespace: default
  labels:
    k8s-app: registry
    kubernetes.io/name: "KubeRegistry"
spec:
  selector:
    k8s-app: registry
  ports:
    - port: 5000
      protocol: TCP

注: 我也不知道 label 中的 kubernetes.io/name 有什么作用

创建ingress

由于该私有仓库需要对外服务, 我采用 ingress 方式

---
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  name: ingress.registry
  namespace: default
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
spec:
  tls:
    - hosts:
        - registry.k8s.cloud
      secretName: registry-tls-secret
  rules:
    - host: registry.k8s.cloud
      http:
        paths:
          - path: /
            backend:
              serviceName: registry
              servicePort: 5000
  • 命名空间必须与之前保持一致

  • host 必须与之前创建证书时的 CN 保持一致

测试Docker仓库

修改host

由于 registry.k8s.cloud 并不是一个真实的域名,所以采用修改 /etc/hosts 方式进行测试和访问

echo "10.8.235.202 registry.k8s.cloud" >> /etc/hosts

更新Docker证书

mkdir -p /etc/docker/certs.d/registry.k8s.cloud
cp certs/tls.crt /etc/docker/certs.d/registry.k8s.cloud/ca.crt

如果是不同机器,复制 certs/tls.crt 的内容到另一台机器上即可

注:不需要重启Docker

登陆仓库

docker login registry.k8s.cloud

输入之前创建时用的用户名密码, 一般就可以登陆成功

# docker login registry.k8s.cloud
Username: xxxx
Password: 
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

测试上传与下载

准备任意 Dockerfile

  • 构建本地镜像

    docker build -t registry.k8s.cloud/mytest .
    
  • 上传镜像

    # docker push registry.k8s.cloud/mytest:latest
    The push refers to repository [registry.k8s.cloud/mytest]
    1357220cb89: Layer already exists
    a965e1c532c: Pushed
    beaee123df5: Layer already exists
    2568a15e513: Layer already exists
    badc4d6e828: Pushed
    3c730fdba93: Pushed
    9d9e1e59db0: Pushed
    9dfa0a0da3b: Pushed
    latest: digest: sha256:c20c32ba413e6555159aa0722a866a576481fe4a177116c4919xxxxxxx2a5a size: 1995
    
  • 拉取镜像 当镜像上传成功,可在另一台机器上测试下载

    # docker pull registry.k8s.cloud/mytest:latest
    latest: Pulling from mytest
    605cebd3f31: Pull complete
    d08402297b9: Pull complete
    68c3b3bcbd0: Pull complete
    9ee925a87cc: Pull complete
    ebb13422b6b: Pull complete
    59c8302ac13: Pull complete
    7149c344f10: Pull complete
    7d8abdd4c32: Pull complete
    Digest: sha256:c20c32ba1e6555159aa0722a866a576481fe4a177116419ade0708652a5a
    Status: Downloaded newer image for registry.k8s.cloud/mytest:latest
    registry.k8s.cloud/mytest:latest
    

FAQ

x509: certificate is valid for ingress.local, not registry.k8s.cloud

Error response from daemon: Get https://registry.k8s.cloud/v2/: x509: certificate is valid for ingress.local, not registry.k8s.cloud

这一般是 ingress tls 的配置问题, 比如创建 secret 时采用 generic 而不是 tls

x509: certificate signed by unknown authority

Error response from daemon: Get https://registry.k8s.cloud/v2/: x509: certificate signed by unknown authority

这是未更新Docker证书导致的, 需要更新

/etc/docker/certs.d/registry.k8s.cloud/ca.crt

413 Request Entity Too Large

error parsing HTTP 413 response body: invalid character '<' looking for beginning of value: "<html>\r\n<head><title>413 Request Entity Too Large</title></head>\r\n<body>\r\n<center><h1>413 Request Entity Too Large</h1></center>\r\n<hr><center>nginx/1.17.7</center>\r\n</body>\r\n</html>\r\n"

当上传镜像时遇到413, 需要更新 ingress 配置

nginx.ingress.kubernetes.io/proxy-body-size: 1024m

参考资料

  • https://docs.docker.com/registry/deploying/

  • http://www.huamo.online/2017/06/07/Kubernetes%E6%90%AD%E5%BB%BATLS%E7%A7%81%E6%9C%89docker%E4%BB%93%E5%BA%93/

作者: honmaple
链接: https://honmaple.me/articles/2020/02/K8S笔记之私有仓库搭建.html
版权: 知识共享署名-非商业性使用-相同方式共享4.0国际许可协议
wechat