前言
最近由于公司业务发展到了瓶颈，原有的技术架构已经逐渐无法满足业务开发和测试的需求，出现了应用测试环境搭建复杂，有许多套（真的很多很多）应用环境，应用在持续集成/持续交付也遇到了很大的困难，经过讨论研究决定对应用和微服务进行容器化，这就是我首次直面docker和k8s的契机（好吧，我是菜鸟）
Kubernetes 介绍
Kubernetes 是 Google 团队发起的开源项目，它的目标是管理跨多个主机的容器，提供基本的部署，维护以及运用伸缩，主要实现语言为
Go 语言。Kubernetes的特点：
- 易学：轻量级，简单，容易理解
- 便携：支持公有云，私有云，混合云，以及多种云平台
- 可拓展：模块化，可插拔，支持钩子，可任意组合
- 自修复：自动重调度，自动重启，自动复制
好吧，这是从别人ppt上拷下来的=。=
下面就正式开始部署我们自己的k8s吧准备工作
注：以下操作都是在root权限下执行的
- 安装docker-ce，这里使用docker-ce-17.09.0.c版本，安装方法见
-
安装Kubeadm
安装 Kubeadm 首先我们要配置好阿里云的国内源，执行如下命令：cat <
/etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=0 EOF 之后，执行以下命令来重建yum缓存：
yum -y install epel-releaseyum clean all yum makecache
接下来需要安装指定版本的Kubeadm（这里要安装指定版本，因为后续依赖的镜像由于有墙无法拉取，这里我们只有指定版本的镜像），注意：这里是安装指定版本的Kubeadm，k8s的版本更新之快完全超出你的想象！
yum -y install kubelet-1.11.0-0yum -y install kubeadm-1.11.0-0yum -y install kubectl-1.11.0-0yum -y install kubernetes-cni
执行命令启动Kubeadm服务：
systemctl enable kubelet && systemctl start kubelet
-
配置 Kubeadm 所用到的镜像
这里是重中之重，因为在国内的原因，无法访问到 Google 的镜像库，所以我们需要执行以下脚本来从 Docker Hub 仓库中获取相同的镜像，并且更改 TAG 让其变成与 Google 拉去镜像一致。 新建一个 Shell 脚本，填入以下代码之后保存。#docker.sh#!/bin/bashimages=(kube-proxy-amd64:v1.11.0 kube-scheduler-amd64:v1.11.0 kube-controller-manager-amd64:v1.11.0 kube-apiserver-amd64:v1.11.0 etcd-amd64:3.2.18 coredns:1.1.3 pause-amd64:3.1 kubernetes-dashboard-amd64:v1.8.3 k8s-dns-sidecar-amd64:1.14.9 k8s-dns-kube-dns-amd64:1.14.9 k8s-dns-dnsmasq-nanny-amd64:1.14.9 ) for imageName in ${images[@]} ; dodocker pull keveon/$imageNamedocker tag keveon/$imageName k8s.gcr.io/$imageNamedocker rmi keveon/$imageNamedone# 个人新加的一句，V 1.11.0 必加docker tag da86e6ba6ca1 k8s.gcr.io/pause:3.1保存后使用chmod命令赋予脚本执行权限
chmod -R 777 ./docker.sh
执行脚本拉取镜像
sh docker.sh#这里就开始了漫长的拉取镜像之路
关闭swap
sudo swapoff -a#要永久禁掉swap分区，打开如下文件注释掉swap那一行 # sudo vi /etc/stab
关闭selinux
# 临时禁用selinux# 永久关闭 修改/etc/sysconfig/selinux文件设置sed -i 's/SELINUX=permissive/SELINUX=disabled/' /etc/sysconfig/selinux # 这里按回车，下面是第二条命令setenforce 0
关闭防火墙
systemctl disable firewalld.service && systemctl stop firewalld.service
配置转发参数
# 配置转发相关参数，否则可能会出错 cat <
/etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1net.bridge.bridge-nf-call-iptables = 1vm.swappiness=0EOF # 这里按回车，下面是第二条命令 sysctl --system 这里就完成了k8s集群搭建的准备工作，集群搭建的话以上操作结束后将操作完的系统制作成系统镜像，方便集群搭建
正式安装
以下的操作都只在主节点上进行：
正式开始安装k8s
初始化镜像，执行以下命令：kubeadm init --kubernetes-version=v1.11.0 #这里是之前所安装K8S的版本号 --pod-network-cidr=10.10.0.0/16 #这里填写集群所在网段
之后的输出会是这样：
I0712 10:46:30.938979 13461 feature_gate.go:230] feature gates: &{map[]}[init] using Kubernetes version: v1.11.0[preflight] running pre-flight checksI0712 10:46:30.961005 13461 kernel_validator.go:81] Validating kernel versionI0712 10:46:30.961061 13461 kernel_validator.go:96] Validating kernel config [WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 18.03.1-ce. Max validated version: 17.03 [WARNING Hostname]: hostname "g2-apigateway" could not be reached [WARNING Hostname]: hostname "g2-apigateway" lookup g2-apigateway on 100.100.2.138:53: no such host[preflight/images] Pulling images required for setting up a Kubernetes cluster[preflight/images] This might take a minute or two, depending on the speed of your internet connection[preflight/images] You can also perform this action in beforehand using 'kubeadm config images pull'[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"[preflight] Activating the kubelet service[certificates] Generated ca certificate and key.[certificates] Generated apiserver certificate and key.[certificates] apiserver serving cert is signed for DNS names [g2-apigateway kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.16.8.62][certificates] Generated apiserver-kubelet-client certificate and key.[certificates] Generated sa key and public key.[certificates] Generated front-proxy-ca certificate and key.[certificates] Generated front-proxy-client certificate and key.[certificates] Generated etcd/ca certificate and key.[certificates] Generated etcd/server certificate and key.[certificates] etcd/server serving cert is signed for DNS names [g2-apigateway localhost] and IPs [127.0.0.1 ::1][certificates] Generated etcd/peer certificate and key.[certificates] etcd/peer serving cert is signed for DNS names [g2-apigateway localhost] and IPs [172.16.8.62 127.0.0.1 ::1][certificates] Generated etcd/healthcheck-client certificate and key.[certificates] Generated apiserver-etcd-client certificate and key.[certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"[controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"[controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"[controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"[init] waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests" [init] this might take a minute or longer if the control plane images have to be pulled[apiclient] All control plane components are healthy after 41.001672 seconds[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace[kubelet] Creating a ConfigMap "kubelet-config-1.11" in namespace kube-system with the configuration for the kubelets in the cluster[markmaster] Marking the node g2-apigateway as master by adding the label "node-role.kubernetes.io/master=''"[markmaster] Marking the node g2-apigateway as master by adding the taints [node-role.kubernetes.io/master:NoSchedule][patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "g2-apigateway" as an annotation[bootstraptoken] using token: o337m9.ceq32wg9g2gro7gx[bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster[bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace[addons] Applied essential addon: CoreDNS[addons] Applied essential addon: kube-proxyYour Kubernetes master has initialized successfully!To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/configYou should now deploy a pod network to the cluster.Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/You can now join any number of machines by running the following on each nodeas root:kubeadm join 10.10.207.253:6443 --token t69z6h.lr2etdbg9mfx5r15 --discovery-token-ca-cert-hash sha256:90e3a748c0eb4cb7058f3d0ee8870ee5d746214ab0589b5e841fd5d68fec8f00 这里注意最后一行：
kubeadm join 10.10.207.253:6443 --token t69z6h.lr2etdbg9mfx5r15 --discovery-token-ca-cert-hash sha256:90e3a748c0eb4cb7058f3d0ee8870ee5d746214ab0589b5e841fd5d68fec8f00
证明集群主节点安装成功，这里要记得保存这条命令，以便之后各个节点加入集群
配置kubetl认证信息
export KUBECONFIG=/etc/kubernetes/admin.conf # 如果你想持久化的话，直接执行以下命令【推荐】echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
安装flanel网络
依次执行以下命令：mkdir -p /etc/cni/net.d/cat </etc/cni/net.d/10-flannel.conf { “name”: “cbr0”,“type”: “flannel”,“delegate”: { “isDefaultGateway”: true} } EOFmkdir /usr/share/oci-umount/oci-umount.d -pmkdir /run/flannel/cat < /run/flannel/subnet.env FLANNEL_NETWORK=10.244.0.0/16 FLANNEL_SUBNET=10.244.1.0/24 FLANNEL_MTU=1450 FLANNEL_IPMASQ=true EOF
最后需要新建一个flannel.yml文件：
---kind: ClusterRoleapiVersion: rbac.authorization.k8s.io/v1beta1metadata: name: flannelrules: - apiGroups: - "" resources: - pods verbs: - get - apiGroups: - "" resources: - nodes verbs: - list - watch - apiGroups: - "" resources: - nodes/status verbs: - patch---kind: ClusterRoleBindingapiVersion: rbac.authorization.k8s.io/v1beta1metadata: name: flannelroleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: flannelsubjects:- kind: ServiceAccount name: flannel namespace: kube-system---apiVersion: v1kind: ServiceAccountmetadata: name: flannel namespace: kube-system---kind: ConfigMapapiVersion: v1metadata: name: kube-flannel-cfg namespace: kube-system labels: tier: node app: flanneldata: cni-conf.json: | { "name": "cbr0", "type": "flannel", "delegate": { "isDefaultGateway": true } } net-conf.json: | { "Network": "10.10.0.0/16", #这里换成集群所在的网段 "Backend": { "Type": "vxlan" } }---apiVersion: extensions/v1beta1kind: DaemonSetmetadata: name: kube-flannel-ds namespace: kube-system labels: tier: node app: flannelspec: template: metadata: labels: tier: node app: flannel spec: hostNetwork: true nodeSelector: beta.kubernetes.io/arch: amd64 tolerations: - key: node-role.kubernetes.io/master operator: Exists effect: NoSchedule serviceAccountName: flannel initContainers: - name: install-cni image: quay.io/coreos/flannel:v0.9.1-amd64 command: - cp args: - -f - /etc/kube-flannel/cni-conf.json - /etc/cni/net.d/10-flannel.conf volumeMounts: - name: cni mountPath: /etc/cni/net.d - name: flannel-cfg mountPath: /etc/kube-flannel/ containers: - name: kube-flannel image: quay.io/coreos/flannel:v0.9.1-amd64 command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr" ] securityContext: privileged: true env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace volumeMounts: - name: run mountPath: /run - name: flannel-cfg mountPath: /etc/kube-flannel/ volumes: - name: run hostPath: path: /run - name: cni hostPath: path: /etc/cni/net.d - name: flannel-cfg configMap: name: kube-flannel-cfg 执行：
kubectl create -f ./flannel.yml
默认情况下，master节点不参与工作负载，但如果希望安装出一个all-in-one的k8s环境，则可以执行以下命令，让master节点成为一个node节点：
kubectl taint nodes --all node-role.kubernetes.io/master-
执行之后，运行以下命令，查看节点信息：
kubectl get nodes
会看到如下的输出：
NAME STATUS ROLES AGE VERSIONk8s-master Ready master 18h v1.11.0
以下是节点配置
在配置好主节点之后，就可以配置集群的其他节点了，这里建议直接安装之前做好准备工作的系统镜像
进入节点机器之后，直接执行之前保存好的命令kubeadm join 10.10.207.253:6443 --token t69z6h.lr2etdbg9mfx5r15 --discovery-token-ca-cert-hash sha256:90e3a748c0eb4cb7058f3d0ee8870ee5d746214ab0589b5e841fd5d68fec8f00
执行完后会看到：
[preflight] running pre-flight checks [WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs_wrr ip_vs_sh ip_vs ip_vs_rr] or no builtin kernel ipvs support: map[ip_vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{} nf_conntrack_ipv4:{} ip_vs:{}]you can solve this problem with following methods: 1. Run 'modprobe -- ' to load missing kernel modules;2. Provide the missing builtin kernel ipvs supportI0725 09:59:27.929247 10196 kernel_validator.go:81] Validating kernel versionI0725 09:59:27.929356 10196 kernel_validator.go:96] Validating kernel config[discovery] Trying to connect to API Server "10.10.207.253:6443"[discovery] Created cluster-info discovery client, requesting info from "https://10.10.207.253:6443"[discovery] Requesting info from "https://10.10.207.253:6443" again to validate TLS against the pinned public key[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "10.10.207.253:6443"[discovery] Successfully established connection with API Server "10.10.207.253:6443"[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.11" ConfigMap in the kube-system namespace[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"[preflight] Activating the kubelet service[tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "k8s-node1" as an annotationThis node has joined the cluster:* Certificate signing request was sent to master and a response was received.* The Kubelet was informed of the new secure connection details.Run 'kubectl get nodes' on the master to see this node join the cluster. 这里就表示执行完毕了，可以去主节点执行命令：
kubectl get nodes
可以看到节点已加入集群：
NAME STATUS ROLES AGE VERSIONk8s-master Ready master 20h v1.11.0k8s-node1 Ready20h v1.11.0k8s-node2 Ready 20h v1.11.0
这期间可能需要等待一段时间，状态才会全部变为ready
kubernetes-dashboard安装
详见：
采坑指南
-
有时会出现master节点一直处于notready的状态，这里可能是没有启动flannel，只需要按照上面的教程配置好flannel，然后执行：
kubectl create -f ./flannel.yml
结语
参考资料：