20 Kubernetes Best Practices and Recommendations

Containers have become the default standard for cloud-native application development, and Kubernetes, also known as K8s, has emerged as one of the most popular container orchestration tools. A 2020 Cloud Native Computing Foundation (CNCF) survey noted that 91% of its respondents reported using Kubernetes, 83% of them in production. That’s up from 78% last year the previous year and 58% in 2018.

However, popular isn’t the same as easy to use. Kubernetes is a complex system. Getting started with it entails a significant learning curve. While the following Kubernetes best practices and recommendations may not be applicable to your situation, those that are can help you get more out of Kubernetes ─ easier and faster.

Common Kubernetes Best Practices

1. Go with Vendor Hosting

Use external hosting to kickstart your Kubernetes deployment. Depending on the type of hosted service you choose, your team won’t have to deal with manually provisioning the control plane components, adding nodes to expand your cluster, or configuring the pods that support your containers. They also won’t have to implement role-based access control (RBAC) to secure your separate environments or instrument monitoring and logging for your cluster.

2. Use the Latest Version

Use the newest version of Kubernetes. In addition to updates and additional features, the latest release will have patches to previous version security issues. This is critical for mitigating many of the vulnerabilities that could affect your cluster. Older versions also don’t get as much support from the Kubernetes community.

3. Use a Version Control System

You should store configuration files related to your deployment, ingress, services, and other files in a version control system before pushing to a cluster. This helps you keep track of who made changes and implement a change approval process to improve your cluster’s stability and security.

4. Use Labels and Annotations

As the number of objects in your cluster grows, it gets more difficult to find and organize them. Labels ─ key/value structures assigned to objects ─ let you attach meaningful and relevant metadata to cluster objects so they can be categorized, found, and operated on in bulk.

You can use labels to determine whether a pod is part of a production or a canary deployment and whether it’s front-end or back-end. They can also specify which layer an object belongs to and its release version, as well as differentiate between stable and alpha releases.

5. Use Readiness and Liveness Probes

Readiness and liveness probes are essentially health checks. A readiness probe ensures a given pod is up and running before allowing the load to get directed to that pod. If the pod isn’t ready, requests are taken away from your service until the probe verifies the pod is up. A liveness probe verifies if the application is still running. It tries to ping the pod for a response and then check its health. If there’s no response, then the application isn’t running on the pod. The liveness probe launches a new pod and starts the application on it if the check fails.

6. Set Resource Requests and Limits

Resource requests specify the minimum resources a container can use, while resource limits specify the maximum resources.

Without resource requests and limits, pods in a cluster can use more resources than required. If the pod starts consuming more CPU or memory on the node, then the scheduler may not be able to place new pods. Even the node itself may crash.

For both requests and limits, it’s typical to define CPU in millicores. Memory is defined in megabytes or mebibytes.

7. Use Smaller Container Images

Base images can consist of up to 80% of packages and libraries that won’t be needed. Instead, use smaller container images such as alpine images which are 10 times smaller than base images. They require less storage space and will help you pull and build the image faster. Also, the smaller the container image, the less chance of security issues. You can then add any packages and libraries required to run your application.

8. Monitor Control Plane Components

Monitor workload and resource consumption and the performance of control plane components, including Kubernetes API, kubelet, etcd, controller-manager, kube-proxy, and kube-dns. This will identify issues/threats within the cluster and increase its latency.

You should also use automated monitoring tools rather than manually managing the alerts.

9. Use Autoscaling

Leverage autoscaling mechanisms in Kubernetes to automatically scale cluster services when there’s a surge in resource consumption.

Horizontal pod autoscaler automatically scales the number of pods in deployment, a replication controller, replica set, or stateful set based on perceived CPU utilization.

Vertical pod autoscaler recommends suitable values to be set for CPU and memory requests and limits. And it can automatically update the values.

Cluster autoscaler expands and shrinks the size of the pool of worker nodes. It adjusts the size of a Kubernetes cluster depending on the current utilization.

10. Adopt GitOps

Use GitOps, a Git-based workflow, as the preferred model for using Git as the single source of truth for all automation, including CI/CD pipelines. Using a GitOps framework can help improve productivity by speeding up application development and lowering deployment times. It can also enhance error traceability and automate CI/CD workflows.

11. Monitor Disk Usage

Because high-disk usage can affect cluster performance, you should regularly monitor all disk volumes associated with your cluster, as well as the root file system. With alert monitoring, you’ll be able to take corrective actions either by scaling or freeing disk space when needed.

12. Start with Stateless Applications

With a stateless backend, development teams can make sure there aren’t long-running connections that make scaling more challenging. Using stateless also enables developers to deploy applications more efficiently with zero downtime.

Stateless applications make it easier to migrate and scale as and when required according to business needs.

Kubernetes Best Practices – Authorization and Authentication

13. Employ Role-based Access Control

To help enhance the security of your Kubernetes workloads, enable Kubernetes Role-Based Access Control (RBAC). RBAC is usually enabled by default on Kubernetes 1.6 and higher. Because Kubernetes combines authorization controllers, when you enable RBAC, you must also disable the legacy Attribute Based Access Control (ABAC).

Assign roles to each user in your cluster and each service account running in your cluster. Roles in RBAC contain several permissions that a user or service account can perform. You can assign the same role to multiple people and each role can have multiple permissions. RBAC settings can also be applied on namespaces. If you assign roles to a user allowed in one namespace, they will not have access to other namespaces in the cluster. Kubernetes provides RBAC properties such as role and cluster role to define security policies.

14. Use Third-Party Authentication for API Server

Integrate Kubernetes with a third-party authentication provider to take advantage of additional security features such as multi-factor authentication. Doing so also helps ensure that kube-apiserver doesn’t change when you add or remove users. Make sure users aren’t managed at the API server level if possible.

Kubernetes Best Practices – Security Issues

15. Use Network Policies

Use network policies to restrict access to services within a Kubernetes cluster. They can also restrict access to your cloud’s metadata API from pods in the cluster, and control traffic flow at the IP address or port level.

You’ll find information on setting up a network policy in the Kubernetes documentation.

16. Don’t Run as Root

The UID (the Kubernetes systems-generated string that uniquely identifies objects) of the user running a container maps directly to the host. If the container runs as UID 0 (root), it will also appear as root on the node it’s running on. Kubernetes has built-in protections to prevent escalation of privileges with this mechanism, However, there’s always a chance that security issues could allow for escalating privileges. Avoid the situation by not running run your containers as root. Instead, modify the Dockerfile for your built containers to create and use a user with a known UID.

17. Create a Firewall

Create a firewall for your API server to prevent attackers from sending connection requests to your API server from the internet. You can use regular firewalling rules or port firewalling rules. If you are using something like GKE, you can use a master authorized network feature to limit the IP addresses that can access the API server.

18. Restrict API Access

Most cloud implementations for Kubernetes already restrict access to the Kubernetes API for your cluster by using RBAC, Identity & Access Management (IAM), or Active Directory (AD). If your cluster doesn’t use these methods, set them up using open-source projects for interacting with various authentication methods.

19. Restrict SSH Access

Another important security measure is to restrict SSH access to your Kubernetes nodes. You typically wouldn’t have port 22 open on any node but may need it to debug issues at some point. Configure your nodes via your cloud provider to block access to port 22 except via your organization’s VPN or a bastion host. You’ll be able to quickly get SSH access but outside attackers won’t.

20. Audit Policy Logs

Regularly audit all logs stored at /var/log/audit.log to identify threats, monitor resource consumption, and capture key event heartbeats of the Kubernetes cluster. The Kubernetes cluster default policies are defined in the /etc/kubernetes/audit-policy.yaml file. You can customize them for specific requirements. You can also use Fluentd, an open-source tool, to maintain a centralized logging layer for your containers.

Partner with ClearScale

These Kubernetes best practices are just some of the many that are available to help make Kubernetes an easier and more valuable system to use in application modernization. Nonetheless, there’s still a lot to learn to use Kubernetes effectively.

That can be overwhelming for development teams already bogged down with the many tasks required in modern application development ─ even with the ever-growing number of tools and services to accelerate the processes entailed. That’s why many organizations with complex application development projects choose to partner with companies that have a depth and breadth of expertise and experience in using Kubernetes.

ClearScale not only has proficiency in Kubernetes best practices, but we also have the customer stories to prove our successes. Here are just a few of them:

Three AWS-centric Projects Help Agari Maintain Email Security Leadership

Bloomreach Modernizes Cloud Infrastructure and Reduces IT Costs with AWS Refresh

First Street Foundation Optimizes Application Performance with AWS Containerization Services

If you’re interested in leveraging Kubernetes best practices – or have any cloud container services needs, contact ClearScale to find out how we can help.

Call us at 1-800-591-0442
Send us an email at sales@clearscale.com
Fill out a Contact Form
Read our Customer Case Studies