Here’s a brief tutorial to understand and get started with DevOps.

DevOps… CI/CD… Docker… Kubernetes… I’m sure you’ve been bombarded with these words a lot the past year. Seems like the entire world is talking about it. The rate at which this segment is progressing, it won’t be long before we reach the stage of NoOps. It’s about time we break down what DevOps really is.

The objective of this article is to set a solid foundation for you to build on top of. So let’s start with the obvious question.

What is DevOps?

DevOps is the simplification or automation of established IT processes.

I’ve seen so many people start this journey to adopt DevOps to only find themselves lost. There seems to be a pattern to this.

It usually starts with a video on how a fancy tech startup has automated its release cycle. Deployments happen automatically once all the tests pass. Rollbacks, in case of failures, is automatic. Thousands of simultaneous A/B test is driving up customer engagement.

We all are tired of releasing a new version like it’s a rollercoaster ride.

Unfortunately, DevOps doesn’t work that way. DevOps isn’t a magic wand that can solve all your problems in an instant.

Instead, it is a systematic process of choosing the right tools and technology to get the job done.

All of This Starts With a Process

It doesn’t matter what the process is. It could be simplifying the deployment of your app or automate testing. Whatever your process is, the act of making your life easier is what DevOps is all about.

But you always need to start with a process.

In fact, if your process cannot be done manually (on a smaller scale), you should probably re-examine your process.

I mean it.

Enough talking. Let’s take a real-world example to understand things better.

Let’s Take a Real DevOps Example

Let’s take a simple example of making a Nodejs app live on a VM in the cloud.

The Process

Here’s what our process looks like:

• Start with the source code: This is our source of truth. We can run our process from anywhere as long as we have access to the source code.
• Build an Artifact: We then package our source code to build an Artifact. In the case of a compiled language, the compiled output (JAR file, in the case of JAVA) would be our artifact. In our case, our source code itself is the artifact to be released.
• Publish to an Artifact Repository: Next, we push our artifact to a repository. This is a location from where our target environment can pull the artifact from. We could stick with something like Github since we are working with source code here.
• Pull and run your app: Finally, we pull the artifact onto our VM and schedule a Nodejs process by running npm start.

It’s okay if you do things a slightly different way. We are here to focus on the journey and not the destination.

Our First DevOps Project

Let’s not do anything fancy here.

The easiest way to automate this process would be to write a simple shell script to run all the commands in sequence.

Congratulations!!! That’s our first DevOps project!!!

I know shell scripts sounds too simple to be taken seriously. I suspect you already have such scripts in place. But believe me, that’s DevOps!

Don’t worry; we will get to the fancy stuff in a minute. But it’s important to understand that this is how DevOps works.

Importance of Repeatability

Let me ask you one question. Which one of these would you prefer?

• An automated deployments pipeline which works 60% of the time, or,
• A boring shell script that gets the job done every time it’s executed.

If you have dealt with production failures in the middle of the night, you’ll choose the shell script.

The reason is simple.

Reliability is far more critical than the degree of automation.

In other words,

A DevOps process must be able to produce consistent results every time it’s run.

Making Our Process Repeatable

Let’s take the example of our shell script.

Currently, our shell script depends on Node.js to be installed on the VM we want to deploy the app to.

What would happen if the Nodejs runtime was missing? These days, an incorrect version of the runtime is enough to break our application.

This problem only gets worse in polyglot environments where we deal with multiple programming languages.

A simple solution would be to archive the Nodejs runtime along with our source code in a zip file. The zip file can then be sent to the VM. This way, the VM can use the local Nodejs runtime present in the archive to run our app.

Luckily, there is a tool to make our lives easier.

In Comes Docker and Containers

If you are new to this, think of Docker as a way to package your artifact along with all its OS dependencies, including Nodejs, into a container image.

Using containers, we can deploy any application on a VM which has Docker installed.

With Docker, our flow will look something like this:

There is a lot more to containers than just this. However, this was one of the reasons why containers got so popular.

Docker Vs. Containers

Let me clarify this. Docker and containers are not the same things anymore.

Docker is a set of utility tools to build and ship container images which container runtimes like containerd use to make and run containers.

Many are concerned about the future of Docker, given the recent events which have taken place.

It is important to understand that Docker is not going anywhere anytime soon. It provides the best DX and will continue to play a major role in building and shipping container images.

Getting Serious With DevOps

We have made some serious progress already. Hopefully, we understand how Docker fits into the DevOps process.

It’s time to take things to the next level.

Triggering Deployment Based on Events

Our script looks pretty solid, but it’s still triggered manually.

Wouldn’t it be great if we could trigger this script automatically whenever someone pushes code on GitHub? In other words, we want to trigger this script on an event.

GitHub can invoke webhooks on a certain set of events.

To achieve this, we need to make a simple HTTP server that executes our shell script whenever its endpoint is hit. We can configure GitHub to hit our endpoint on the Push Event.

Let’s call this server Colorful Daemons or CD.

Our new flow will look something like this:

Congratulations! You just set up what we call a CD pipeline.

And no… I don’t mean Colorful Daemons. I’m talking about Continous Deployments.

Continuous Deployments is a piece of software responsible for taking your app from something like GitHub all the way to your target environment where it finally gets deployed.

This is basically the CI/CD stuff you keep hearing about. When people talk about tools like Jenkins and CircleCI, they are usually referring to CI/CD.

What we just made with Colorful Daemons was a continuous deployments pipeline. Don’t confuse it with continuous integration or delivery. We’ll get to those some other day.

The DevOps Pattern

I guess you’ve already found a pattern here. We start with a process, find a section we aren’t happy with and then introduce some software component to simplify or automate it.

That’s getting the dev in ops. And that’s all that there is to it.

This is the real answer to the question, ‘What is DevOps?.’

Introducing Container Orchestration

Let’s finish up by making one small improvement.

Till now, we have been dealing with deploying our app to a single VM or a single node. What if we wanted to deploy our app to multiple nodes?

The easiest way to achieve this would be to modify our CD server to ssh into all the VMs and deploy our container to each one of them.

While this method works, we’ll need to change our script every time our infrastructure changes. In a world where applications are always autoscaling, and VMs are considered disposable, this is unacceptable.

A better way would be to make another HTTP server to track infrastructure changes. We can call this server ‘Pilot.’

This server will be responsible for performing health checks on the various VMs in our cluster to maintain a list of active VMs. It could even communicate with the cloud vendor to make things more robust.

Pilot will expose an endpoint as well to accept the details of the container to spawn. It can then talk to the various VMs to get the job done.

Now, our CD server can simply request Pilot instead of talking to each VM individually.

Our new flow will look something like this:

The second server, Pilot, is called a container orchestrator. That’s what Kubernetes is!

You just designed a mini version of Kubernetes!

Also, Kubernetes is greek for Pilot. Isn’t that a pleasant co-incidence?

Where to Start?

We covered quite a few tools together. This brings me to my last point. Ever wondered why the DevOps space is so fragmented?

If you think about it, there are so many tools out there, making it hard to decide: what’s the right choice or where you should even start?

Every organisation has its own way, its own process to do things. And since their paths are different, the tools they need to use are different.

Your job is not to find which tool is the best. Your job is to find what process works for you best. Once you have that figured out, the tools are just a google search away.

So now you know where to start. It’s not with the tools out there.

Start by understanding how your company and teams do things.

I’m literally asking you to open up a Word document and copy-paste the commands you need to run to do stuff.

Wrapping Up

I hope this post has been helpful in understanding how the DevOps field is arranged and how different tools depend and coexist with each other.

I’d like to add:

Your DevOps process is only as strong as its foundation.

So work on the underlying process. It’s okay if you need to tweak your current process a bit.

An excellent foundation to build upon could be using tools like SpaceCloud. Space Cloud is a Kubernetes-based serverless platform that helps you develop, deploy and secure cloud-native applications.

In a nutshell, SpaceCloud gives you an excellent starting point to build your DevOps practices on top of. It makes performing rolling upgrades, canary deployments, and autoscaling your applications easy. You can configure everything using the space-cli or REST APIs.

The post What is DevOps? appeared first on Tech Chronicles.

Kubernetes is quickly becoming the new standard for deploying and managing software in the cloud. With all the power Kubernetes provides, however, comes a steep learning curve. As a newcomer, trying to parse the official documentation can be overwhelming. There are many different pieces that make up the system, and it can be hard to tell which ones are relevant for your use case. This blog post will provide a simplified view of Kubernetes, but it will attempt to give a high-level overview of the most important components and how they fit together.

First, lets look at how hardware is represented

Hardware

Nodes

A node is the smallest unit of computing hardware in Kubernetes. It is a representation of a single machine in your cluster. In most production systems, a node will likely be either a physical machine in a datacenter, or virtual machine hosted on a cloud provider like Google Cloud Platform. Don’t let conventions limit you, however; in theory, you can make a node out of almost anything.

Thinking of a machine as a “node” allows us to insert a layer of abstraction. Now, instead of worrying about the unique characteristics of any individual machine, we can instead simply view each machine as a set of CPU and RAM resources that can be utilized. In this way, any machine can substitute any other machine in a Kubernetes cluster.

The Cluster

Although working with individual nodes can be useful, it’s not the Kubernetes way. In general, you should think about the cluster as a whole, instead of worrying about the state of individual nodes.

In Kubernetes, nodes pool together their resources to form a more powerful machine. When you deploy programs onto the cluster, it intelligently handles distributing work to the individual nodes for you. If any nodes are added or removed, the cluster will shift around work as necessary. It shouldn’t matter to the program, or the programmer, which individual machines are actually running the code.

If this kind of hivemind-like system reminds you of the Borg from Star Trek, you’re not alone; “Borg” is the name for the internal Google project Kubernetes was based on.

Persistent Volumes

Because programs running on your cluster aren’t guaranteed to run on a specific node, data can’t be saved to any arbitrary place in the file system. If a program tries to save data to a file for later, but is then relocated onto a new node, the file will no longer be where the program expects it to be. For this reason, the traditional local storage associated to each node is treated as a temporary cache to hold programs, but any data saved locally can not be expected to persist.

To store data permanently, Kubernetes uses Persistent Volumes. While the CPU and RAM resources of all nodes are effectively pooled and managed by the cluster, persistent file storage is not. Instead, local or cloud drives can be attached to the cluster as a Persistent Volume. This can be thought of as plugging an external hard drive in to the cluster. Persistent Volumes provide a file system that can be mounted to the cluster, without being associated with any particular node.

Software

Containers

Programs running on Kubernetes are packaged as Linux containers. Containers are a widely accepted standard, so there are already many pre-built images that can be deployed on Kubernetes.

Containerization allows you to create self-contained Linux execution environments. Any program and all its dependencies can be bundled up into a single file and then shared on the internet. Anyone can download the container and deploy it on their infrastructure with very little setup required. Creating a container can be done programmatically, allowing powerful CI and CD pipelines to be formed.

Multiple programs can be added into a single container, but you should limit yourself to one process per container if at all possible. It’s better to have many small containers than one large one. If each container has a tight focus, updates are easier to deploy and issues are easier to diagnose.

Pods

Unlike other systems you may have used in the past, Kubernetes doesn’t run containers directly; instead it wraps one or more containers into a higher-level structure called a pod. Any containers in the same pod will share the same resources and local network. Containers can easily communicate with other containers in the same pod as though they were on the same machine while maintaining a degree of isolation from others.

Pods are used as the unit of replication in Kubernetes. If your application becomes too popular and a single pod instance can’t carry the load, Kubernetes can be configured to deploy new replicas of your pod to the cluster as necessary. Even when not under heavy load, it is standard to have multiple copies of a pod running at any time in a production system to allow load balancing and failure resistance.

Pods can hold multiple containers, but you should limit yourself when possible. Because pods are scaled up and down as a unit, all containers in a pod must scale together, regardless of their individual needs. This leads to wasted resources and an expensive bill. To resolve this, pods should remain as small as possible, typically holding only a main process and its tightly-coupled helper containers (these helper containers are typically referred to as “side-cars”).

Deployments

Although pods are the basic unit of computation in Kubernetes, they are not typically directly launched on a cluster. Instead, pods are usually managed by one more layer of abstraction: the deployment.

A deployment’s primary purpose is to declare how many replicas of a pod should be running at a time. When a deployment is added to the cluster, it will automatically spin up the requested number of pods, and then monitor them. If a pod dies, the deployment will automatically re-create it.

Using a deployment, you don’t have to deal with pods manually. You can just declare the desired state of the system, and it will be managed for you automatically.

Ingress

Using the concepts described above, you can create a cluster of nodes, and launch deployments of pods onto the cluster. There is one last problem to solve, however: allowing external traffic to your application.

By default, Kubernetes provides isolation between pods and the outside world. If you want to communicate with a service running in a pod, you have to open up a channel for communication. This is referred to as ingress.

There are multiple ways to add ingress to your cluster. The most common ways are by adding either an Ingress controller, or a LoadBalancer. The exact tradeoffs between these two options are out of scope for this post, but you must be aware that ingress is something you need to handle before you can experiment with Kubernetes.

What’s Next

What’s described above is an oversimplified version of Kubernetes, but it should give you the basics you need to start experimenting.

To experiment with Kubernetes locally, Minikube will create a virtual cluster on your personal hardware. If you’re ready to try out a cloud service, Google Kubernetes Engine has a collection of tutorials to get you started.

If you are new to the world of containers and web infrastructure, I suggest reading up on the 12 Factor App methodology. This describes some of the best practices to keep in mind when designing software to run in an environment like Kubernetes.

The post Kubernetes 101: Pods, Nodes, Containers, and Clusters appeared first on Tech Chronicles.

As an industry, we aspire to something that delivers on our collective vision of fully-automated, self-governing infrastructure. In the 00s we called it autonomic computing, then software defined data center (SDDC) and, now, IaC. Sadly, IaC is not it—or, more specifically, it’s not it alone.

The actual definition of IaC is way smaller than what people want it to be.

My take-away from IaC discussions is they are built around six key automation themes. Let’s take a few minutes to examine each of the themes. Then we’ll have a basis for answering “If that’s not IaC, then what is it?”

Programmatic Configuration (IaC)

Programmatic configuration captures the idea that we can define our infrastructure in a machine readable format. For example, you describe a planned cluster build in YAML to execution by a tool such as Terraform that requests the resources from your dynamic infrastructure and, hopefully, tracks that resource descriptors you get back from that request. Basically, it’s just your IaaS input format.

Dynamic Infrastructure (or IaaS)

It describes a system that allows operators to change the resource allocations via an API. It’s a critical part of the discussion because there must be a dependable way to pragmatically change the infrastructure. Some would consider this cloud; however, we should not assumes either managed services or VMs because dynamic infrastructure is much broader.

Desired State (or State Reconciler)

Desired state actively compares dynamic infrastructure to programmatic manifest and makes automatic adjustments to keep them synchronized. Typically, the idea of maintaining state is separated from configuring systems. Kubernetes and other Autoscaling cluster managers perform this role.

Deployment from Source Control (or Immutability)

It implies that all of the configuration components of the system can be stored as versioned artifacts and deployed from source. A Dockerfile that builds a container is a good example of an immutable process and the generated artifact. The repeatability offered by deploying from source control is transformative compared to past generations of operations that relied on either manual or scripted changes to production systems. Immutability is not IaC: We are focused on an generated deployment artifact rather than managing the infrastructure.

Composable (or Modular) Automation

It allows us to manage complex structures through layers of abstraction. In concept, composability is as simple as using platform plugin to script against an API.

CI/CD Pipeline

CI/CD pipeline is the glue that pulls all these pieces together to create a resilient environment by continuously adjusting the system components based on the operators’ IaC descriptions. The pipeline relies on having dynamic infrastructure managed to desired state as a target; however, it adds the critical concepts of staged changes over time to the system. Since we need to create systems that can adapt and improve over time, the CI/CD pipeline provides mechanism for controlled implementation change over time.

Together, these six concepts reflect an integrated new way to think about the infrastructure that is beyond IaC. They allow us to maintain and control our systems over time in a way that is resilient and automatic. More importantly, they bring complex infrastructure problems back into terms that people can comprehend by creating more deterministic and predictable chains of change. That allows us to keep our hands out of the infrastructure and our minds focused on what we are delivering: a continuously integrated data center.

Since all six concepts are interdependent, continuously integrated data center (CI DC) completely captures our aspirations for IaC. The term intuitively builds from source code into a fully operational infrastructure. More importantly, CI DC demands that we acknowledge the ongoing maintenance required. For that reason, I’ve embraced CI DC to replace IaC.

The post Infrastructure as Code and Six Key Automation Concepts appeared first on Tech Chronicles.

Why do we need to use a Dockerfile?

Dockerfile is not yet-another shell. Dockerfile has its special mission: automation of Docker image creation.

Once, you write build instructions into Dockerfile, you can build the same image just with docker build command.

Dockerfile is also useful to tell the knowledge of what a job the container does to somebody else. Your teammates can tell what the container is supposed to do just by reading Dockerfile. They don’t need to know login to the container and figure out what the container is doing by using ps command.

For these reasons, you must use Dockerfile when you build images. However, writing Dockerfile is sometimes painful. In this post, I will write a few tips and gochas in writing Dockerfile so that you love the tool.

ADD and understanding context in Dockerfile

ADD is the instruction to add local files to Docker image. The basic usage is very simple. Suppose you want to add a local file called myfile.txt to /myfile.txt of image.

1
2

$ ls
Dockerfile  mydir  myfile.txt

Then your Dockerfile looks like this.

1

ADD myfile.txt /

Very simple. However, if you want to add /home/vagrant/myfile.txt, you can’t do this.

1
2
3
4
5
6
7
8
9

# Your have this in your Dockerfile
# ADD /home/vagrant/myfile.txt /

$ docker build -t blog .
Uploading context 10240 bytes
Step 1 : FROM ubuntu
 ---> 8dbd9e392a96
Step 2 : ADD /home/vagrant/myfile.txt /
Error build: /home/vagrant/myfile.txt: no such file or directory

You got no such file or directory error even if you have the file. Why? This is because /home/vagrant/myfile.txt is not added to the context of Dockerfile. Context in Dockerfile means files and directories available to the Dockerfile instructions. Only files and directories in the context can be added during build. Files and sub directories under the current directory are added to the context. You can see this when you run build command.

1
2

$ docker build -t blog .
Uploading context 10240 bytes

What’s happening here is Docker client makes tarball of entries under the current directory and send it to Docker daemon. The reason why thiis is required is because your Docker daemon may be running on remote machine. That’s why the above command says Uploading.

There is a pitfall, though. Since automatically entries under current directories are added to the context, it tries to upload huge files and take longer time for build even if you don’t add the file.

1
2
3
4
5
6

$ ls
Dockerfile  very_huge_file

$ docker build -t blog .
Uploading context xxxxxx bytes
..... # Takes very long time

So the best practice is only placing files and directories that you need to add to image under current directory.

Treat your container like a binary with CMD

By using CMD instruction in Dockerfile, your container acts like a single executable binary. Suppose you have these instructions in your Dockerfile.

1
2
3

# Suppose you have run.sh in the same directory as the Dockerfile
ADD run.sh /usr/local/bin/run.sh
CMD ["/usr/local/bin/run.sh"]

When you build a container from this Dockerfile and run with docker run -i run_image, it runs /usr/local/bin/run.sh script and exists.

If you don’t use CMD, you always have to pass the command to the argument: docker run -i run_image /usr/local/bin/run.sh.

This is not just cumbersome, but also considered to be a bad practice from the perspective of operation.

If you have CMD instruction, the purpose of the container becomes explicit: all what the container wants to do is running the command.

But, if you don’t have the instruction, anybody except the person who made the container need to rely on external documentation to know how to run the container properly.

So, in general, you should have CMD instruction in your Dockerfile.

Difference between CMD and ENTRYPOINT

CMD and ENTRYPOINT are confusing.

Every commands, either passed as an argument or specified from CND instruction are passed as argument of binary specified in ENTRYPOINT.

/bin/sh -c is the default entrypoint. So if you specify CMD date without specifying entrypoint, Docker executes it as /bin/sh -c date.

By using entrypoint, you can change the behaviour of your container at run time that makes container operation a bit more flexible.

1

ENTRYPOINT ["/bin/date"]

With the entrypoint above, the container prints out current date with different format.

1
2
3
4
5

$ docker run -i clock_container +"%s"
1404214000

$ docker run -i clock_container +"%F"
2014-07-01

exec format error

There is one caveat in default entrypoint. For example, you want to execute the following shell script.

/usr/local/bin/run.sh

1

echo "hello, world"

Dockerfile

1
2
3

ADD run.sh /usr/local/bin/run.sh
RUN chmod +x /usr/local/bin/run.sh
CMD ["/usr/local/bin/run.sh"]

When you run the container, your expectation is the container prints out hello, world. However, what you will get is a error message that doesn’t make sense.

1
2

$ docker run -i hello_world_image
2014/07/01 10:53:57 exec format error

You see this message when you didn’t put shebang in your script, and because of that, default entrypoint /bin/sh -c does not know how to run the script.

To fix this, you can either add shebang

/usr/local/bin/run.sh

1
2

#!/bin/bash
echo "hello, world"

or you can specify from command line.

1

$ docker run -entrypoint="/bin/bash" -i hello_world_image

Build caching: what invalids cache and not?

Docker creates a commit for each line of instruction in Dockerfile. As long as you don’t change the instruction, Docker thinks it doesn’t need to change the image, so use cached image which is used by the next instruction as a parent image. This is the reason why docker build takes long time in the first time, but immediately finishes in the second time.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

$ time docker build -t blog .
Uploading context 10.24 kB
Step 1 : FROM ubuntu
 ---> 8dbd9e392a96
Step 2 : RUN apt-get update
 ---> Running in 15705b182387
Ign http://archive.ubuntu.com precise InRelease
Hit http://archive.ubuntu.com precise Release.gpg
Hit http://archive.ubuntu.com precise Release
Hit http://archive.ubuntu.com precise/main amd64 Packages
Get:1 http://archive.ubuntu.com precise/main i386 Packages [1641 kB]
Get:2 http://archive.ubuntu.com precise/main TranslationIndex [3706 B]
Get:3 http://archive.ubuntu.com precise/main Translation-en [893 kB]
Fetched 2537 kB in 7s (351 kB/s)

 ---> a8e9f7328cc4
Successfully built a8e9f7328cc4

real  0m8.589s
user  0m0.008s
sys   0m0.012s

$ time docker build -t blog .
Uploading context 10.24 kB
Step 1 : FROM ubuntu
 ---> 8dbd9e392a96
Step 2 : RUN apt-get update
 ---> Using cache
 ---> a8e9f7328cc4
Successfully built a8e9f7328cc4

real  0m0.067s
user  0m0.012s
sys   0m0.000s

However, when cache is used and what invalids cache are sometimes not very clear. Here is a few cases that I found worth to note.

Cache invalidation at one instruction invalids cache of all subsequent instructions

This is the basic rule of caching. If you cause cache invalidation at one instruction, subsequent instructions doesn’t use cache.

1
2
3
4
5
6
7
8
9
10

# Before
From ubuntu
Run apt-get install ruby
Run echo done!

# After
From ubuntu
Run apt-get update
Run apt-get install ruby
Run echo done!

Since you add Run apt-get update instruction, all instructions after that have to be done from the scratch even if they are not changed. This is inevitable because Dockerfile uses the image built by the previous instruction as a parent image to execute next instruction. So, if you insert an instruction that creates a new parent image, all subsequent instructions cannot use cache because now parent image differs.

Cache is invalid even when adding commands that don’t do anything

This invalidates caching. For example,

1
2
3
4
5

# Before
Run apt-get update

# After
Run apt-get update && true

Even if true command doesn’t change anything of the image, Docker invalids the cache.

Cache is invalid when you add spaces between command and arguments inside instruction

This invalids cache

1
2
3
4
5

# Before
Run apt-get update

# After
Run apt-get               update

Cache is used when you add spaces around commands inside instruction

Cache is valid even if you add space around commands

1
2
3
4
5

# Before
Run apt-get update

# After
Run                apt-get update

Cache is used for non-idempotent instructions

This is kind of pitfall of build caching. What I mean by non-idempotent instructions is the execution of commands that may return different result each time. For example, apt-get update is not idempotent because the content of updates changes as time goes by.

1
2

From ubuntu
Run apt-get update

You made this Dockerfile and create image. 3 months later, Ubuntu made some security updates to their repository, so you rebuild the image by using the same Dockerfile hoping your new image includes the security updates. However, this doesn’t pick up the updates. Since no instructions or files are changed, Docker uses cache and skips doing apt-get update.

If you don’t want to use cache, just pass -no-cache option to build.

1

$ docker build -no-cache .

Instructions after ADD never cached (Only versions prior to 0.7.3)

If you use Docker before v7.3, watch out!

1
2
3
4

From ubuntu
Add myfile /
Run apt-get update
Run apt-get install openssh-server

If you have Dockerfile like this, Run apt-get update and Run apt-get install openssh-server will never be cached.

The behavior is changed from v7.3. It caches even if you have ADD instruction, but invalids cache if file content is changed.

1
2
3
4
5
6
7
8
9
10
11

$ echo "Jeff Beck" > rock.you

From ubuntu
Add rock.you /
Run add rock.you

$ echo "Eric Clapton" > rock.you

From ubuntu
Add rock.you /
Run add rock.you

Since you change rock.you file, instructions after Add doesn’t use cache.

Hack to run container in the background

If you want to simplify the way to run containers, you should run your container on background with docker run -d image your-command. Instead of running with docker run -i -t image your-command, using -d is recommended because you can run your container with just one command and you don’t need to detach terminal of container by hitting Ctrl + P + Q.

However, there is a problem with -d option. Your container immediately stops unless the commands are not running on foreground.

Let me explain this by using case where you want to run apache service on a container. The intuitive way of doing this is

1

$ docker run -d apache-server apachectl start

However, the container stops immediately after it is started. This is because apachectl exits once it detaches apache daemon.

Docker doesn’t like this. Docker requires your command to keep running in the foreground. Otherwise, it thinks that your applications stops and shutdown the container.

You can solve this by directly running apache executable with foreground option.

1
2
3
4
5
6
7

$ docker run -e APACHE_RUN_USER=www-data \
                    -e APACHE_RUN_GROUP=www-data \
                    -e APACHE_PID_FILE=/var/run/apache2.pid \
                    -e APACHE_RUN_DIR=/var/run/apache2 \
                    -e APACHE_LOCK_DIR=/var/lock/apache2 \
                    -e APACHE_LOG_DIR=/var/log/apache2 \
                    -d apache-server /usr/sbin/apache2 -D NO_DETACH -D FOREGROUND

Here we are manually doing what apachectl does for us and run apache executable. With this approach, apache keeps running on foreground.

The problem is that some application does not run in the foreground. Also, we need to do extra works such as exporting environment variables by ourselves. How can we make it easier?

In this situation, you can add tail -f /dev/null to your command. By doing this, even if your main command runs in the background, your container doesn’t stop because tail is keep running in the foreground. We can use this technique in the apache case.

1

$ docker run -d apache-server apachectl start && tail -f /dev/null

Much better, right? Since tail -f /dev/null doesn’t do any harm, you can use this hack to any applications.

The post Gotchas in Writing Dockerfile appeared first on Tech Chronicles.