Here’s a brief tutorial to understand and get started with DevOps.

DevOps… CI/CD… Docker… Kubernetes… I’m sure you’ve been bombarded with these words a lot the past year. Seems like the entire world is talking about it. The rate at which this segment is progressing, it won’t be long before we reach the stage of NoOps. It’s about time we break down what DevOps really is.

The objective of this article is to set a solid foundation for you to build on top of. So let’s start with the obvious question.

What is DevOps?

DevOps is the simplification or automation of established IT processes.

I’ve seen so many people start this journey to adopt DevOps to only find themselves lost. There seems to be a pattern to this.

It usually starts with a video on how a fancy tech startup has automated its release cycle. Deployments happen automatically once all the tests pass. Rollbacks, in case of failures, is automatic. Thousands of simultaneous A/B test is driving up customer engagement.

We all are tired of releasing a new version like it’s a rollercoaster ride.

Unfortunately, DevOps doesn’t work that way. DevOps isn’t a magic wand that can solve all your problems in an instant.

Instead, it is a systematic process of choosing the right tools and technology to get the job done.

All of This Starts With a Process

It doesn’t matter what the process is. It could be simplifying the deployment of your app or automate testing. Whatever your process is, the act of making your life easier is what DevOps is all about.

But you always need to start with a process.

In fact, if your process cannot be done manually (on a smaller scale), you should probably re-examine your process.

I mean it.

Enough talking. Let’s take a real-world example to understand things better.

Let’s Take a Real DevOps Example

Let’s take a simple example of making a Nodejs app live on a VM in the cloud.

The Process

Here’s what our process looks like:

• Start with the source code: This is our source of truth. We can run our process from anywhere as long as we have access to the source code.
• Build an Artifact: We then package our source code to build an Artifact. In the case of a compiled language, the compiled output (JAR file, in the case of JAVA) would be our artifact. In our case, our source code itself is the artifact to be released.
• Publish to an Artifact Repository: Next, we push our artifact to a repository. This is a location from where our target environment can pull the artifact from. We could stick with something like Github since we are working with source code here.
• Pull and run your app: Finally, we pull the artifact onto our VM and schedule a Nodejs process by running npm start.

It’s okay if you do things a slightly different way. We are here to focus on the journey and not the destination.

Our First DevOps Project

Let’s not do anything fancy here.

The easiest way to automate this process would be to write a simple shell script to run all the commands in sequence.

Congratulations!!! That’s our first DevOps project!!!

I know shell scripts sounds too simple to be taken seriously. I suspect you already have such scripts in place. But believe me, that’s DevOps!

Don’t worry; we will get to the fancy stuff in a minute. But it’s important to understand that this is how DevOps works.

Importance of Repeatability

Let me ask you one question. Which one of these would you prefer?

• An automated deployments pipeline which works 60% of the time, or,
• A boring shell script that gets the job done every time it’s executed.

If you have dealt with production failures in the middle of the night, you’ll choose the shell script.

The reason is simple.

Reliability is far more critical than the degree of automation.

In other words,

A DevOps process must be able to produce consistent results every time it’s run.

Making Our Process Repeatable

Let’s take the example of our shell script.

Currently, our shell script depends on Node.js to be installed on the VM we want to deploy the app to.

What would happen if the Nodejs runtime was missing? These days, an incorrect version of the runtime is enough to break our application.

This problem only gets worse in polyglot environments where we deal with multiple programming languages.

A simple solution would be to archive the Nodejs runtime along with our source code in a zip file. The zip file can then be sent to the VM. This way, the VM can use the local Nodejs runtime present in the archive to run our app.

Luckily, there is a tool to make our lives easier.

In Comes Docker and Containers

If you are new to this, think of Docker as a way to package your artifact along with all its OS dependencies, including Nodejs, into a container image.

Using containers, we can deploy any application on a VM which has Docker installed.

With Docker, our flow will look something like this:

There is a lot more to containers than just this. However, this was one of the reasons why containers got so popular.

Docker Vs. Containers

Let me clarify this. Docker and containers are not the same things anymore.

Docker is a set of utility tools to build and ship container images which container runtimes like containerd use to make and run containers.

Many are concerned about the future of Docker, given the recent events which have taken place.

It is important to understand that Docker is not going anywhere anytime soon. It provides the best DX and will continue to play a major role in building and shipping container images.

Getting Serious With DevOps

We have made some serious progress already. Hopefully, we understand how Docker fits into the DevOps process.

It’s time to take things to the next level.

Triggering Deployment Based on Events

Our script looks pretty solid, but it’s still triggered manually.

Wouldn’t it be great if we could trigger this script automatically whenever someone pushes code on GitHub? In other words, we want to trigger this script on an event.

GitHub can invoke webhooks on a certain set of events.

To achieve this, we need to make a simple HTTP server that executes our shell script whenever its endpoint is hit. We can configure GitHub to hit our endpoint on the Push Event.

Let’s call this server Colorful Daemons or CD.

Our new flow will look something like this:

Congratulations! You just set up what we call a CD pipeline.

And no… I don’t mean Colorful Daemons. I’m talking about Continous Deployments.

Continuous Deployments is a piece of software responsible for taking your app from something like GitHub all the way to your target environment where it finally gets deployed.

This is basically the CI/CD stuff you keep hearing about. When people talk about tools like Jenkins and CircleCI, they are usually referring to CI/CD.

What we just made with Colorful Daemons was a continuous deployments pipeline. Don’t confuse it with continuous integration or delivery. We’ll get to those some other day.

The DevOps Pattern

I guess you’ve already found a pattern here. We start with a process, find a section we aren’t happy with and then introduce some software component to simplify or automate it.

That’s getting the dev in ops. And that’s all that there is to it.

This is the real answer to the question, ‘What is DevOps?.’

Introducing Container Orchestration

Let’s finish up by making one small improvement.

Till now, we have been dealing with deploying our app to a single VM or a single node. What if we wanted to deploy our app to multiple nodes?

The easiest way to achieve this would be to modify our CD server to ssh into all the VMs and deploy our container to each one of them.

While this method works, we’ll need to change our script every time our infrastructure changes. In a world where applications are always autoscaling, and VMs are considered disposable, this is unacceptable.

A better way would be to make another HTTP server to track infrastructure changes. We can call this server ‘Pilot.’

This server will be responsible for performing health checks on the various VMs in our cluster to maintain a list of active VMs. It could even communicate with the cloud vendor to make things more robust.

Pilot will expose an endpoint as well to accept the details of the container to spawn. It can then talk to the various VMs to get the job done.

Now, our CD server can simply request Pilot instead of talking to each VM individually.

Our new flow will look something like this:

The second server, Pilot, is called a container orchestrator. That’s what Kubernetes is!

You just designed a mini version of Kubernetes!

Also, Kubernetes is greek for Pilot. Isn’t that a pleasant co-incidence?

Where to Start?

We covered quite a few tools together. This brings me to my last point. Ever wondered why the DevOps space is so fragmented?

If you think about it, there are so many tools out there, making it hard to decide: what’s the right choice or where you should even start?

Every organisation has its own way, its own process to do things. And since their paths are different, the tools they need to use are different.

Your job is not to find which tool is the best. Your job is to find what process works for you best. Once you have that figured out, the tools are just a google search away.

So now you know where to start. It’s not with the tools out there.

Start by understanding how your company and teams do things.

I’m literally asking you to open up a Word document and copy-paste the commands you need to run to do stuff.

Wrapping Up

I hope this post has been helpful in understanding how the DevOps field is arranged and how different tools depend and coexist with each other.

I’d like to add:

Your DevOps process is only as strong as its foundation.

So work on the underlying process. It’s okay if you need to tweak your current process a bit.

An excellent foundation to build upon could be using tools like SpaceCloud. Space Cloud is a Kubernetes-based serverless platform that helps you develop, deploy and secure cloud-native applications.

In a nutshell, SpaceCloud gives you an excellent starting point to build your DevOps practices on top of. It makes performing rolling upgrades, canary deployments, and autoscaling your applications easy. You can configure everything using the space-cli or REST APIs.

The post What is DevOps? appeared first on Tech Chronicles.

Configuring Jenkins To Build WebGoat

We’re going to scan a known vulnerable webapp, WebGoat, which is an OWASP project used for learning basic web penetration testing skills and vulnerabilities. A good scanner should find a lot of things!

A quick note: We were initially going to use Mutillidae, another vulnerable app written in PHP. However we couldn’t find any good open source PHP Static analyzers that would catch the vulnerabilities.

Anyway, let’s get on with Jenkins. Navigate in your browser to http://localhost:8080 and enter the admin password shown in the terminal running docker. Go ahead and install the default plugins (for a deployed instance, I would recommend only installing plugins you will actually use) and create your first admin user.

WebGoat requires Java 11 to build, which Jenkins won’t install automatically. Head over to the main page -> Manage Jenkins -> Global Tool Configuration. There are two sections here we will update now: JDK installation and Maven installations. We need to add a link to a Java 11 installer – we used https://download.java.net/java/GA/jdk11/13/GPL/openjdk-11.0.1_linux-x64_bin.tar.gz and we can use the default maven. Your config should look like this:

We sometimes see Jenkins have trouble installing a JDK this way if more than one JDK is installed in the system. If this is the first one, there should be no problems.

Finally, we have to set the JAVA_HOME variable. In the Jenkins -> Manage Jenkins -> Configure System menu, enable environment varaibles and set JAVA_HOME equal to /var/jenkins_home/tools/hudson.model.JDK/openjdk11/jdk-11.0.1/.

Now let’s create a pipeline for WebGoat and make sure it builds successfully. Back on the main page choose new item -> freestyle project.

The initial setup is pretty simple:

• Add Webgoat to the various github setting locations (https://github.com/WebGoat/WebGoat/)
• Set the target branch to */develop
• Create a maven build step (“Invoke top level maven targets”) and give it the command “clean install”

Here is the full pipeline configuration:

Try running it and making sure that everything builds successfully.

Adding SonarQube and DependencyCheck

SonarQube setup & security

We already have a SonarQube instance running, we just need to link and configure Jenkins to use it. Log in to http://localhost:9000 and use the default sonarQube login of admin/admin.

Although this is only for practice, I still want to secure our SonarQube instance, so do the following:

• Change the admin password
• Go to administration-> security and turn on “Force user authentication”
• Create a new user for Jenkins.
• Log into the new user, go to the profile -> security section, and generate a token. Copy this for later use.

Finally, create a project named “webgoat” with your jenkins user.

Configure the plugins for Jenkins

We will need two new plugins for jenkins. In the Jenkins home page, go to Mange Jenkins -> Manage Plugins. On the Available tab find and select “OWASP Dependency-Check Plugin” and “SonarQube Scanner for Jenkins”. Install them without restarting.

Back on the Jenkins home, go to Manage Jenkins -> Global Tool Configuration. You should see a new option for SonarQube Scanner. Add an installation here (I just chose the latest from Maven Central) and save.

Finally, head over to Jenkins -> Manage Jenkins -> Configure System and add a sonarqube instance. The URL with our docker container is http://sonarqube:9000 and the token should be the one you saved while setting up the Jenkins user in SonarQube. Here is my setup:

One other thing I had to do to get SonarQube working properly. For some reason I couldn’t completely determine, the SonarQube startup script was truncating the JAVA_HOME path incorrectly, causing errors during the pipeline. To solve this, log into the docker container manually and update the sonar script to the proper JAVA_HOME.

$ docker exec -it secure_pipeline_jenkins_1 bash
jenkins@2ea0acb5905d:/$ cd /var/jenkins_home/tools/hudson.plugins.sonar.SonarRunnerInstallation/sonarqube
jenkins@2ea0acb5905d:~/tools/hudson.plugins.sonar.SonarRunnerInstallation/sonarqube$ head bin/sonar-scanner
#!/bin/sh
#
# SonarQube Scanner Startup Script for Unix
#
# Optional ENV vars:
#   SONAR_SCANNER_OPTS - Parameters passed to the Java VM when running the SonarQube Scanner
#   SONAR_SCANNER_DEBUG_OPTS - Extra parameters passed to the Java VM for debugging
#   JAVA_HOME - Location of Java's installation

JAVA_HOME="/var/jenkins_home/tools/hudson.model.JDK/openjdk11-remote/jdk-11.0.1"

Add SonarQube and DependencyCheck to the pipeline

Now we can add these to our pipeline and start scanning with every build.

In the pipeline created earlier, add two new build steps – Invoke Dependency-Check analysis and Execute SonarQube Scanner. In the SonarQube scanner, add the configuration settings required – the project key and name should match the project you created in SonarQube.

sonar.projectKey=webgoat
sonar.projectName=webgoat
sonar.projectVersion=1.0
sonar.language=java
sonar.java.binaries=**/target/classes
sonar.exclusions=**/*.ts

I am excluding the TypeScript files above since we did not setup Node or a JS build step for our project. In a real project, we would want to ensure that they were also scannable.

In the DependencyCheck advanced section, check to generate HTML reports as well for easier viewing.

Here is my full pipeline configuration now:

Kick off a build and make sure it runs correctly. Afterwards, you should be able to see results.

Viewing Reports

If all runs successfully, logging into SonarQube will show you security scan details (with plenty of findings!) and the pipeline can show you the dependencyCheck results in the workspace -> dependency-check-report.html file.

You can and should at this point consider additional SonarQube plugins (or other SAST tools) that are specifically for your languages and frameworks.

Breaking the Build

We want to know when something isn’t working right at the build phase. SonarQube gives us this for free with the plugin (you should see a nice red ERROR tag under the SonarQube Quality gate) but DependencyCheck requires one more configuration.

Add a post-build check for “Publish Dependency Check Results” and expand the advanced tabs. Just add some threshold data and the build will fail or be marked unstable according to the rules set.

Here  is our final pipeline configuration, fully expanded.

Final Thoughts

Getting a CI/CD pipeline running with some basic security checks can be done within a few minutes. This will help keep your published artifacts in better shape and ensure the team has an opportunity to learn about security issues as soon as they emerge.

The post Creating a Secure Pipeline: Jenkins with SonarQube and DependencyCheck appeared first on Tech Chronicles.

Anteater is a CI/CD Security Gate Check Framework to prevent the unwanted merging of nominated strings, filenames, binaries, deprecated functions, staging environment code/credentials etc.

It’s main function is to block content based on regular expressions.

Anything that can be specified with regular expression syntax, can be sniffed out by Anteater. You tell Anteater exactly what you don’t want to get merged, and anteater looks after the rest.

How Anteater CI/CD Security Gate Check Framework Works

If Anteater finds something, it exits with a non-zero code which in turn fails the build of your CI tool, with the idea that it would prevent a pull request merging. Any false positives are easily negated by using the same RegExp framework to cancel out the false match.

Entire projects may also be scanned also, using a recursive directory walk. With a few simple steps, it can be easily implemented into a CI/CD workflow with tooling such as Travis CI, CircleCI, Gitlab CI/CD and Jenkins.

Anteater also provides integrates with the Virus Total API, so any binaries, public IP addresses or URL’s found by Anteater, will be sent to the Virus Total API and a report will be returned. If any object is reported as malicious, it will fail the CI build job.

You can also set it to block all binaries or tamper with existing binaries (this includes PDFs, Images etc.) and you can whitelist desired binaries using a SHA256 checksum.

Using Anteater CI/CD Security Gate Checks

There is some excellent documentation for Anteater here:

Docs » Anteater – CI/CD Gate Check Framework

This includes how to get it working with CircleCI which is my personal choice for CI tooling.

In order to use the VirusTotal API, you will first require an API key. These are free to get and can be obtained by signing up to the service here.

Once you have your key, it needs to be set as an environment variable.

You can download Anteater here:

anteater-master.zip

Or read more here.

The post Anteater – CI/CD Security Gate Check Framework appeared first on Tech Chronicles.