Here’s a brief tutorial to understand and get started with DevOps.

DevOps… CI/CD… Docker… Kubernetes… I’m sure you’ve been bombarded with these words a lot the past year. Seems like the entire world is talking about it. The rate at which this segment is progressing, it won’t be long before we reach the stage of NoOps. It’s about time we break down what DevOps really is.

The objective of this article is to set a solid foundation for you to build on top of. So let’s start with the obvious question.

What is DevOps?

DevOps is the simplification or automation of established IT processes.

I’ve seen so many people start this journey to adopt DevOps to only find themselves lost. There seems to be a pattern to this.

It usually starts with a video on how a fancy tech startup has automated its release cycle. Deployments happen automatically once all the tests pass. Rollbacks, in case of failures, is automatic. Thousands of simultaneous A/B test is driving up customer engagement.

We all are tired of releasing a new version like it’s a rollercoaster ride.

Unfortunately, DevOps doesn’t work that way. DevOps isn’t a magic wand that can solve all your problems in an instant.

Instead, it is a systematic process of choosing the right tools and technology to get the job done.

All of This Starts With a Process

It doesn’t matter what the process is. It could be simplifying the deployment of your app or automate testing. Whatever your process is, the act of making your life easier is what DevOps is all about.

But you always need to start with a process.

In fact, if your process cannot be done manually (on a smaller scale), you should probably re-examine your process.

I mean it.

Enough talking. Let’s take a real-world example to understand things better.

Let’s Take a Real DevOps Example

Let’s take a simple example of making a Nodejs app live on a VM in the cloud.

The Process

Here’s what our process looks like:

• Start with the source code: This is our source of truth. We can run our process from anywhere as long as we have access to the source code.
• Build an Artifact: We then package our source code to build an Artifact. In the case of a compiled language, the compiled output (JAR file, in the case of JAVA) would be our artifact. In our case, our source code itself is the artifact to be released.
• Publish to an Artifact Repository: Next, we push our artifact to a repository. This is a location from where our target environment can pull the artifact from. We could stick with something like Github since we are working with source code here.
• Pull and run your app: Finally, we pull the artifact onto our VM and schedule a Nodejs process by running npm start.

It’s okay if you do things a slightly different way. We are here to focus on the journey and not the destination.

Our First DevOps Project

Let’s not do anything fancy here.

The easiest way to automate this process would be to write a simple shell script to run all the commands in sequence.

Congratulations!!! That’s our first DevOps project!!!

I know shell scripts sounds too simple to be taken seriously. I suspect you already have such scripts in place. But believe me, that’s DevOps!

Don’t worry; we will get to the fancy stuff in a minute. But it’s important to understand that this is how DevOps works.

Importance of Repeatability

Let me ask you one question. Which one of these would you prefer?

• An automated deployments pipeline which works 60% of the time, or,
• A boring shell script that gets the job done every time it’s executed.

If you have dealt with production failures in the middle of the night, you’ll choose the shell script.

The reason is simple.

Reliability is far more critical than the degree of automation.

In other words,

A DevOps process must be able to produce consistent results every time it’s run.

Making Our Process Repeatable

Let’s take the example of our shell script.

Currently, our shell script depends on Node.js to be installed on the VM we want to deploy the app to.

What would happen if the Nodejs runtime was missing? These days, an incorrect version of the runtime is enough to break our application.

This problem only gets worse in polyglot environments where we deal with multiple programming languages.

A simple solution would be to archive the Nodejs runtime along with our source code in a zip file. The zip file can then be sent to the VM. This way, the VM can use the local Nodejs runtime present in the archive to run our app.

Luckily, there is a tool to make our lives easier.

In Comes Docker and Containers

If you are new to this, think of Docker as a way to package your artifact along with all its OS dependencies, including Nodejs, into a container image.

Using containers, we can deploy any application on a VM which has Docker installed.

With Docker, our flow will look something like this:

There is a lot more to containers than just this. However, this was one of the reasons why containers got so popular.

Docker Vs. Containers

Let me clarify this. Docker and containers are not the same things anymore.

Docker is a set of utility tools to build and ship container images which container runtimes like containerd use to make and run containers.

Many are concerned about the future of Docker, given the recent events which have taken place.

It is important to understand that Docker is not going anywhere anytime soon. It provides the best DX and will continue to play a major role in building and shipping container images.

Getting Serious With DevOps

We have made some serious progress already. Hopefully, we understand how Docker fits into the DevOps process.

It’s time to take things to the next level.

Triggering Deployment Based on Events

Our script looks pretty solid, but it’s still triggered manually.

Wouldn’t it be great if we could trigger this script automatically whenever someone pushes code on GitHub? In other words, we want to trigger this script on an event.

GitHub can invoke webhooks on a certain set of events.

To achieve this, we need to make a simple HTTP server that executes our shell script whenever its endpoint is hit. We can configure GitHub to hit our endpoint on the Push Event.

Let’s call this server Colorful Daemons or CD.

Our new flow will look something like this:

Congratulations! You just set up what we call a CD pipeline.

And no… I don’t mean Colorful Daemons. I’m talking about Continous Deployments.

Continuous Deployments is a piece of software responsible for taking your app from something like GitHub all the way to your target environment where it finally gets deployed.

This is basically the CI/CD stuff you keep hearing about. When people talk about tools like Jenkins and CircleCI, they are usually referring to CI/CD.

What we just made with Colorful Daemons was a continuous deployments pipeline. Don’t confuse it with continuous integration or delivery. We’ll get to those some other day.

The DevOps Pattern

I guess you’ve already found a pattern here. We start with a process, find a section we aren’t happy with and then introduce some software component to simplify or automate it.

That’s getting the dev in ops. And that’s all that there is to it.

This is the real answer to the question, ‘What is DevOps?.’

Introducing Container Orchestration

Let’s finish up by making one small improvement.

Till now, we have been dealing with deploying our app to a single VM or a single node. What if we wanted to deploy our app to multiple nodes?

The easiest way to achieve this would be to modify our CD server to ssh into all the VMs and deploy our container to each one of them.

While this method works, we’ll need to change our script every time our infrastructure changes. In a world where applications are always autoscaling, and VMs are considered disposable, this is unacceptable.

A better way would be to make another HTTP server to track infrastructure changes. We can call this server ‘Pilot.’

This server will be responsible for performing health checks on the various VMs in our cluster to maintain a list of active VMs. It could even communicate with the cloud vendor to make things more robust.

Pilot will expose an endpoint as well to accept the details of the container to spawn. It can then talk to the various VMs to get the job done.

Now, our CD server can simply request Pilot instead of talking to each VM individually.

Our new flow will look something like this:

The second server, Pilot, is called a container orchestrator. That’s what Kubernetes is!

You just designed a mini version of Kubernetes!

Also, Kubernetes is greek for Pilot. Isn’t that a pleasant co-incidence?

Where to Start?

We covered quite a few tools together. This brings me to my last point. Ever wondered why the DevOps space is so fragmented?

If you think about it, there are so many tools out there, making it hard to decide: what’s the right choice or where you should even start?

Every organisation has its own way, its own process to do things. And since their paths are different, the tools they need to use are different.

Your job is not to find which tool is the best. Your job is to find what process works for you best. Once you have that figured out, the tools are just a google search away.

So now you know where to start. It’s not with the tools out there.

Start by understanding how your company and teams do things.

I’m literally asking you to open up a Word document and copy-paste the commands you need to run to do stuff.

Wrapping Up

I hope this post has been helpful in understanding how the DevOps field is arranged and how different tools depend and coexist with each other.

I’d like to add:

Your DevOps process is only as strong as its foundation.

So work on the underlying process. It’s okay if you need to tweak your current process a bit.

An excellent foundation to build upon could be using tools like SpaceCloud. Space Cloud is a Kubernetes-based serverless platform that helps you develop, deploy and secure cloud-native applications.

In a nutshell, SpaceCloud gives you an excellent starting point to build your DevOps practices on top of. It makes performing rolling upgrades, canary deployments, and autoscaling your applications easy. You can configure everything using the space-cli or REST APIs.

The post What is DevOps? appeared first on Tech Chronicles.

The post Git Commands to Live By – Beyond Git basics appeared first on Tech Chronicles.

The command palette is one of the most foundational of all VS Code shortcuts. It’s the shortcut that gives you access to all of the functionality of VS Code.

MacOS: Command + Shift + P
Windows: Ctrl + Shift + P

2. Duplicate Line Down/Up

This is a shortcut I use a lot in my workflow to boost productivity. Basically, you can take a line of code and duplicate it on the line above or below. This is helpful with single lines like <li> tags and also with whole blocks of code like a CSS block or JS function.

MacOS: Shift + Option + Up / Shift + Option + Down
Windows: Shift + Alt + Up / Shift + Alt + Down

3. Move Line Down/Up

Similar to the shortcut above, but this moves the line of code up or down. Again helpful with single lines like reordering <li> tags as well as blocks of code.

MacOS: Option + Up / Option + Down
Windows: Alt + Up / Alt + Down

4. Indent / Outdent Line

Another of the very helpful VS Code shortcuts and one to memorize. When you copy and paste something in Visual Studio Code, it doesn’t always line up right (actually it rarely does). This shortcut allows you to indent or outdent a line or even an entire block of code. 

MacOS: Command + ] / Command + [
Windows: Ctrl + ] / Ctrl + [

5. Multi-Select Selection

So you decide to change all of the H3 tags on your template to H4 tags. So you go through and select each one and change it? No. You multi-select them all and change them all in one take.

In that example you would highlight just one H3 tag, and as you hit the shortcut below it goes through and selects the other instaces of that tag, providing you an active cursor with each one.

Of all the VS Code shortcuts this is one of my favorites and will boost your productivity.

MacOS: Command + D (to select next occurrence(s) after you make an initial selection)
Windows: Ctrl + D (to select next occurrence(s) after you make an initial selection).
* There is a shortcut to select ALL occurrences, but I like to go through them one by one to be sure I want to change that one.

6. Multi-Select Cursor

Similar to the above, but instead of selecting the next occurrences of a selection, you can place your cursor in multiple locations, and make edits to all and one time.

MacOS: Option + Click
Windows: Alt + Click

7. Toggle Line Comment

So we use line comments to “disable” a line or block of code or HTML or CSS right?

This Visual Studio Code shortcut allows you to disable a line or block of code by commenting it out. You can also use it again to re-enable that line or block of code by removing the comment markup.

MacOS: Command + /
Windows: Ctrl + /

8. Show / Hide Terminal

I almost exclusively use the VS Code Integrated terminal and rarely ever the OS Terminal. So of all the VS Code shortcuts, this is especially useful for me and for you if you do the same.

MacOS: Control + ̀
Windows: Ctrl + ̀

9. Add Line Comment

Adding a line comment to HTML and CSS is somewhat cumbersome to type out. 

Thankfully VS Code has a shortcut to pop it in there for you. Use your right hand to hit the K and left to this the C while holding Command.

MacOS: Command + K Command + C
Windows: Ctrl + K Ctrl + C

10. Toggle Sidebar

Of all the VS Code shortcuts, this is another major one for me. I don’t like that sidebar there when I’m writing code. At the same time, when I need it back, I want it back with ease.

MacOS: Command + B
Windows: Ctrl + B

More Info

Visual studio code can be downloaded at their website. It’s my favorite code editor and I use it exclusively. VS Code is also a full IDE, is packed with loads of features, extensions, is free, and thankfully open source.

Conclusion

Take the time to memorize these VS Code extensions and incorporate them into your workflow. You will surely find a boost in your productivity and your efficiency as a programmer.

The post 10 Visual Studio Code Shortcuts that will boost your productivity appeared first on Tech Chronicles.

The term reverse engineering, refers to the disassembling of an object, following a thorough examination of its composition/construction so to understand how it works to duplicate or upgrade the object.

Taken by older industries, the practice of reverse engineering is now widely used in the software engineering sector.

In this digital-information age, reverse engineering has become a tool that can be used as a way to create compatible products that are cheaper than the existing ones or even free in some cases, uniquely modify the software, and exchange knowledge as a result into making better, more reliable and secure products.

Can be applied to various aspects of both software, and hardware development to understand how they behave under various conditions, to retrieve the source code that was lost, fix issues, to adapt existing software programs with new hardware, etc.

The use of reverse engineering is also greatly exercised to identify malicious content in the source code of a software, such as viruses, or to expose security flaws(backdoors, virus, misconfigurations) and address possible privacy issues.

Researchers can also use this technique to reverse engineer malware to understand how it works to nullify its properties, identify the potential owner, and use the knowledge gained to update their virus databases and prepare mitigation measures for future malware attacks.

Legalities of Reverse Engineering

The law is not discouraging from taking apart the products available from almost any of the technologies that exist, including electronic, chemical, mechanical, software engineering, etc.

 Although reverse engineering a prototype or a source code of a program prior to its release will result in legal consequences of its proven to be so.

Patent law is one of the laws that protect inventions where it prevents other parties from copying an invention, to reverse engineer it. 

 In return, if the product is disclosed as a patent, the developer must reveal all the technical details.

The supreme court of the United States of America though, established standards that prevent many software inventions from being eligible in the first place for patent protection.

Copyright law protects software from both direct copying and close paraphrasing. The law protects details like software graphics, interface design, file structure, and organization, etc.

The copyright law also protects the software code from being reconstructed, a third party can break the law when copying the key elements of the original software, even if it doesn’t include the original code line by line.

However, the developer has to disclose the source code as a part of the registration process.

A downside of this law is when a case is opened about potential copyright violation, the developer must be able to prove that the copier had access to its source code while proving that the copied code is similar to his.

Reverse Engineering Tools Overview

Reverse engineering tools are a must for the “library” of a hacker, software developer, and a security researcher.

 Using reverse engineering, hackers can compromise any security system, the use of those reverse engineering programs can allow them to manipulate data into a useful form, thanks to the development of digitizing devices.

Reverse engineering to be achieved at its highest needs the appropriate knowledge combined with the proper tools.

There are a variety of reverse engineering tools and can be divided into categories for their various uses which can be tracking the application running in real times, dissect binary codes into assembly codes, view, and edit binaries or embedded resources in EXE files, etc.

Disassemblers and Decompilers

When a user knows what he/she is dealing with, meaning what programming language and compiler the intended software uses can start analyzing it, their task is to analyze compiled, binary file and display its source code’s structure in such a way a human can understand it.

By extracting strings, functions, libraries, etc., a user can know what fragments of code references to them and what functions of the operating system are used by the program along with what functions are exported.

Disassemblers can show us what the object code of the program in the form of x64 or x86, by analyzing what high-level language the program is compiled to.

Decompilers use is for recreating the original high-level code from the code that the program is compiled.

Debuggers

Debuggers by supporting CPU registers, hex dumping of programs, and more, can help programmers to track application running in real-time, observe how certain instructions affect the contents of memory, edit assembly codes in real-time and detect possible errors/”bugs”.

When the program is open source, debugging a high-level code can be easy compared to when you don’t have access to the source code.

Dedicated debuggers performing advance analysis of binary application structures can solve this issue but, in the end, their use requires knowledge of low-level languages as well as the functioning of the processor for which the program was compiled to.

Hex Editors

Can allow the developer to view or edit binaries in order to make corrections or fixes etc., according to software requirements.

Hex editor makes it possible to manipulate the fundamental binary data that makes up a computer file.

There is a variety of hex editors base on their different functions and applications, some allowing them to visually display the internal structure of a file.

Resource editors

Windows applications resources like icons, images, localized texts, version information, etc., can be saved in PE files within the resources area.

Resource editors allow the view and edit of the resources that are embedded in the EXE file.

As all application files are saved in an EXE or DLL format, when a developer needs to change some data the application, as long as their size remains unchanged, you can edit them using the hex editor.

When their size (bigger images, longer text) needs to be changed or there is a need to add new data, is then where a resource editor comes in use for the developer.

Identifiers

Identifiers can help when we are not so sure about how the intended program is created, it can distinguish features lie, section names, imported libraries, etc.

Using the identifiers analysis, which includes a signature base to identify compilers, cryptographic libraries, security systems, etc., developers can decide what their next step should be(e.g. unpacking the application).

Virtual environments

Can be used to analyze unknown/suspicious software programs.

By running the unknown program in an uncontrolled environment, can cause irreversible damage if the program runs a payload in the background, so the use of a virtual environment, is a must when we are dealing with an unknown software so we can be safe running and analyzing it first in that environment.

Reverse Engineering Hacking Tools

Ghidra

In 2017, Wikileaks released Vault 7, a substantial collection of material about CIA/NSA cyber-activities along with series of documents, Ghidra became known.

In 2019, NSA officially released the source code of the reverse engineering framework which was developed in the US.

Its suite includes tools for analyzing compiled code on various platforms like Windows, Mac OS, and Linux.

It can be run in both user-interface or command line mode, while It’s GUI is designed for less expert users and features assembler, disassembler, decompiler, and other features including processor instruction sets and executable formats.

Its headless mode enables reverse engineer at scale or if it is used as a server to enable group collaboration when dealing with larger binaries.

A basic programming experience, along with some knowledge of assembly language is required.

When decompiling a code, if you select a portion of the assembly, it automatically highlights in the decompiler window the decompiled code, providing a good way of understanding how high-level code matches the disassembled code.

Getting Started with Ghidra

Ghidra, in order to run, requires Java along with Java SE Development Kit 11 to be installed on the intended device.

Upon opening the program Ghidra provides TIP of the day that can be useful for new users or even experienced users, providing with something of new information about the usage of the program.

Features of Ghidra

Context Help

Ghidra comes with a contextual menu, by hovering over the most interface elements and pressing F1, a pop-up window with the help menu appears providing the user with more information about the element.

Organize project sections

Ghidra can organize your project sections of disassembly code in various ways, just by hitting right click on the folder of your project, select “Modularize By” and choose between “Subroutine”, “Complexity Depth” or “Dominance”.

The next window under “Program Trees” is “Symbol Tree” which enables viewing import, export, functions, labels, classes, and namespaces of a binary file.

Listing Window

“Listing” window. Here you can see the reverse-engineered code. 

Users can configure the listing fields by clicking on the icon “Edit the listing fields” in the top right corner and then the “Instruction/Data” tab. 

Any element of the listing interface may be changed, relocated, disabled, or removed.

Loading an executable

Support drag and drop function, a file can be loaded by dropping it into the projected window of Ghidra, launching a dialog box where a format is selected, destination folder, and the name of the program.

Import results summary information appears once the file is imported.

If the file is not analyzed, a list of Analyzers will appear in order for the user to enable various analyzers depending on the format of the file.

Modifying Display Elements

By using CodeBrowser for reviewing the target file, Ghidra offers customizable display elements(where it can help to enhance readability for the user) and various options where can be accessed by clicking edit on the top menu, and then selecting tool options.

Suggested environment changes:

Listing Display: Can increase the font size and enable bold formatting for easier reading.

Listing Fields – Bytes Field: Change “Maximum Lines to Display” to 1 to simplify spacing between lines of assembly code.

Listing Fields – Cursor Text Highlight: “Mouse Button to Activate”, change to left.

It will highlight all instances of the selected text when the left mouse button is clicked — similarly to other disassemblers.

Listing Fields – EOL Comments Field: Check “Show Semicolon at Start of Each Line” to better separate the assembly text from inserted comments

Listing Fields – Operands Field: Check “Add Space After Separator” for improved text readability

View Decompiler Output

Ghidra comes with a built-in decompiler output.

 It can display the high-level language of the assembly code.

By highlighting one of the operators in the high-level language decompiler window, it highlights the relevant assembly providing the user with a good idea on how and which groups of the assembler instructions match the high-level instructions.

Scripting

Ghidra includes support for writing Java and Python (via Jython) scripts to automate analysis. 

To view built-in scripts, go to Window – Script Manager.

A user can add its own script by choosing the “create a new script” option in the script manager window top header menu.

It supports scripting with Java and Python.

Investigate a String Reference
 
Ghidra gives a review of the strings embedded within a target file.
To navigate, click on Window – Defined Strings. 
 
Clicking on the row associated with a string populates the Listing window with the data on the intended address.
 
To identify references to a string, the user should right-click in the blue area in the listing window – References – Show References to Address:

 The window can show how many references are to a particular string.

 Function call Graph

The function call graph displays the relationships between functions and provides a high-level overview of function calls.
 

Binary Ninja

 Binary Ninja is a reverse engineering tool, it’s a program analysis platform and includes a disassembler, it’s not a debugger or a decompiler.
 
There Is a personal, commercial, and enterprise license to use this tool.
 
Commercial version provides the capability to use more cores on the machine that is running for making the process faster and also supports a headless mode where the user can create and automate scripts without loading the UI(in the personal license user can also access the same API’s but only through the UI).
 

Features of Binary Ninja

 
Cross-Platform
 
Works on OS X, Windows, or Linux.
 
Undo Function
 
Users don’t have to be worried about losing useful information accidentally, Edit — Undo.
 
Smooth Scrolling, Zooming
 
Scanning through large functions

Multi-threaded Disassembly
 
 This is included as mentioned above only in the commercial edition for multi-threaded analysis of large binaries files.
 

LowLevelIL
 
An easy to write to IL powers automatic analysis across all architectures.

Inline assembly editing
 
For fast binary patches.

Right-Click Patching
 
Faster than inline editing.

Handy Transforms
 
Born from the fast-paced CTF environment.

Simultaneous Live View
 
A feature that a user can write hex and immediately see the changes live in disassembly.

Bug Hunting

It good to be used for:

• • PE, Mach-O, ELF file formats.
• • Raw binaries executable code for x86,x64, armv7,armv8, MIPS, PPC, and others.
  • Firmware — some knowledge for assembly language is required

  And it’s not good for things that are not binary code like:
   
  • Web apps
  • Java
  • .NET
  • Virtual machines

  Visit the official documentation here.

Apktool
 
  A tool for reverse engineering Android APK files.
 
It can be used to reverse engineering 3rd party, closed binary Android Apps, and also can decode resources to the nearly original form and rebuild them.
 
It could be used for localizing, adding some features or support for custom platforms, analyzing applications, and more.
 
 
Apktool Features
 
• Disassembling resources to the nearly original form
 
• Rebuilding decoded resources back to binary APK/JAR
 
• Organizing and handling APKs that depend on framework resources
 
• Smali Debugging (Removed in 2.1.0 in favor of IdeaSmali)
 
• Helping with repetitive tasks
It can make working with an app easier because of this project like file structure and automation of some repetitive tasks like building APK, etc.
 

Example decoding APK Facebook lite
 
Using debug mode (d) to decode the given APK file
 

Upon decoding the files, after making changes or inspecting the APK, a user can build the decoded files using the build function (b) by selecting the file with the decoded files of the APK and an output name of APK (-o).

Javasnoop
 
 
It’s a default tool in the Kali Linux reverse engineering family tool and can intercept Java applications locally.
 It can be accessed from the terminal with a simple command — root@kali:~# javasnoop
Intercepts Java applications locally.

It’s a tool developed by Aspect Security to help people intercept Java function calls (e.g. toString) from Java applications.
The tool can allow the user to attach to a process (like a debugger) and intercept a Java function call, view and modify parameter values and print the stackrace or save function calls.
 

Features
 
The following features of the tool make the testing easier for any kind of Java-based apps.
 
• Allows easy interception of any method in the JVM
• Allows the editing of return values and parameters
• Allows custom Java to be inserted into any method
• Able to work on any type of Java application (J2SE, Applet, or Java Web Start)
• Able to work on already-running Java processes
• Not require any target source code (original or decompiled)
 

 Javasnoop’s interface functionality
 

The first box (top left)
 
Is where a user can select class or method to be hooked and intercepted.
The interface provides a button to add a new Hook so the user can add a method from a specific class available from the list. 
 

Second box(top right)
 
Provides features for setting various options for intercepting the method calls.
Set regular expression conditions for matching and intercepting traffic from the method calls.
 
On execution – Third box(bottom left)
 
The interface here helps in deciding what to do with a particular hook that the user selected from the first box.
 
Various options like:
 
• Printing the parameters/stacktrace on to the console or a particular file
• Running custom scripts
• Tampering with parameters
• Tampering with a return value
• Pausing program

Fourth box (bottom right)
 
Here in this area, it shows the output from the hooks and the decompiled classes from the target application. 

Benefits of reverse engineering:

 
Using creativity and knowledge, researchers can now innovate through reverse engineering, by understanding, taking apart, modifying, or even creating a new better software than before.
 
• Examining the structures and processes of an application can lead to product innovation.
• Knowledge gain by other researcher’s work
• Reconstructing a product that is outdated 
• The discovery of software vulnerabilities 
• The development of applications that are more efficient and cheap.
 

Importance of reverse engineering and the dark side of it.

In the security world, researchers use reverse engineering to find security risks in programs, to understand malicious applications, etc.
Cyber-criminals can also use this technique to exploit security bugs in applications, the difference with the security researchers is what they do with that vulnerability information.
 
In a report conducted on April 2019, found that 97% of the 30 mobile financial apps that were tested, were lacking binary protection, making decompiling and review of the source code possible.
 
Upon decompiling an app all kinds of sensitive information can be exposed, API URLs API keys hardcoded into the apps.
The URLs can lead attackers to nonstandard port numbers, development servers, private keys, application file directories that were used for testing by developers, quality assurance engineers and can lead an attacker to compromise entire applications.
 
Attackers use many anti-reverse engineering techniques to prevent detection or make it more difficult to recover the contents of their malicious apps.
Some of them are:
 Detection of virtual machines, sandboxes — used to study malware, crypters — to encrypt the executable files, 
exe-protectors — to compress executable and add code to detect the presence of debuggers and eventually hide or corrupt the true structure of the executable.
 
To counter reverse-engineering attacks, security teams need to know what tools are available, how they work, and lastly what attackers use for evading those tools.

The post Reverse Engineering appeared first on Tech Chronicles.

The post 7 Skills of Highly “Effective” Programmers appeared first on Tech Chronicles.

A list of the (in my opinion) best Cheatsheets and tutorials for some of the top languages, frameworks and tools out there. 

General

Awesome

001 https://github.com/topics/awesome
002 https://github.com/detailyang/awesome-cheatsheet

Tech interview

003 https://yangshun.github.io/tech-interview-handbook/cheatsheet/
004 https://github.com/TSiege/Tech-Interview-Cheat-Sheet
005 https://dev.to/elliot/my-google-technical-interview-cheat-sheet-1lbd

Complexity

006 https://www.bigocheatsheet.com/
007 http://cooervo.github.io/Algorithms-DataStructures-BigONotation/

SEO

008 https://dev.to/pagely/seo-cheat-sheet-for-devs-5h1g

Programming, Scripting and Query Languages

C

009 https://developerinsider.co/c-programming-language-cheat-sheet/

C++

010 https://www.educba.com/c-programming-language-basics/

Go

011 https://github.com/a8m/golang-cheat-sheet
012 https://devhints.io/go
013 https://dev.to/codehakase/how-i-learned-go-programming

GraphQL

014 https://devhints.io/graphql
015 https://raw.githubusercontent.com/sogko/graphql-shorthand-notation-cheat-sheet/master/graphql-shorthand-notation-cheat-sheet.png
016 https://dev.to/methodcoder/graphql-crash-course-in-10-pics-3b04

Java

017 https://overapi.com/java
018 https://cheatography.com/son9912/cheat-sheets/java-oop-concept/
019 https://www.educba.com/category/software-development/software-development-tutorials/java-tutorial/
020 https://dev.to/monknomo/java-8-stream-cheatsheet–oad

JavaScript, ES2015+

021 http://www.developer-cheatsheets.com/es6
022 https://htmlcheatsheet.com/js/
023 https://devhints.io/es6
024 https://overapi.com/javascript
025 https://dev.to/samanthaming/javascript-module-cheatsheet-5b4o
026 https://dev.to/samanthaming/es6-arrow-functions-cheatsheet-1cn

PHP

027 https://overapi.com/php
028 https://phpcheatsheets.com/
029 https://dev.to/biros/awesome-php-resources-on-dev—-3a18

Python

030 https://devhints.io/python
031 https://overapi.com/python
032 https://www.pythoncheatsheet.org
033 https://dev.to/codingsafari/python-cheat-sheet-for-javascripters-2c4m

Regex

034 https://www.debuggex.com/cheatsheet/regex/javascript
035 https://devhints.io/regexp
036 https://overapi.com/regex
037 https://dev.to/emmabostian/regex-cheat-sheet-2j2a
038 https://dev.to/catherinecodes/a-regex-cheatsheet-for-all-those-regex-haters-and-lovers–2cj1

Ruby

039 https://overapi.com/ruby
040 https://dev.to/teekay/idiomatic-ruby-writing-beautiful-code-56ef
041 https://dev.to/philnash/top-10-errors-from-1000-ruby-on-rails-projects-and-how-to-avoid-them-24m

SQL, MySQL

042 https://devhints.io/mysql
043 https://overapi.com/mysql

TypeScript

044 https://learnxinyminutes.com/docs/typescript/
045 https://devhints.io/typescript

Frameworks, libraries and their extensions

Angular

046 https://angular.io/guide/cheatsheet
047 https://dev.to/lysofdev/an-angular-testing-cheatsheet-5hj2

jQuery

048 https://oscarotero.com/jquery/
049 https://overapi.com/jquery

React

050 http://www.developer-cheatsheets.com/react
051 https://devhints.io/react
052 https://dev.to/codeartistryio/the-react-cheatsheet-for-2020-real-world-examples-4hgg

Redux

053 http://www.developer-cheatsheets.com/redux
054 https://devhints.io/redux
055 https://github.com/linkmesrl/react-journey-2016/blob/master/resources/egghead-redux-cheat-sheet-3-2-1.pdf

Vue.js

056 https://marozed.ma/vue-cheatsheet/
057 https://devhints.io/vue
058 https://dev.to/adnanbabakan/vue-cheat-sheet-1-194a
059 https://dev.to/adnanbabakan/vue-cheat-sheet-2-5aj8
060 https://dev.to/adnanbabakan/vue-cheat-sheet-3-advanced-4khj
061 https://dev.to/devmount/how-to-kickstart-a-vuejs-project-2mj5

Vuex

062 https://vuejs-tips.github.io/vuex-cheatsheet/
063 https://dev.to/napoleon039/when-why-and-how-to-use-vuex-9fl

Editors, IDEs and other tools

Atom

064 https://devhints.io/atom

Bash

065 https://devhints.io/bash
066 http://johnstowers.co.nz/pages/bash-cheat-sheet.html
067 https://dev.to/devmount/9-evil-bash-commands-explained-4k5e

Docker

068 https://devhints.io/docker
069 https://dev.to/emarsys/the-missing-docker-cheatsheet-4dbg

Git

070 http://www.ndpsoftware.com/git-cheatsheet.html
071 https://overapi.com/git
072 https://dev.to/maxpou/git-cheat-sheet-advanced-3a17
073 https://dev.to/duomly/git-cheatsheet-for-beginners-5apl
074 https://dev.to/antjanus/my-personal-git-tricks-cheatsheet-23j1
075 https://dev.to/hengnee/yet-another-git-cheatsheet-4gjk
076 https://dev.to/devmount/how-to-correct-git-commit-messages-402d
077 https://dev.to/devmount/signed-git-commits-in-vs-code-36do

Node.js

078 https://devhints.io/nodejs
079 https://overapi.com/nodejs
080 https://dev.to/santypk4/bulletproof-node-js-project-architecture-4epf
081 https://dev.to/jorge_rockr/everything-you-need-to-know-about-node-js-lnc

Sublime Text

082 https://devhints.io/sublime-text
083 https://shortcuts.design/toolspage-sublimetext.html

Vim

084 https://vim.rtorr.com/
085 https://devhints.io/vim
086 https://sheet.shiar.nl/vi

Visual Studio Code

087 https://devhints.io/vscode
088 https://code.visualstudio.com/shortcuts/keyboard-shortcuts-windows.pdf
089 https://code.visualstudio.com/shortcuts/keyboard-shortcuts-macos.pdf
090 https://code.visualstudio.com/shortcuts/keyboard-shortcuts-linux.pdf
091 https://dev.to/devmount/23-lesser-known-vs-code-shortcuts-as-gif-80

Markup and Styling

CSS, CSS3, Flexbox

092 https://htmlcheatsheet.com/css/
093 https://devhints.io/css
094 http://overapi.com/css
095 https://yoksel.github.io/flex-cheatsheet/
096 https://dev.to/proticm/css-variables-cheat-sheet-32id
097 https://dev.to/iggredible/css-selectors-cheatsheet-24bh
098 https://dev.to/devmount/learn-css-animation-by-creating-pure-css-loaders-3lm6
099 https://dev.to/devmount/8-games-to-learn-css-the-fun-way-4e0f

Emmet

100 https://docs.emmet.io/cheat-sheet/
101 https://devhints.io/emmet
102 https://dev.to/lalawuhan/emmet-love-4be5

HTML, HTML5

103 https://htmlcheatsheet.com/
104 https://devhints.io/html
105 http://overapi.com/html

JSON

106 https://cheatography.com/mackan90096/cheat-sheets/json/

LaTeX

107 https://wch.github.io/latexsheet/
108 https://dev.to/nickymarino/lets-use-latex

Markdown

109 https://www.markdownguide.org/cheat-sheet
110 https://devhints.io/markdown
111 https://dev.to/rattanakchea/markdown-cheatsheet-for-developers-2bjj

Pug

112 https://devhints.io/pug

SASS, SCSS

113 https://sass-cheatsheet.brunoscopelliti.com/
114 https://devhints.io/sass
115 https://dev.to/finallynero/scss-cheatsheet-7g6

Stylus

116 https://devhints.io/stylus

Xpath

117 https://devhints.io/xpath

Yaml

118 https://lzone.de//cheat-sheet/YAML
119 https://devhints.io/yaml

Platforms, web apps and code management

Github

120 https://github.com/tiimgreen/github-cheat-sheet
121 https://github.github.com/training-kit/downloads/github-git-cheat-sheet/

Gitlab

122 https://docs.gitlab.com/ee/user/markdown.html

Jupyter Notebook

123 https://s3.amazonaws.com/assets.datacamp.com/blog_assets/Jupyter_Notebook_Cheat_Sheet.pdf

NPM

124 https://devhints.io/npm
125 https://guide.freecodecamp.org/developer-tools/npm-cheatsheet/
126 https://dev.to/therealdanvega/creating-your-first-npm-package-2ehf

Yarn

127 https://devhints.io/yarn
128 https://infinite.red/files/yarn.pdf

The post A CheatSheet of 128 CheatSheets for Developers appeared first on Tech Chronicles.

HTTPX

Arrow

Python Fire

Starlette

Mypy

FastAPI

Immutables

Expiring Dict

VCR.py

Transformers

Modin

Dash

Detectron 2

Streamlit

Imbalanced-learn

PyTorch

The post 16 Python Libraries to ‘pip’ in 2020 appeared first on Tech Chronicles.

pip install tensorflow keras pickle nltk 

How do Chatbots work?

1. NLU (Natural Language Understanding): The ability of machines to understand human language like English.
2. NLG (Natural Language Generation): The ability of a machine to generate text similar to human written sentences.

Project file structure 

1. Train_chatbot.py – In this file, we will build and train the deep learning model that can classify and identify what the user is asking to the bot.
2. Gui_Chatbot.py – This file is where we will build a graphical user interface to chat with our trained chatbot.
3. Intents.json – The intents file has all the data that we will use to train the model. It contains a collection of tags with their corresponding patterns and responses.
4. Chatbot_model.h5 – This is a hierarchical data format file in which we have stored the weights and the architecture of our trained model.
5. Classes.pkl – The pickle file can be used to store all the tag names to classify when we are predicting the message.
6. Words.pkl – The words.pkl pickle file contains all the unique words that are the vocabulary of our model.

How to build your own chatbot?

import numpy as np
from keras.models import Sequential
from keras.layers import Dense, Activation, Dropout
from keras.optimizers import SGD
import random
import nltk
from nltk.stem import WordNetLemmatizer
lemmatizer = WordNetLemmatizer()
import json
import pickle
intents_file = open('intents.json').read()
intents = json.loads(intents_file)

words=[]
classes = []
documents = []
ignore_letters = ['!', '?', ',', '.']
for intent in intents['intents']:
    for pattern in intent['patterns']:
        #tokenize each word
        word = nltk.word_tokenize(pattern)
        words.extend(word)
        #add documents in the corpus
        documents.append((word, intent['tag']))
        # add to our classes list
        if intent['tag'] not in classes:
            classes.append(intent['tag'])
print(documents)

# lemmaztize and lower each word and remove duplicates
words = [lemmatizer.lemmatize(w.lower()) for w in words if w not in ignore_letters]
words = sorted(list(set(words)))
# sort classes
classes = sorted(list(set(classes)))
# documents = combination between patterns and intents
print (len(documents), "documents")
# classes = intents
print (len(classes), "classes", classes)
# words = all words, vocabulary
print (len(words), "unique lemmatized words", words)
pickle.dump(words,open('words.pkl','wb'))
pickle.dump(classes,open('classes.pkl','wb'))

# create the training data
training = []
# create empty array for the output
output_empty = [0] * len(classes)
# training set, bag of words for every sentence
for doc in documents:
    # initializing bag of words
    bag = []
    # list of tokenized words for the pattern
    word_patterns = doc[0]
    # lemmatize each word - create base word, in attempt to represent related words
    word_patterns = [lemmatizer.lemmatize(word.lower()) for word in word_patterns]
    # create the bag of words array with 1, if word is found in current pattern
    for word in words:
        bag.append(1) if word in word_patterns else bag.append(0)
    # output is a '0' for each tag and '1' for current tag (for each pattern)
    output_row = list(output_empty)
    output_row[classes.index(doc[1])] = 1
    training.append([bag, output_row])
# shuffle the features and make numpy array
random.shuffle(training)
training = np.array(training)
# create training and testing lists. X - patterns, Y - intents
train_x = list(training[:,0])
train_y = list(training[:,1])
print("Training data is created")

# deep neural networds model
model = Sequential()
model.add(Dense(128, input_shape=(len(train_x[0]),), activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(64, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(len(train_y[0]), activation='softmax'))
# Compiling model. SGD with Nesterov accelerated gradient gives good results for this model
sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])
#Training and saving the model 
hist = model.fit(np.array(train_x), np.array(train_y), epochs=200, batch_size=5, verbose=1)
model.save('chatbot_model.h5', hist)
print("model is created")

import nltk
from nltk.stem import WordNetLemmatizer
lemmatizer = WordNetLemmatizer()
import pickle
import numpy as np
from keras.models import load_model
model = load_model('chatbot_model.h5')
import json
import random
intents = json.loads(open('intents.json').read())
words = pickle.load(open('words.pkl','rb'))
classes = pickle.load(open('classes.pkl','rb'))
def clean_up_sentence(sentence):
    # tokenize the pattern - splitting words into array
    sentence_words = nltk.word_tokenize(sentence)
    # stemming every word - reducing to base form
    sentence_words = [lemmatizer.lemmatize(word.lower()) for word in sentence_words]
    return sentence_words
# return bag of words array: 0 or 1 for words that exist in sentence
def bag_of_words(sentence, words, show_details=True):
    # tokenizing patterns
    sentence_words = clean_up_sentence(sentence)
    # bag of words - vocabulary matrix
    bag = [0]*len(words)  
    for s in sentence_words:
        for i,word in enumerate(words):
            if word == s: 
                # assign 1 if current word is in the vocabulary position
                bag[i] = 1
                if show_details:
                    print ("found in bag: %s" % word)
    return(np.array(bag))
def predict_class(sentence):
    # filter below  threshold predictions
    p = bag_of_words(sentence, words,show_details=False)
    res = model.predict(np.array([p]))[0]
    ERROR_THRESHOLD = 0.25
    results = [[i,r] for i,r in enumerate(res) if r>ERROR_THRESHOLD]
    # sorting strength probability
    results.sort(key=lambda x: x[1], reverse=True)
    return_list = []
    for r in results:
        return_list.append({"intent": classes[r[0]], "probability": str(r[1])})
    return return_list
def getResponse(ints, intents_json):
    tag = ints[0]['intent']
    list_of_intents = intents_json['intents']
    for i in list_of_intents:
        if(i['tag']== tag):
            result = random.choice(i['responses'])
            break
    return result
#Creating tkinter GUI
import tkinter
from tkinter import *
def send():
    msg = EntryBox.get("1.0",'end-1c').strip()
    EntryBox.delete("0.0",END)
    if msg != '':
        ChatBox.config(state=NORMAL)
        ChatBox.insert(END, "You: " + msg + '\n\n')
        ChatBox.config(foreground="#446665", font=("Verdana", 12 ))
        ints = predict_class(msg)
        res = getResponse(ints, intents)
        ChatBox.insert(END, "Bot: " + res + '\n\n')
        ChatBox.config(state=DISABLED)
        ChatBox.yview(END)
root = Tk()
root.title("Chatbot")
root.geometry("400x500")
root.resizable(width=FALSE, height=FALSE)
#Create Chat window
ChatBox = Text(root, bd=0, bg="white", height="8", width="50", font="Arial",)
ChatBox.config(state=DISABLED)
#Bind scrollbar to Chat window
scrollbar = Scrollbar(root, command=ChatBox.yview, cursor="heart")
ChatBox['yscrollcommand'] = scrollbar.set
#Create Button to send message
SendButton = Button(root, font=("Verdana",12,'bold'), text="Send", width="12", height=5,
                    bd=0, bg="#f9a602", activebackground="#3c9d9b",fg='#000000',
                    command= send )
#Create the box to enter message
EntryBox = Text(root, bd=0, bg="white",width="29", height="5", font="Arial")
#EntryBox.bind("<Return>", send)
#Place all components on the screen
scrollbar.place(x=376,y=6, height=386)
ChatBox.place(x=6,y=6, height=386, width=370)
EntryBox.place(x=128, y=401, height=90, width=265)
SendButton.place(x=6, y=401, height=90)
root.mainloop()

Running the chatbot

python train_chatbot.py

The post How To Build a Chatbot Project Using Python appeared first on Tech Chronicles.

• Compiling (Parsing, Checks, and Semantics)
• Binding
• Optimising
• Executing

SQL STATEMENT --> Parsing -->Binding --> Query Optimization --> Query Execution --> Result

The post How SQL Database Engines Work appeared first on Tech Chronicles.

Configuring Jenkins To Build WebGoat

We’re going to scan a known vulnerable webapp, WebGoat, which is an OWASP project used for learning basic web penetration testing skills and vulnerabilities. A good scanner should find a lot of things!

A quick note: We were initially going to use Mutillidae, another vulnerable app written in PHP. However we couldn’t find any good open source PHP Static analyzers that would catch the vulnerabilities.

Anyway, let’s get on with Jenkins. Navigate in your browser to http://localhost:8080 and enter the admin password shown in the terminal running docker. Go ahead and install the default plugins (for a deployed instance, I would recommend only installing plugins you will actually use) and create your first admin user.

WebGoat requires Java 11 to build, which Jenkins won’t install automatically. Head over to the main page -> Manage Jenkins -> Global Tool Configuration. There are two sections here we will update now: JDK installation and Maven installations. We need to add a link to a Java 11 installer – we used https://download.java.net/java/GA/jdk11/13/GPL/openjdk-11.0.1_linux-x64_bin.tar.gz and we can use the default maven. Your config should look like this:

We sometimes see Jenkins have trouble installing a JDK this way if more than one JDK is installed in the system. If this is the first one, there should be no problems.

Finally, we have to set the JAVA_HOME variable. In the Jenkins -> Manage Jenkins -> Configure System menu, enable environment varaibles and set JAVA_HOME equal to /var/jenkins_home/tools/hudson.model.JDK/openjdk11/jdk-11.0.1/.

Now let’s create a pipeline for WebGoat and make sure it builds successfully. Back on the main page choose new item -> freestyle project.

The initial setup is pretty simple:

• Add Webgoat to the various github setting locations (https://github.com/WebGoat/WebGoat/)
• Set the target branch to */develop
• Create a maven build step (“Invoke top level maven targets”) and give it the command “clean install”

Here is the full pipeline configuration:

Try running it and making sure that everything builds successfully.

Adding SonarQube and DependencyCheck

SonarQube setup & security

We already have a SonarQube instance running, we just need to link and configure Jenkins to use it. Log in to http://localhost:9000 and use the default sonarQube login of admin/admin.

Although this is only for practice, I still want to secure our SonarQube instance, so do the following:

• Change the admin password
• Go to administration-> security and turn on “Force user authentication”
• Create a new user for Jenkins.
• Log into the new user, go to the profile -> security section, and generate a token. Copy this for later use.

Finally, create a project named “webgoat” with your jenkins user.

Configure the plugins for Jenkins

We will need two new plugins for jenkins. In the Jenkins home page, go to Mange Jenkins -> Manage Plugins. On the Available tab find and select “OWASP Dependency-Check Plugin” and “SonarQube Scanner for Jenkins”. Install them without restarting.

Back on the Jenkins home, go to Manage Jenkins -> Global Tool Configuration. You should see a new option for SonarQube Scanner. Add an installation here (I just chose the latest from Maven Central) and save.

Finally, head over to Jenkins -> Manage Jenkins -> Configure System and add a sonarqube instance. The URL with our docker container is http://sonarqube:9000 and the token should be the one you saved while setting up the Jenkins user in SonarQube. Here is my setup:

One other thing I had to do to get SonarQube working properly. For some reason I couldn’t completely determine, the SonarQube startup script was truncating the JAVA_HOME path incorrectly, causing errors during the pipeline. To solve this, log into the docker container manually and update the sonar script to the proper JAVA_HOME.

$ docker exec -it secure_pipeline_jenkins_1 bash
jenkins@2ea0acb5905d:/$ cd /var/jenkins_home/tools/hudson.plugins.sonar.SonarRunnerInstallation/sonarqube
jenkins@2ea0acb5905d:~/tools/hudson.plugins.sonar.SonarRunnerInstallation/sonarqube$ head bin/sonar-scanner
#!/bin/sh
#
# SonarQube Scanner Startup Script for Unix
#
# Optional ENV vars:
#   SONAR_SCANNER_OPTS - Parameters passed to the Java VM when running the SonarQube Scanner
#   SONAR_SCANNER_DEBUG_OPTS - Extra parameters passed to the Java VM for debugging
#   JAVA_HOME - Location of Java's installation

JAVA_HOME="/var/jenkins_home/tools/hudson.model.JDK/openjdk11-remote/jdk-11.0.1"

Add SonarQube and DependencyCheck to the pipeline

Now we can add these to our pipeline and start scanning with every build.

In the pipeline created earlier, add two new build steps – Invoke Dependency-Check analysis and Execute SonarQube Scanner. In the SonarQube scanner, add the configuration settings required – the project key and name should match the project you created in SonarQube.

sonar.projectKey=webgoat
sonar.projectName=webgoat
sonar.projectVersion=1.0
sonar.language=java
sonar.java.binaries=**/target/classes
sonar.exclusions=**/*.ts

I am excluding the TypeScript files above since we did not setup Node or a JS build step for our project. In a real project, we would want to ensure that they were also scannable.

In the DependencyCheck advanced section, check to generate HTML reports as well for easier viewing.

Here is my full pipeline configuration now:

Kick off a build and make sure it runs correctly. Afterwards, you should be able to see results.

Viewing Reports

If all runs successfully, logging into SonarQube will show you security scan details (with plenty of findings!) and the pipeline can show you the dependencyCheck results in the workspace -> dependency-check-report.html file.

You can and should at this point consider additional SonarQube plugins (or other SAST tools) that are specifically for your languages and frameworks.

Breaking the Build

We want to know when something isn’t working right at the build phase. SonarQube gives us this for free with the plugin (you should see a nice red ERROR tag under the SonarQube Quality gate) but DependencyCheck requires one more configuration.

Add a post-build check for “Publish Dependency Check Results” and expand the advanced tabs. Just add some threshold data and the build will fail or be marked unstable according to the rules set.

Here  is our final pipeline configuration, fully expanded.

Final Thoughts

Getting a CI/CD pipeline running with some basic security checks can be done within a few minutes. This will help keep your published artifacts in better shape and ensure the team has an opportunity to learn about security issues as soon as they emerge.

The post Creating a Secure Pipeline: Jenkins with SonarQube and DependencyCheck appeared first on Tech Chronicles.