Here is an adjusted version of our internal documentation at Nomad Interactive about our git workflow explained. We almost exclusively write javascript on all platforms (node.js back-end, react.js web front-end, and desktop, react native for mobile with minimal swift and android java for mobile apps). There are a lot of biased opinions and structures in this article that may be too javascript focused but it can be applied to other languages with equivalent solutions. 

We want to have a code base and development process with best practices that reflect ideal workflow and produce clean, understandable, testable. This article outlines few key goals we want to achieve with this ideal workflow. 

Continuous Integration Requirements (Pipelines)

Clean Code, Unified Developer Language

Development shouldn’t be built around a lot of rules or “musts”. Allowing developers to define their own style is important, but understandable code is one of the most important elements of an effective development team. It’s famously known that big teams (Google, Facebook, Airbnb…) utilizes a large set of rules under a good practice definition that allows every developer to speak the same language and be able to adapt to each other’s code and process very easily.

In order to do that, we will follow similar practices to have our javascript code to be unified in a single style. Few tools we will start utilizing to achieve this.

REQUIREMENT #1: WRITE JS in ES6

We will have a good practice of writing modern ES6 and in this case, we have to, use transpilers (babel, webpack) to compile our code to VanillaJS for plain javascript rendering.

REQUIREMENT #2: Clean code enforced using ESLINT

We inherit Airbnb’s javascript guidelines and write javascript/JSX in the same style. We have built our own eslintrc and maintain that in a separate repository. Our eslint configuration is packaged in an npm package and our project eslint configurations are generally just a few lines and inherit all rules from this package: https://www.npmjs.com/package/@nomadinteractive/eslint-config

We will set up our continuous integration server to only accept error/warning-less code commits.

REQUIREMENT #3: 100% Test passing

We don’t want to worry about breaking stuff, spend time on unnecessary human energy to make sure we cover all regression tests within our development iterations. For this, we need to embrace some form of test-driven development. For each environment and often per-project specific test approach. But we generally do unit testing via jest on our express back-end apps (mostly APIs), some unit testing on web apps but mostly e2e test with puppeteer, and some unit + e2e using detox on iOS and Android apps.

We don’t have any code coverage requirements yet.

Branching and Tagging Practices

• We always lock the master branch for all users except the CI server, locked for pushes and merge requests as well.
• Dev branch is also locked for direct commit/pushes. Only pull request and merge requests are going to be allowed.
• Development code is pulled or merged to the “dev” branch when the CI pipeline succeeds.
• Every developer works on their own local development branches per feature, fix, or other categorized with uniform branch naming strategy.
• We should avoid having long-living remote branches other than the protected branches that are specific to a developer – at all costs.
• When it’s time to push your changes to a build or the latest development branch where all developers push their branch remote, then merge their code. You can create merge requests to the dev branch from your branch easily on GitLab or GitHub UI or from the command line. Then, the CI server picks up your commit, makes sure your code is clean and acceptable, then it approves and finishes merge to the dev branch. You continue working on your local branch and repeat this process.

Diagram below demonstrates the branching and development workflow.

Automated deliveries

Shipment of the code has to be automated in all cases from development to production submissions. We need to utilize all code distribution methods to automate this workflow.

We currently use Test Flight for iOS apps, Firebase Beta for Android apps.
Obviously, web-based apps (both front-end and back-ends) deploy to their respective environments via CD automatically.

For mobile apps (react native), we also utilize Microsoft Code Push for over the air updates for react native apps aside from the bundled native builds to be distributed in the traditional ways mentioned above. This allows us to ship smaller builds faster.

Merge Request Approvals – Code Reviews

We conduct code reviews, every merge request to be assigned to another developer for them to review and approve the changes, then CI will proceed with deployment.

Example 

1. Ethan makes some changes,
2. Ethan sends merge request to the dev branch,
3. CI server runs tests and prerequisites for the project, if succeeds,
4. CI server assigns a random developer (from the approvers’ list) for code review (Let’s say Fatih)
5. Fatih reviews code on GitLab UI and approve the merge request
6. CI server finishes CI and CD pipeline (finish merge to dev and if its release, merge it to master and deploy as well).

Based on the team size, we will only enable code review for releases. Making code review part of our regular builds will be an extra step for a remote development team and may cause delays.

CI /CD Pipeline Types

A) Development Merges

Regular code check-ins when finishing sprints, screens, bugfixes, etc. The primary reason to do this as often as possible to sync clean and stable codebase between developers when working on larger/longer builds.

1. Matt commits and pushes all his changes to his local branch “matt-dev” and created a “merge request” from the matt-dev → dev branch.
2. Gitlab CI picks up the requests and starts the CI pipeline to check the code.
3. With successful CI pipeline result, merge request will be approved by the CI server and code will be merged to the dev branch.

With successful build and code change notifications on slack channels, all developers will be able to see any change on the dev branch. Everybody will pull the latest code to their local development branch.

B) Development Builds

Same as Dev CI but triggered with a final build to be pushed for distribution for testing

1. Deployments (builds) will be triggered either:
   1. Manually over GitLab UI, slack commands, or a potential mobile app, we will build for deployments.
   2. Every morning with a scheduler, if there are merges that are not built yet. This will allow our development pipeline to be always shipping the changes to testable code. This process effectively eliminates the “deployment queue” concept.
2. Deployments will start with CI pipeline re-run including code quality checks.
   1. CD steps are actually steps in the CI pipeline that are only enabled for deployments. These steps will package and deliver the app bundles to various distribution platforms TF, Fabric/Firebase, OTA based on the platform.

C) Releases (production)

When the development builds are stable and it’s time to package the code for production. 

1. One of the developers of the project takes the lead and manually triggers a release build by creating a merge request from the dev branch to the master branch. CI server picks up this merge request and fires up a CI pipeline.
2. When the CI pipeline successfully passes, it notifies all “reviewers” defined in the project to review the code. We will have multiple developers and team leads review the code and approve the release.
3. Once all reviewers approve the merge request and pushed the code to the master branch, the CI pipeline also tags the master branch with the version number of the release,
4. Gitlab merges the code to master and continues on CI pipeline for the release.
5. After packaging the app, CD pipelines distribute the app to existing internal distribution services TF + Fabric/Firebase + OTA. On top of this, CD prepares the build on App Store and Google Play for final release operation. Since the App Store uses TF builds as the way to push builds to the app store, we don’t need anything extra on that but we need to implement a way to upload the release build to Google Play as a “Timed Publishing” model.

D) Hotfix releases

This is a special branch that mimics the “dev” branch and has the same release workflow. Hotfix releases have an extra step on the “merge to master” step, it also runs pull operation to the “dev” branch after hotfix changes are merged to master. This makes hotfix changes to become available in the “dev” branch.

Working without a code versioning system is unthinkable. And we progressed from the days of svn to git. Despite having a big competitor like the most popular git service is github.com, gitlab.com is for me a more appealing git hosting (and much more) service compared to github.com.

Gitlab is a fantastic platform that is ahead of github.com in certain aspects. We use GitLab as our primary code management, quality control, continuous integration, and deployment platform in our team. Although Microsoft is pushing hard to catch up on the parts (i.e GitHub actions to catch up GitLab CI/CD pipelines) that they follow, and obviously, much more ahead of gitlab.com on other features.

One of the biggest separating factors for gitlab.com is its self-hosted nature of it. IT’s open source license and software that you can install and host on your own infrasture. In some projects or clients, you may want to host your own isolated code revisioning and build platform.

With this nature of it’s licensing model, we first choose to use GitLab self-hosted to run and maintain GitLab versions over years until last year that we migrated to gitlab.com cloud service.

The biggest underlying reason was to not want to maintain gitlab.com hosted versions and be responsible for the server management, updates/upgrades, and stuff. To be honest, in one of the instances, we spent too much time figuring out upgrade details and making sure we didn’t cause any downtime while we performed upgrades. We also rigorously wanted everything running on the docker swarm pool of servers we manage. And Gitlab itself is not the easiest service to get it running with our current toolset of docker/traefik…

Gitlab.com has the exact same licensing structure between the cloud and the self-hosted versions. This is why it doesn’t make any sense to self-host and maintain infrastructure if you don’t have other reasons not using the cloud version of the service. That’s exactly what we were doing and we quickly experimented on few repos to be migrated and see we could have almost no change needed on our CI/CD pipelines settings that were 100% compatible to continue running on the cloud version. It really was seamless to migrate to the cloud version from the hosted version.

Another beauty of using gitlab.com is how easy to set up hosted “runners” and set them up to be used in the cloud repos for CI/CD pipelines. We often have extra build steps that are generally just optional but just because we could run them on our own runners without worrying about build minutes.

I suggest gitlab.com and its cloud version for both basic git hosting use cases as well as more advanced use cases like ci/cd pipelines or even adding 3rd party services on your code quality process.

https://about.gitlab.com/pricing/

 Many years ago, we used IronWorker as our worker running, management service. We were also delegating all infrastructure resources for the workers that required more and more resources as we scaled our app back then. It was great to have the workers separated from our main applications (API, website…).

Just to give you context, the workers – or background scripts – are generally independent and isolated scripts that run in their own schedules to do one and only one type of task – like sending birthday emails, or calculating an index that will be used for a function on the website that is used frequently. Each worker/script has separate schedules. Some run once a day (generally runs nightly), or some run every hour or some run even more frequently like every 5 minutes or so. There are also workers that are designed to always-run, but these tasks also can be designed and coded as scripts that can run in batches and scheduled accordingly. Workers are mostly packaged, all dependencies included scripts that only require a runtime and its external dependencies like database connection, etc…

We used different solutions in different projects over the years along with cloud services like Iron Worker. But I was never satisfied because I wanted something both convenient as well as had nice web UI to be able to have some control – like start/stop/check status of workers as well as see output and error logs. I also didn’t want to run another daemon for this along with my main application and the workers.

I passively searched for a nice solution for years until I realized when we were using gitlab.com runners to run a lot of our CI/CD pipelines and coded up many CI pipelines included multiple steps, often using 3rd party services and bots to control the flow of the pipelines. At the end of the day, a pipeline was a script (or series of scripts or steps) that runs on a runner, in a temporary container in docker/Kubernetes infrastructure. This allows us to use any tool/script language we want, add many environment preparation steps we want, and see the output of the scripts we run.

Gitlab.com CI/CD pipelines support ways to run a pipeline on a schedule without any other trigger (like code push, merge…). This allows us to design our workers as custom pipeline steps and be able to schedule these steps as we want. We can also pass any payload we want from the schedule configuration as a command-line argument to the pipeline scripts.

When I realized this, I experimented with few personal scripts that crawl, extract and aggregates stuff for myself – like a script that processes new craigslist posts through RSS feeds, cache them, and sends notifications on slack. Also, I was able to run these scripts on my own self-hosted runner that didn’t incur any CI/CD minutes. It was perfect.

You can design, code, and schedule your background scripts/workers as gitlab.com pipeline steps. Running them on either shared cloud runners (gitlab.com gives 400 minutes per project group per month), or use your own self-hosted runner on a docker swarm (or Kubernetes).

Working with git (or a similar version control system) is not an essential part of coding. We do frequent code commits as part of our workflow. Every developer has their own way of committing their code and labeling the changes made in the commit with commit messages. We often do this without putting enough thought into our commit messages that essentially define our change log in a more readable way when we look in the history of changes.

We need better commit messages

This brings us why having consistent, concise, and easier to follow commit messages very important. Better commit messages will make your code history, easy to follow, and navigate in the event to understand what happened when. In some cases, we want to use these changelogs more formally for internal purposes or external use like public changelogs or release notes. There are even tools that help to automate / to generate changelogs from commit messages.

Better teamwork

Working with multiple developers even as little as two developers on a project will require clear communication on the version control system. The commit messages do not only appear in the code history, but also in blame logs as well as in-IDE helper tools like git lens for Visual Studio Code when the last change was done on the particular line under the cursor, by who, and with which commit message. Features like these make the code writing experience much richer and passively collaborative between team members. So commit messages actually appear in different places in different ways.

How

Team should define clear set of commit message rules starting from their git flow. What I mean by that is how they do their branching, tagging strategy. This generally allows certain rules to be inherited to commit messages.

Regardless of the rules, people will forget. Easiest and best way to implement these rules to make sure there is a automated control mechanism that rejects/alerts if the commit message written is abiding these rules. “commitlinter” is a nicely designed npm package that is, when registered as a git pre-commit hook, checks the commit message and based on the commitlinter configuration, allows or rejects the commit if the commit message follows all the requirements or tell what’s going wrong with the commit message.

commitlinter comes with pre-defined conventions that are adapted by big companies/teams that are different in approach and have different focuses. 

https://www.conventionalcommits.org/en/v1.0.0/

I suggest you to review these different conventions and pick a convention that speaks closer to what you want to follow. And you can extend and customize its rules with your own approach.

At a high level, defining “scope” for your commit messages is the most critical categorization of the change content when commiting. An approach like the below is a good start:

• feat: Add a new feature to the codebase (MINOR in semantic versioning).
• fix: Fix a bug (equivalent to a PATCH in Semantic Versioning).
• docs: Documentation changes.
• style: Code style change (semicolon, indentation…).
• refactor: Refactor code without changing public API.
• perf: Update code performances.
• test: Add a test to an existing feature.
• chore: Update something without impacting the user (ex: bump a dependency in package.json).

We use a simplified version of this scope set. A sample commitlinter config file:

module.exports = {
	parserPreset: 'conventional-changelog-conventionalcommits',
	extends: ["@commitlint/config-conventional"],
	rules: {
		"type-enum": [
			2,
			"always",
			[
				"feat",
				"fix",
				"cont",
				"chore"
			]
		]
	}
};

See all rules in its official documentation: https://github.com/conventional-changelog/commitlint/blob/master/docs/reference-rules.md

commitlinter also can be configured with husky which is another npm package that handles git hook registration in npm install commands.

The combination of both projects in a NodeJS project will allow setting and configuring the hooks easily.

Change logs from commit messages

Conventional commit and commitlinter gets juicer when combined with an auto-generated changelog in a technical project – even for internal use cases. There are changelog generators from git-log that gives you conventional changelog which will be much more consistent throughout the team members who are making commits to a single repo as well as having well consistent updates in the commit messages.

Conventional changelog generators can translate each conventional commit spec to nicely categorized changelog in a simple configuration file like:

{
    "types": [
      {"type": "feat", "section": "Features"},
      {"type": "fix", "section": "Bug Fixes"},
      {"type": "chore", "hidden": true},
      {"type": "docs", "hidden": true},
      {"type": "style", "hidden": true},
      {"type": "refactor", "hidden": true},
      {"type": "perf", "hidden": true},
      {"type": "test", "hidden": true}
    ],
    "commitUrlFormat": "https://github.com/mokkapps/changelog-generator-demo/commits/{{hash}}",
    "compareUrlFormat": "https://github.com/mokkapps/changelog-generator-demo/compare/{{previousTag}}...{{currentTag}}"
  }

That generates a nicely organized markdown or HTML document. AngularJS is one of the known project utilized conventional commit messages and conventional changelog. See their changelog for an example of this; https://github.com/angular/angular/blob/master/CHANGELOG.md

Happy conventional commits…