In the ever-evolving landscape of software development, efficiency and automation are the keystones that keep the wheels turning. Today, I'm excited to share with you a powerful tool that has significantly streamlined my development process: GitHub Actions. Whether you're looking to enhance your continuous integration (CI) and continuous delivery (CD) processes, or simply automate repetitive tasks, GitHub Actions can transform the way you approach development workflows.

What Are GitHub Actions?

At its core, GitHub Actions is an automation powerhouse, provided by GitHub, that allows you to automate your software workflows directly within your repository. What makes it stand out is its event-driven nature, enabling actions to be triggered by specific GitHub events like a push, a pull request, or on a scheduled basis. This flexibility means you can automate just about anything, from building and testing your code to deploying it.

Setting Up GitHub Actions

The beauty of GitHub Actions lies in its simplicity. You define your workflows in YAML files within the .github/workflows directory of your repository. These workflows can be as simple or complex as you need, orchestrating every step of your build, test, and deployment processes.

Here's a step-by-step guide to get you started:

1. Navigate to the .github/workflows directory: This is where all your workflow files will live. If it doesn’t exist yet, now’s the time to create it.
2. Create a new YAML file: Name it something descriptive, like ci.yml, to indicate its purpose. This file will contain the configuration for your workflow.
3. Define your workflow: Start with specifying the events that will trigger this workflow. It could be on a push or pull request to specific branches, or even a scheduled event. Then, outline the jobs and steps that should be executed. Here's a basic example:

name: CI Workflow
on: [push, pull_request]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Run a script
      run: echo Hello, world!

Testing GitHub Actions Locally with act

Before pushing your workflows to the live repository, you'll want to test them to ensure they work as expected. Enter act, a tool that allows you to run your GitHub Actions locally. This is especially useful for debugging and iterating on your workflows without the wait time associated with pushing changes.

Getting Started with act

1. Install act: If you're using a Mac, simply run brew install act in your terminal. This will install the act tool, which simulates GitHub Actions on your local machine.
2. Run your workflow: With act, you can trigger your workflow by simulating a GitHub event. For example, to simulate a push event, you'd use:

act push -W .github/workflows/your_workflow.yml --secret-file .env.local --container-architecture linux/amd64

This command allows you to specify which workflow to run, simulate secret variables, and ensure compatibility with your machine's architecture.

Managing Secrets and Variables

Keeping sensitive information secure while automating your workflows is crucial. GitHub Actions allows you to use 'secrets' for sensitive data and environment variables for non-sensitive data. Secrets are encrypted and can be added to your repository or organization settings, ensuring they are not exposed in logs or workflow files.

Conclusion

GitHub Actions offers a robust, flexible platform for automating software workflows, significantly reducing the friction in the CI/CD process. By leveraging GitHub Actions and tools like act for local testing, developers can ensure their workflows are efficient, secure, and error-free.

Remember, the key to mastering GitHub Actions is experimentation and iteration. Don't be afraid to explore the vast ecosystem of actions available in the GitHub Marketplace and to share your custom actions with the community.

Happy automating!

Languages that is. Separation of concerns has often been taught that the languages and technologies are where we should separate our concerns. In my 10+ years doing development, the amount of times I have needed my SQL separate from my backend code, or my CSS separate from my HTML code has been next to never. If you are separating your concerns this way, consider a new approach. Separation my job or feature.

Understanding Separation of Concerns

At its heart, the Separation of Concerns (SoC) is a design principle for organizing software into distinct sections, where each section addresses a separate ‘concern’. A concern is essentially any piece of interest or focus in a program. Why does this matter? Because, in the sprawling complexity of modern web development, keeping our code clean, understandable, and manageable is not just a luxury; it’s a necessity.

The Importance of Separation of Concerns

SoC is not just about writing code; it’s about writing stories. Each piece of your application tells a part of the story, and SoC ensures that these stories remain distinct and coherent. This clarity is not for the computer’s benefit—it doesn’t care how tangled our narratives become—but for us, the humans who spend more time reading code than writing it.

By dividing our applications into manageable, focused areas, we drastically improve our ability to maintain and evolve our codebase. Changes in one area, such as the data model, have minimal impact on others, like the UI. This isolation reduces bugs, simplifies debugging, and speeds up development, as teams can work on separate concerns simultaneously without stepping on each other’s toes.

The Misconception of Language-Based Separation in Web Development

Traditionally, we’ve been taught to split our concerns by language: HTML for structure, CSS for presentation, and JavaScript for behavior. While this approach seems logical, it fundamentally misunderstands the nature of our work. We don’t create websites or apps by neatly compartmentalizing structure, style, and logic. Our work is messier, intertwined, and ultimately, more holistic.

Take Angular, for instance. Angular’s component-based architecture encourages developers to think about UIs as a collection of isolated units. However, each component intertwines HTML, CSS, and JavaScript, challenging the traditional SoC model. Critics argue this blending muddles concerns, but I’d argue the opposite. Angular understands that the real ‘concern’ is the component itself—its functionality, appearance, and behavior as a cohesive unit. The issue with Angular, and similar frameworks, lies not in their lack of separation but in how they define a ‘concern.’

Reframing Separation of Concerns: Finishing Jobs, Not Tasks

The true power of SoC emerges when we focus on completing ‘jobs’ rather than performing ‘tasks’. Consider a developer tasked with adding a feature to a web application. If we adhere too rigidly to language-based separation, we risk fragmenting their focus across disparate files and folders, losing sight of the job’s overarching goal.

Instead, imagine if our focus shifts to finishing a specific job, such as implementing a user authentication feature. This feature spans multiple ‘concerns’—UI elements, styling, validation logic—but it’s a single, coherent job. By grouping all related aspects of this job together, developers can think in terms of features, not files, leading to more intuitive and maintainable codebases.

This approach doesn’t dismiss SoC; rather, it redefines it. It encourages us to ask: “What job am I trying to finish?” rather than “What specific task am I performing?” This mindset shift can lead to more effective organization of code, reducing the cognitive load on developers and enhancing collaboration within teams.

Conclusion

Separation of Concerns is a foundational principle in software engineering, but its true essence often gets lost in translation. It’s not about segregating code by languages or technologies but about organizing it in a way that reflects the real-world jobs we’re trying to accomplish. As web development evolves, so too should our understanding and application of SoC. By focusing on finishing jobs rather than ticking off tasks, we can create more robust, intuitive, and maintainable applications that stand the test of time.

In the spirit of continuous learning and improvement, let’s challenge ourselves to rethink our approach to SoC, embracing the complexity and nuance of building modern web applications. Together, we can write not just better code, but better stories.

In the world of software development, the debate over which framework or language reigns supreme is as old as the field itself. Whether it's Rails vs. Laravel, .NET vs. Java, or any other comparison, the conversation often boils down to one question: which is faster? But if there's one thing I've learned over the years, it's that this is usually the wrong question to ask. Let me tell you why.

Changing Your Framework Now Won’t Solve Your Problems

Throughout my career, I've been a part of many teams: team .NET, team CodeIgniter, team Django, and, most recently, team Node.js. Each switch was made in the belief that the new framework or language offered a recruitment advantage, better documentation, or simply a "better way" of doing things. However, the real issue often lay not with the technology itself but with our understanding and usage of it.

Quick Story

I once joined a company as a fervent supporter of modern JavaScript, only to find myself working with what I considered "old technology": Rails. The codebase was outdated, cluttered with custom implementations, and frankly, a nightmare to debug. My initial disdain for Rails was palpable.

However, six months in, a leadership change and a dedicated effort to update and optimize our Rails application turned my opinion on its head. Our once sluggish app became significantly more responsive and easier to work with, proving that the problem wasn't Rails itself but how we were using it. Despite my early reservations, I became a proponent of the framework, even if it wasn't my top recommendation.

Fast forward two years, and our application, now paired with a React front end and a streamlined Rails API, was more efficient and performant than ever. This transformation wasn't the result of abandoning Rails but rather embracing it fully—learning its intricacies, monitoring its performance, and dedicating time to address its issues.

Turbo Pascal Might Be Dead, Actually, So There Are Exceptions

While my journey with Rails illustrates the benefits of digging deeper into your current framework, I acknowledge that there are exceptions. Technologies do become obsolete, and there are legitimate reasons for switching frameworks or languages:

1. Probably never: As a rule of thumb, switching should be a last resort, not a first step.
2. Recruitment challenges: If finding talent for your technology stack is becoming increasingly difficult, it may be time to consider alternatives.
3. Technical limitations: When your current language or framework cannot adequately support the solutions your projects require, a switch becomes necessary.

Conclusion

The allure of switching to a "better" framework or language is strong, especially when faced with performance issues or the latest tech trends. However, more often than not, the key to improvement lies not in the tools themselves but in how we use them. Deep knowledge of your current stack, performance monitoring, and continuous optimization are crucial. Only in specific circumstances—such as recruitment difficulties or technical limitations—should a complete switch be considered. Remember, the grass isn't always greener on the other side; sometimes, you just need to water the grass you're standing on.

TLDR: this repo has the starting point, and this is it completed project.

I have used many languages to make apps over the years. After .NET and some node development, I found myself in a job using rails. ICK! RAILS! I thought. This is old, antiquated, and hard to use!

Well, 3 years later I changed my mind. Many of the issues I had with Rails came from the old code base that I was working in. My mentor and boss when I started removed over 200K lines of code before he left, and I removed 220K lines before I left. Removing the baggage made it better, and really, using it for backend development for a REST API really made me like it. It's no-nonsense and handles things like database migrations, uploading to S3, and delayed tasks really well.

So let's get your mac setup for Rails using dev containers, so you don't hate it, like I did.

1. Setting Up Your Mac for Rails Development

1.1 Install Homebrew

Before diving into anything else, ensure you have Homebrew installed on your Mac. It's a package manager that will simplify software installations.

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"


1.2 Install Docker

With Homebrew, installing Docker is as easy as:

brew install --cask docker


2. Setting Up the Rails Project with Docker

2.1 Dockerfile

Create a Dockerfile.local for your Rails project:

FROM ruby:3.2.1
ENV ROOT="/myapp"
ENV LANG=C.UTF-8
ENV TZ=UTC

# Install essential libraries
RUN apt-get update -qq && apt-get install -y nodejs postgresql-client

RUN mkdir ${ROOT}
WORKDIR ${ROOT}

This will be used in creating the dev container. Don't worry about the production dockerfile, Rails 7.1 ships with one and we will use that later.

2.2 Docker Compose

Create a docker-compose.yml file to set up the services used for your development container. We will need the application container, or web container, and a database. We will use postgres as it is well regarded in the rails community.

version: '3'
services:
  web:
    build:
      context: .
      dockerfile: dockerfile.local
    command: bundle exec rails s -p 3000 -b '0.0.0.0 --port 1234 --dispatcher-port 26162 -- bin/rails s'
    volumes:
      - .:/myapp
    ports:
      - "3000:3000"
    depends_on:
      - db
  db:
    image: postgres
    volumes:
      - ./tmp/db:/var/lib/postgresql/data

3. Setting Up VS Code Remote Debugging

3.1 Install remote debugging

First we need to set up the environment to default to using docker. Go to VS Code and install the plugin below:

Remote Development - Visual Studio Marketplace

Extension for Visual Studio Code - An extension pack that lets you open any folder in a container, on a remote machine, or in WSL and take advantage of VS Code’s full feature set.

Visual Studio Marketplace

Once installed, create a folder named .devcontainer at the root of your project. It will hold the setup files to allow for debugging and coding inside that container.

Next we will add the /.devcontainer/devcontainer.json file to the project. This will set up the dev environment for you and the other devs on your team. We will add the plugins we want to use as well as the working directory of the project within the container. Most settings to get setup will be inside of this file. Here are the ones to get started:

{
	"name": "Existing Docker Compose",
	"dockerComposeFile": [
		"../docker-compose.yml",
		"docker-compose.yml"
	],
	"service": "web",
	"workspaceFolder": "/myapp",
	"customizations": {
		"vscode": {
			"extensions": [
				"castwide.solargraph", // solargraph
				"rebornix.Ruby", // Ruby
				"misogi.ruby-rubocop", // Rubocop
				"KoichiSasada.vscode-rdbg" // Ruby Debug
			]
		}
	}
}

Add another docker-compose file at ./devcontainer/docker-compose.yml

and put this in it:

version: '3'
services:
  web:
    command: /bin/sh -c "while sleep 1000; do :; done"

3.2 Start the container

Now we are going to run some rails commands and we will need to have rails available. We are avoiding rails being installed on your computer, so we will need to get the container up and going. Once the files above are saved, each time you open the folder you will be greeted with a pop up that will ask you if you want to launch inside of the container.

However, it may not do that after you just barely created those files, so lets open it another way. In the lower left hand corner you can click on the green or blue square with ><

and run Reopen in container (shown below) and it will attempt to run docker compose for you and connect to the container.

You should be connected to the container at this point and ready to run commands in the container from your terminal. To open a terminal in VS code type control + ` and it should say your folder is myapp.

3.3 Set up rails project if you don't already have one

At this point we need something to debug. We will need rails, at the time of writing Rails 7.1.0 was just released, so we will be using that version. So let's install rails gem install rails.

Now, let's add the default rails app by running rails new . inside the folder.

If you are using the folder that I made for this article, allow all files to be overridden upon creation.

3.4 Debug Configuration

Create or edit the .vscode/launch.json file with the following configuration:

        {
            //Start Rails server
            "name": "Debug Rails",
            "type": "rdbg",
            "request": "launch",
            // "command": "bundle exec rails", #bundle exec rails won't stop in the debugger
            "command": "bin/rails",
            "script": "s",
            "args": ["-b","0.0.0.0"],
        },
        {
            // Run tests on the active rspec file
            "name": "Debug Rspec with current file",
            "type": "rdbg",
            "request": "launch",
            "command": "bundle exec rspec",
            "script": "${file}",
            "args": [],
             // Confirm the executed command in the window. Make it easy to specify options such as execution of
            "askParameters": true
        }

Save the file and click on the run and debug menu (play button with a bug on the left):

Click on the play button that reads Debug Rails which will start the application so you can see it on localhost:3000 and it is in debug mode. Find config/pplication.rb

and add a debug point. Load the project in a browser, and watch the break point catch. This will give you full debugging capability as you develop.

4. GitHub Actions Linting & Docker Image

4.1 Create Workflow for linting

In your repository, create a .github/workflows/rubocop.yml:

name: Rubocop

on:
  pull_request:
  push:
    branches:
      - 'main'

jobs:
  build:
    name: CI Rubocop
    runs-on: ubuntu-latest
    env:
      api-dir: ./

    services:
      postgres:
        image: postgres
        ports: ["5432:5432"]
        env:
          POSTGRES_HOST_AUTH_METHOD: trust
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

      redis:
        image: redis:alpine
        ports: ["6379:6379"]

    steps:
      - uses: actions/checkout@v2
      - uses: ruby/setup-ruby@v1
        with:
          ruby-version: 3.2.1
          bundler-cache: true
      - name: Install PostgreSQL
        run: sudo apt-get -yqq install libpq-dev
      - name: Run bundle install
        working-directory: ${{env.api-dir}}
        run: |
          gem install bundler
          bundle install --jobs 4 --retry 3
      - name: Setup Database
        working-directory: ${{env.api-dir}}
        env:
          RAILS_ENV: test
          PGHOST: localhost
          PGUSER: postgres
        run: bin/rails db:create db:schema:load
      - name: Check Rubocop Styles
        working-directory: ${{env.api-dir}}
        env:
          RAILS_ENV: test
          PGHOST: localhost
          PGUSER: postgres
          RAILS_MASTER_KEY: ${{ secrets.RAILS_MASTER_KEY }}
        run: bundle exec rubocop

Now we need to add the rubocop gems to the project so that they run when they are in production. Open up your Gemfile and look for the development section. Make sure it looks like this:

group :development do
  # Use console on exceptions pages [https://github.com/rails/web-console]
  gem 'rubocop', require: false
  gem 'rubocop-discourse', require: false
  gem 'rubocop-performance', require: false
  gem 'rubocop-rails', require: false
  gem 'rubocop-rails_config', require: false
  gem 'rubocop-rake', require: false
  gem 'rubocop-rspec', require: false

  gem 'ruby-lsp', require: false
  gem 'ruby-lsp-rails'
  gem 'web-console'

  # Add speed badges [https://github.com/MiniProfiler/rack-mini-profiler]
  # gem "rack-mini-profiler"

  # Speed up commands on slow machines / big apps [https://github.com/rails/spring]
  # gem "spring"
end

4.2 Create Workflow for building for production

In your repository, create a .github/workflows/build_and_push.yml:

name: Build and Push Docker Image

on:
  push:
    branches:
      - main

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v2

    - name: Build and push Docker image to GitHub Packages
      id: docker_build
      uses: docker/build-push-action@v2
      with:
        username: ${{ github.actor }}
        password: ${{ github.token }}
        registry: docker.pkg.github.com
        repository: ${{ github.repository }}
        tags: latest

4.3 Create a production environment

If you want specifics, this will be covered in a separate post as it would get too long. But you have some options once you have your docker image created. There is Dokku, AWS App Engine, Kubernetes, Porter.run, and many others that can take a docker image and get it up and running. Pick one, and get going, that is the key. Dokku would be my pick for a small project as it is simple and easy to get going.

5. Wrapping Up

Now, whenever you push to the main branch, GitHub Actions will automatically lint your ruby code and build your Docker image and push it to the GitHub image repository.

Now the reason why this is important comes down to running in an engineering team. If you can remove friction from any process, you can help relationships, and ultimately remove excess meetings or at least "check-ins." By enabling rubocop, there are no more PR reviews that are "I wish you would have used a ternary instead of if then" or other opinions. And the agreed upon code "rules" are built into the project.

The second part is building your project and having it ready for production. By eliminating the friction of going to production, you will ship more often to your users. In my experience, you don't actually go faster, but the size of your deployments go down, which lowers your risk of production bugs.

Rails is a great platform for creating a start up. It has batteries included, and if you can EMBRACE it's conventions, it works really well. When I hated Rails, I was trying to do things against it's conventions.

Enough about Rails, this tutorial will get you a full debug environment, and everything setup for PRs and Production. This really simplifies your engineering time, and will help you focus on shipping features to your users, and above all having a successful business.

I had just switched from my Electrical Engineering job, to IT. Nine months in I was asked to get another computer setup for my co worker. This was 2013, and we were using .Net 3.5, and Visual Basic (don't judge) to automate production processes, and I spent 3 days in DLL hell getting everything set up. My boss hadn't documented the project, he was a team of 1 for the 6-ish years he had been writing this software. And those 3 days fueled my love of Docker 2 years later when I started playing around with it.

Never again, would setting up a dev environment take 3 days. I would write my infrastructure as code from then on, and save my headache of setting up servers and computers to run either GD Picture, or the pre AGPL version of iTextSharp. I haven't done a lot of .NET since then, but they have adopted docker and a package manager first approach. Both of those things help get a dev environment running faster, but let's focus on Docker.

What is Docker?

Docker is an open-source platform that enables developers to automate the deployment, scaling, and management of applications. Docker achieves this by using containerization, a lightweight form of virtualization. Unlike traditional virtual machines, Docker containers share the host system's OS, making them much more efficient and faster to start up. Each container runs as an isolated process in user space on the host operating system.

Motivation

I use Docker for development for several reasons. First, Docker ensures consistency across all dev machines in an org, eliminating the "it works on my machine" problem. It also isolates the dependencies, and simplifies installation. Second, it allows the configuration to be committed to GIT and make changes through a PR process, allowing the use of a review process familiar to those who make software but use that process for infrastructure.  Last, Docker seamlessly integrates with many popular continuous integration/continuous deployment tools, making it easier to set up a CI/CD pipeline.

Dockerfile

A Dockerfile is a text file that Docker reads to build an image. It consists of a series of instructions, each creating a new layer in the image. Locally, layers are cached to help speed up build times.  The typical instructions include FROM to specify the base image, RUN to execute commands, COPY or ADD to add files from the host machine, WORKDIR to set the working directory inside the image, and CMD or ENTRYPOINT to specify the command to run when a container starts. Each Dockerfile should ideally represent one executable application or service. Check the documentation from Docker for more.

Docker Image

A Docker image is a lightweight, standalone, executable package that includes everything needed to run a piece of software, including the code, a runtime, libraries, environment variables, and config files. Docker images are built from Dockerfiles, like I described above, and are the basis of containers. They can be stored and shared using Docker registries. The most common is docker hub, and if you need to send an image to production, a private registry is a good option.

Container

A Docker container is a running instance of a Docker image. Each container runs an application in isolation from other containers, providing a secure application platform. Containers are lightweight and fast, as they directly leverage the host machine’s kernel, yet they maintain the isolation and resource constraints you would get from a full VM.  On my local machine, using docker compose, I run multiple containers for Redis and Postgres.  I might run other containers but they are subject to change.

AWS Docker Registry

Even if you don't use Github or Gitlab, you can create docker images and store them in AWS for your production environment to reference. The AWS Docker Registry, also known as Amazon Elastic Container Registry (ECR), is a fully-managed Docker container registry that allows developers to store, manage, and deploy Docker container images.  It eliminates the need to operate your own container repositories or worry about scaling the underlying infrastructure. It is secure and provides resource-level control of each repository through integration with AWS Identity and Access Management (IAM). 

Docker Compose

Docker compose allows for multiple Docker containers to be ran at once, so they can all send requests to each other in the same network. This allows us to simulate a production environment on our dev machines.  The different containers, sometimes referred to as services, are usually defined in a docker-compose.yml file inside the project folder.  Docker compose files can be used to run in production using other apps, using Kompose an open source software that translates docker compose files into Kubernetes (K8s) config files. But before you do that you should probably look into Nomad, Dokku or Heroku.

In a local docker-compose development environment, I primarily have three services: The App, Database, and Redis. Sometimes there are some more specialized things like RabitMQ or other open source projects, but the point is to run what is in production locally. Docker-compose enables me to define and run these multi-container Docker applications, allowing me to spin up dependencies with a single command.

Your Dev Environment

In my development environment, Docker is used to build, run, and manage my application including it's dependancies. After defining the services in a docker-compose.yml file, you can run from the terminal, in the project folder docker compose up -d to start the underlying infrastructure. I even used docker to replace version managers, think Node Version Manager (NVM) or RBENV which is a ruby version manager. This way you don’t have to install ruby at a certain version, you can run your code in a container that has the specific version of your runtime installed. If you set it up right, changes you make in code will be reflected in the running containers. This is enabled by using the volume mounts linking the containers to the local filesystem.

Docker in Production

The most popular way to deploy docker applications right now is probably Kubernetes.  Kubernetes is a powerful and complex tool, that works like docker-compose.  It allows for specifying multiple containers (pods) that run in services, so applications can autoscale.  This is a use case for organizations and workloads that are highly volatile and require a lot of server resources, and a lot of developer resources.  Most teams usage is pretty static, or at least predictable, so the benefits should outweigh the amount of time you will spend in deploying.

Currently in production, I leverage Docker via Dokku, the Docker-powered mini-Heroku. Dokku allows us to manage deployments, scale applications, and perform zero-downtime deployments, ships logs to s3 using vector, all while using the same Docker images I use in my dev environment. This ensures the environment my code runs in is consistent, from development to production. With Dokku, deploying apps to production is as simple as pushing our code using Git. Without a Dokku pro license, you aren't able to auto scale. As Dokku is meant to be used on 1 Debian server, but most of the time on a new project, that is all you need to get to production.

Summary

If you are just running a Typescript app, you don't need docker. Why did you read this far anyway? If you need your API or server running locally, Docker can take a little extra time to set up. But it operationalizes your setup. Everytime your computer goes awry or you need to add another developer, it's easy to get set up. You are only a few steps away from deploying to production.

A few years ago, I got into watching my alma-matter's football games. This opened my eyes to a world in which I was checking ESPN weekly and sometimes daily. At the time ESPN had the ESPN app, and Watch ESPN for streaming that I used. There were also an ESPN fantasy sports app, among others in their portfolio. A few years ago, they unified many of their offerings into the ESPN app, where you could find streaming, and many of their other functions.

As a user, this is better. I know them as ESPN, and when I go to open an app about sports, I don't want to think, "which department at ESPN would do that? I should look for their corresponding app." No, as a user of ESPN, I just want ESPN.

This is the same with the micro front ends trends that some companies are adopting. Looking at the problem they are trying to solve, I can understand, when you have hundreds of engineers and you need to ship updates to the front end without it affecting one another's code, it can be challenging. So you reach for something like Piral or SingleSPA to enable multiple independent teams to ship to production. Or maybe you are trying to solve, "How can I ship both Angular and React on the same page?" and you find Piral or micro-front-ends and decide it solves your problem. And it will, that problem it will fix.

But let's look back to my experience using ESPN. When I wanted to view a sporting event, the ESPN app had to push me to a different app, with a different login process, that didn't have any context of what I had just been reading or doing. This didn't serve me as a Sports Enthusiast™. The problem that this is surfacing is Conway's Law, which states,

Or in other words, we had an ESPN streaming app not because Sports Enthusiasts™ wanted another ESPN app, we had a streaming app because ESPN had a separate team working on streaming.

Moving Forward

Let's look at what ESPN did. They redesigned their main app, and they brought much of the functionality of each of the other apps to the main ESPN app. This is very much inline with what they were doing on web already. They had unified URI where every user went, and then each piece was cut out separately, but placed on the same domain, except 538, and few stand alone properties. Notice on their website, they still have a streaming section, they have all the different sports, and they probably still have separate teams working on different projects.

Conway's law is still probably happening within their organization. The difference, the user doesn't see Conway's law happening. The added friction of a different app was great enough, that they integrated the streaming features in the main app in 2015 and sunset the streaming app in 2019.

"But micro front ends is on the same domain!" you might exclaim. And I would agree, it is much better than having a separate app, or separate domains to visit. But there is still some friction when none of your teams are coordinating UI updates, and not allowing design to make the best version possible for your users.

Process

1. Create a sitemap of the features you want your users to experience.
2. Split teams based on persona --> feature
   1. Usually a team of 4-5 engineers with at least one who has "been there, done that"
3. Create a shared front end project, web or mobile, and split out the routes to the teams. Changes for the routes under each team will be owned by them, and them only.
4. At least one designer or PM should have the responsibility to make sure the experience across the whole app is cohesive.
5. Require integration tests
   1. these should test that main functionality doesn't completely break
   2. It should run after each PR is opened to ensure no conflicting code is written.
6. Don't follow this process to a "T". Do what your user's need, and expect, but as a general rule don't split your front end.

When you should split your front end

When your persona(s) differ greatly. That's it. When you need to address a persona that has different needs, that should be a big clue that you need to split your front end.

Consider most B2B SAAS products. The individual you are selling to, is not often the person using your software. At a previous company, we were having a lot of issues with deciding priority, figuring where each feature should live in the app, and were even shipping slower than we probably should have.

Our organizational problems seamed to melt away when we split the UI into 2. The first was for the person we were selling to, and the second was for the person who was using the software. By splitting by persona, and not by feature, we were able to map what each person needed, and focus on what they needed. Our team's velocity jumped and it was clear where each new feature lived.

Now let's look at our ESPN app example, I was a Sports Enthusiats™ persona, just checking scores, and maybe catching a game here and there. If they decided they wanted to have a platform for athletes to post about themselves, the portal which an athlete entered to put in information about themselves would definitely be a different persona. But splitting between a Basketball Enthusiast™ and a Football Enthusiast™ probably wouldn't make sense as their needs are very similar.

Conclusion

I love the idea of micro front ends, it does solve some problems for the organization. It also allows for some autonomy for your engineering staff. But that autonomy complicates sharing components when you pick different front end frameworks, and it breaks the flow of the user as they try to use your app. These 2 issues will be costly for your business, and any need for a micro front end framework or other splitting the code base should be resisted, if not avoided. The reality is you need to understand your user's use cases and adapt accordingly, before you add the unnecessary overhead of micro front ends.

If you are reading this, you probably work in B2B SAAS as no one else would ever choose to put a work queue in their software.  Alas, most of us work on less than sexy features in seemingly boring jobs.  But the reality is that those boring jobs are the ones that pay the bills.  Businesses want to automate away their problems without throwing more people at them, so the work queue was born.

If you are looking for the code real quick, it's right here.

I’ve worked at a handful of companies over the years, and almost all of them were in some way a B2B SAAS company.  All of them had either built a work queue, or had a work queue on their road map.

What is a work queue?  Good question, well you didn’t ask it but I’m going to tell you because you are reading this article.  A work queue is a list of items that need to be completed by a group of individuals. To give you an example let’s look at a restaurant.

There are multiple queues inside of a restaurant, and operating a restaurant is all about moving people through these queues as fast as you can.  The more tables you turn over, the more money you make, and the more money you make the better chance you have at funding that midlife crisis full of sports cars and “bitches” or “bros” depending on which form of sexism you prefer.

Anyway, you have 4 main queues when you look at the journey of the customer.

1. Seating queue
2. Ordering queue
3. Paying queue
4. GTFO queue

I don’t know if there is a real need for the 4th, but if I was turning tables that would absolutely be a metric for me.

The roles in a restaurant are clear, you have 

1. host(ess)
2. Server
3. Cook
4. Busser

And they really are concerned about filling and emptying each of those queues.

If we were to represent this in software we would have a role or permission for each of these personas and a queue for each of the queue steps above.

Your first Work Queue

This is where the fun begins.  I chose Redis, aka localstorage for servers, because it helps solve problems on the backend well, and to be honest, it’s the one that seemed to fit best. We will be using Sigue, a simple npm package used for GraphQL.  GraphQL is like SOAP api standard, but unlike SOAP there are still people that like it.

A Quick Note

I did evaluate other technologies for this, at least instead of Redis (keyed).  Azure service bus being one of them.  It has a cool Peek & Lock feature that would have worked very well for this, but the same connection, yes connection, has to be used to abandon or complete the message.  This has limitations where our backend could have multiple instances running in either Kubernetes or as a serverless function.  If you don’t have this requirement and have only 1 webserver, Azure Service Bus could be a nice tool for this.

Kafka was also looked at, but since it looked to have mostly queues only, I stuck to Redis, something I know, and that has more datatypes, like Sorted Sets.

The Pseudo Code

Let’s be real, I don’t care about having all 4 queues represented, so we aren’t going to go into each of them.  We are only going to make the reservations queue.  That queue will be what the hostess to keep track of who has and hasn't been seated.  They will pull the name to look for the party with the reservation, then after 2 minutes if they aren’t found they are placed back in the queue, or removed.

Let's start with a reservation, and let's make the data model for a reservation. It first needs a name, phone number, and the amount of people that are in the group. We will store that in the database and not in the queue. Sigue uses sequelize, so let's make that model.

Next we need to make the queue.  I am going to use the npm package ioredis to make connections with Redis, and handle the queues.  Redis has a handful of datatypes, one is a list, which most people call a queue.  And it simply takes in a string and moves them either in or out of the front or the back of the list.  One of its limitations is the inability to reference each of the items in the list and update them, or delete them from the list.  Nor can you update the items once in the list.  Once they are in the list, they kind of just sit there until you need them.

Enter Sorted Sets.  A sorted set allows for the same concept of a queue but based on a score.  That score determines the importance of the item, and makes it so you can insert items wherever you want in the queue and at any time.  This fits our needs a little better.

This is a make believe example, but in the real world a work queue will generally have something that someone has to work on, and if they don’t get it done, it needs to be released so someone else can work on it.  Let’s add that requirement to this queue.

So now we will have a sorted set named reservations.  To implement the Peek & Lock like in Azure Service bus, we will add another sorted set named reservations_delayed.  Here is the Redis code that will be used to help us write to the queues as needed. You can reference it over in the project repo.

Each reservation will be represented in the sorted set with it’s id (primary key) and its score which in this case will be an epoch based date.  Basically, it's a date represented by a number, so the sooner you made your reservation, the sooner you will get seated. 

Now we just need to hook it up to the API and add some testing. Remember, I am using Sigue right now, which helps get the server up and running. Using Sigue, We will add 2 custom resolvers, one to get a new item, and one to complete the item once the server has found their party.

We still need to add a party to the queue once their reservation is created, so now let's add the addToQueue call to the afterSave hook on the model. This should add the party to the queue.

The GraphQL queries

To write the graphQL queries let's use Apollo's sandbox. We will need to start the server so that Apollo Sandbox can do view what is in the API. Check out my blog next week where we cover Docker, but for now just run the docker compose up -d command to get Redis up and running. Then we can start the node api server by running npm start, as shown below.

Once up and running you can type in your http://127.0.0.1:4000/graphql as your server, and if it is all setup correctly, you should be able to see the graphQL endpoints for your server, including the types that are returned. This is meant to be used to explore an API, so feel free to poke around.

Create a reservation

We will attempt to create a reservation, as if someone was taken in by the hostess.

You can follow along by clicking here, or just setting up your own query for CreateReservation. Once created you should be able to run it, and see the result:

So that seems to have added it to the SQLite database that we are running locally. And since we added the addToQueue function to the afterSave Sequelize lifecycle hook, we should be able to see the data in Redis as well. I like to use TablePlus for my data development, it is the best client I have found on the Mac for running SQL. But it also has a Redis viewer. Let's check to make sure the data is there.

Sure enough, it's there as well as the one test reservation I ran before this.

A table is ready

Now let's pretend we called the group up for their reservation. We will call the getNext graphQL query to do this. This places them in a delayed queue until they are seated. Let's run that graphQL query and then check on it in Redis.

Seating the party at the restaurant

Ok, as the hostess, let's pretend we seated the group and we need to remove them from the queue. We will remove record 11 that represents the party that was just seated, and number 10 too. And let's double check the delayed queue after we complete the records.

Ensuring stability with testing

This is all fine and good, but this is tedious checking every part of this process. Automating the test for this is the right thing to do if you want to ensure your code continues to run in the future, so to do that, we will run the tests that were written in mocha. I prefer to run SOME integration tests over many unit tests. These tests test the above flow.

To run the tests, in the terminal type npm run test and watch the output.

Conclusion

This could be a really good way to handle work queues, especially for large organizations with a lot of data. Most of the time, this is probably overkill. This can be implemented with a database and something like a status column.

If you are a startup, please don't use Redis for this. Get your work queue out the door, and know you can implement this in the future.

I added all of the steps because that is what a real feature would look like. Concept to automated test. As long as your CI/CD is set up, then you can trust this won't ever be a bug in the future.

And as always, get it out the door, but don't skip any steps.

Creating types can be tedious. As software engineers, unless you are DHH, we tend to dream in types:

We make our database tables, we make our models in our favorite web framework, and if we love OOP on the back end we may even make DTOs, and even GraphQL types if you have GraphQL. And then you have to make types in Typescript, and maybe even DTOs for your Typescript. If you are lucky like me, you might even get to make those types again for your mobile app. Anders Hejlsberg has finally brought his vision of turbo pascal to our beloved javascript.

We love our types, unless it is pre-2016, at that time we hated them, at least on js twitter:

Let's just pretend that era didn't exist. We love types again, errr, always have. We need types all of the time, trust me.

This got me thinking, what is the reason for types in the first place? What problem are we solving? Developer experience. And heres a list of the main things types do:

1. make refactoring names a breeze
2. communicate data design decisions
3. find bugs before runtime
4. help organize large code bases
5. improve intellisense

This makes coding in my editor so much better, and I truly have embraced the typed world. It has saved me countless times from coding a bug. What is funny, is that we aren't very many years past a time when types weren't cool. I loved types for my first few years, until I was convinced they weren't needed, so between 2017-2020 I largely coded without types, and most of the time it was a mess. I should say I worked for a small company that I shipped a node.js project for, that I haphazardly threw together, with no code reviews, design, or automated tests. I also worked on a node µservice project that had a µservice for every table. But the mess wasn't always because of the lack of types, it also had to do with how fast I tried to ship things, and working code was paramount above maintainability on both of those projects.

I did learn that, for smaller projects, think 5 functions or so, types weren't needed. You may not reap the benefits of fully typing in a file that you will only be touching for an hour. You will spend more time messing with your types than shipping your value to your customer. And only 1 or 2 items of the list above applies to this small of a project. But most projects are bigger than that, and types are nearly always needed.

So it doesn't matter if DHH removed the library from from turbo, as much of the JS ecosystem is moving to server side rendered content, and Rails is likely past it's hey day. But we can appreciate DHH's thoughts, even if it is to help us re-examine our own opinions about our code.