Skip to content

Latest commit

Β 

History

History
380 lines (271 loc) Β· 17.5 KB

lec1.md

File metadata and controls

380 lines (271 loc) Β· 17.5 KB

πŸ“Œ Lecture 1 β€” Introduction to DevOps: From Conflict to Collaboration


πŸ“ Slide 1 – πŸ’₯ When a 45-Minute Deploy Killed a Bank

  • πŸ—“οΈ August 1, 2012, 9:30 a.m. ET β€” Knight Capital, the largest US equities trader, pushes a new release of its order-routing software (SMARS) to 8 production servers
  • 🚨 One server was missed during the manual deploy β€” it kept running old, dead code that re-activated a long-disused flag called Power Peg
  • πŸ’Έ In the first 45 minutes of market open, the rogue server submitted 4 million erroneous orders, accumulating a $440 million loss
  • 🏦 Knight had to be rescued, then acquired by Getco by December 2012 β€” gone as an independent firm
  • πŸŽ“ Lesson: A single bad deploy can sink a 1,000-person company. How you ship is now as important as what you ship

πŸ€” Think: Knight had top-tier traders and top-tier code. What they didn't have was DevOps β€” automated deploys, kill switches, rollback. Could the same incident hit you tomorrow?


πŸ“ Slide 2 – 🎯 Learning Outcomes

By the end of this lecture you will:

# πŸŽ“ Outcome
1 βœ… Explain what DevOps is, who coined the term, and the problem it solves
2 βœ… Describe the Three Ways and the CALMS framework
3 βœ… Cite the four DORA metrics and what elite vs low performance looks like
4 βœ… Use Git's mental model: working tree β†’ index β†’ repository
5 βœ… Set up signed commits and a PR template to make collaboration safer
6 βœ… Recognize that GitHub and GitLab are both valid paths for the rest of this course

πŸ“ Slide 3 – πŸ—ΊοΈ Lecture Overview

graph LR
    A["πŸ’₯ Knight Capital"] --> B["πŸ“œ DevOps History"]
    B --> C["πŸ›€οΈ Three Ways<br/>CALMS"]
    C --> D["πŸ“Š DORA Metrics"]
    D --> E["🌳 Git Workflow"]
    E --> F["πŸ” Signed Commits<br/>PRs"]
    F --> G["πŸ“ Real Incidents<br/>Blameless Postmortems"]
Loading
  • πŸ“ Slides 1-4 β€” Why DevOps exists at all
  • πŸ“ Slides 5-9 β€” History, frameworks, and how to measure DevOps
  • πŸ“ Slides 10-13 β€” Git from first principles
  • πŸ“ Slides 14-18 β€” Your course project, forkβ†’PR, signed commits, templates
  • πŸ“ Slides 19-22 β€” Real incidents, culture, and what Lab 1 asks of you

πŸ“ Slide 4 – 🧱 The Wall of Confusion

Before DevOps had a name, software shops looked like this:

graph LR
    D["πŸ‘¨β€πŸ’» Dev<br/>'Ship features fast!'"] -- 🧱 throws code over --> O["πŸ›‘οΈ Ops<br/>'Stop breaking prod!'"]
    O -- 🧱 throws blame back --> D
Loading
  • 🏒 Dev got rewarded for new features β†’ pushed code as fast as possible
  • πŸ›‘οΈ Ops got rewarded for uptime β†’ resisted every release
  • πŸ’₯ The incentives directly conflicted, so every deploy was a fight
  • 🎒 Releases happened once a quarter, at 2 a.m., under a freeze, with rollback plans written in Word docs

πŸ’¬ "It's like a relay race where the runners hate each other." β€” paraphrasing John Allspaw, ca. 2009


πŸ“ Slide 5 – πŸ“œ The Birth of DevOps (2008–2009)

  • πŸ‡¨πŸ‡¦ August 2008, Toronto β€” at Agile Conference, Andrew Shafer proposes a session called "Agile Infrastructure". Almost no one shows up β€” except Patrick Debois
  • πŸ‡§πŸ‡ͺ October 30–31, 2009, Ghent β€” Debois organizes the first DevOpsDays, named so it would fit a Twitter hashtag
  • 🐦 The conversation explodes on #devops and never stops
  • πŸ“Ί Earlier that year (June 2009 at Velocity), John Allspaw and Paul Hammond of Flickr give the legendary talk "10+ Deploys per Day: Dev and Ops Cooperation" β€” proof the model works
  • πŸ“š 2013 β€” Gene Kim's novel The Phoenix Project makes DevOps mainstream literature

πŸ’¬ "DevOps was a movement before it was a term." β€” Patrick Debois


πŸ“ Slide 6 – πŸ›€οΈ The Three Ways (Gene Kim)

From The Phoenix Project (2013) and The DevOps Handbook (2016):

# Way Mental model What it asks of you
1 πŸ›€οΈ Flow Left-to-right: idea β†’ code β†’ prod Optimize the whole stream, not the local task
2 πŸ” Feedback Right-to-left: prod telling Dev what happened Shorter loops, more signals, no blame
3 πŸ§ͺ Continual Learning Experiment + amplify what worked Blameless postmortems; deliberate practice

πŸ€” Think: Which "Way" was Knight Capital missing the most?


πŸ“ Slide 7 – πŸ”€ The CALMS Framework

Letter What it means Concrete signal
C Culture Devs and Ops sit together; nobody says "throw it over the wall"
A Automation A deploy is a git push, not a 14-step runbook
L Lean Small batch sizes; finish a thing before starting the next
M Measurement You can name your deploy frequency and MTTR off the top of your head
S Sharing Postmortems, blogs, internal demos β€” knowledge is not hoarded

Coined as CAMS by John Willis & Damon Edwards in 2010; L for Lean added later by Jez Humble.

πŸ’¬ "If you can't measure it, you can't improve it." β€” Peter Drucker, often invoked by the M


πŸ“ Slide 8 – πŸ“Š DORA: The Four Key Metrics

The DevOps Research and Assessment group (DORA, founded ~2014, acquired by Google 2018) studied tens of thousands of teams. Four metrics predict performance:

Metric What it captures Elite teams
πŸš€ Deployment frequency How often you ship to prod Multiple times per day
⏱️ Lead time for changes Commit β†’ in production < 1 hour
πŸ” Change failure rate % of deploys that need a fix 0–15%
πŸ› οΈ MTTR Mean time to recover from an incident < 1 hour

From 2021, a fifth metric β€” Reliability β€” was added to track operational stability.

πŸ“– Read: Accelerate by Forsgren, Humble & Kim (2018) β€” the scientific paper-form of these findings


πŸ“ Slide 9 – πŸ₯‡ Elite vs Low Performers

graph LR
    L["🐌 Low<br/>once / 6 mo<br/>MTTR weeks"] --> M["🚢 Medium<br/>weekly<br/>MTTR days"]
    M --> H["πŸƒ High<br/>daily<br/>MTTR hours"]
    H --> E["🏎️ Elite<br/>on demand<br/>MTTR < 1h"]
Loading
  • 🐌 Low performers and Elite performers differ by ~200Γ— on deployment frequency
  • πŸ” Elite teams also have lower change failure rates β€” speed and safety go together
  • ❌ "Move fast and break things" is not the elite pattern. Elite is move fast and don't break things, because feedback loops are tight

πŸ€” Think: Which bucket does the project you'll graduate to next year sit in? Which would you want it in?


πŸ“ Slide 10 – 🌳 Why Git? Why Now?

DevOps needs a single source of truth for code, infra, and process. Git is that truth:

  • πŸ“¦ The Git repository is all of history plus the current state β€” every laptop has a full copy
  • 🀝 Pull requests turn "review my code" into a collaboration protocol, not a favor
  • πŸ” Branches let you experiment without breaking the trunk
  • πŸ€– CI/CD systems (next week) react to Git events β€” a push triggers a pipeline, a tag triggers a release

πŸ’¬ "Git is the assembly language of collaboration." β€” folklore, attributed to many


πŸ“ Slide 11 – πŸ“œ Where Git Came From

  • 🐧 April 2005 β€” Linus Torvalds writes Git in roughly 10 days after the Linux kernel loses access to BitKeeper
  • 🎯 Design goals: speed, distributed model, integrity (cryptographic hashes), strong support for non-linear development
  • 😏 On the name: "I'm an egotistical bastard, and I name all my projects after myself. First Linux, now Git." β€” Linus
  • πŸͺͺ In November 2021 (Git 2.34), Git gained native SSH commit signing β€” no GPG keyring required
  • πŸ†• By April 2026 we're on Git 2.49+ β€” git switch, git restore, git maintenance are the modern ergonomics

πŸ“– Book: Pro Git (Scott Chacon & Ben Straub) β€” free at git-scm.com/book


πŸ“ Slide 12 – 🧠 Git's Mental Model

graph LR
    W["πŸ“‚ Working tree"] -- "git add" --> I["πŸ“‹ Index<br/>(staging)"]
    I -- "git commit" --> R["πŸ—„οΈ Local repo"]
    R -- "git push" --> O["☁️ Remote"]
    O -- "git fetch / pull" --> R
Loading
Stage What it is Common command
πŸ“‚ Working tree Files on disk you can edit git status
πŸ“‹ Index "What I plan to commit" git add -p
πŸ—„οΈ Local repo Permanent history on your machine git commit -s -S
☁️ Remote Shared truth (GitHub / GitLab) git push origin feature/x

πŸ€” Think: Why does Git separate "Index" from "Working tree"? (Hint: ever made a typo just before committing?)


πŸ“ Slide 13 – 🌿 Branching Strategies in 90 Seconds

Strategy Branch lifetime Where it shines
🌲 Trunk-based Hours Continuous delivery, large teams, feature flags
🌳 GitHub Flow Days SaaS apps, this course
🌴 GitFlow Weeks-Months Versioned products with long support windows
  • βœ… This course uses GitHub Flow β€” short-lived feature/labN branches, one PR per lab
  • ❌ Avoid: a develop branch nobody merges, six-month feature branches that never come home, "let's just push to main on Friday afternoon"

πŸ’¬ "The longer your branch lives, the more it hurts to merge." β€” every senior engineer eventually


πŸ“ Slide 14 – πŸ§ͺ Your Course Project: QuickNotes

Across all 10 labs you'll work on one small Go service: QuickNotes.

graph LR
    C["πŸ’» curl / browser"] -->|HTTP :8080| S["🟒 quicknotes<br/>Go 1.24"]
    S --> F["πŸ“„ data/notes.json"]
    S -->|GET /metrics| P["πŸ“Š Prometheus<br/>(Week 8)"]
Loading
  • 🟒 Endpoints: GET /notes, GET /notes/{id}, POST /notes, DELETE /notes/{id}, GET /health, GET /metrics
  • πŸ› οΈ You don't write the app β€” you package it, ship it, observe it, and harden it
  • πŸ“ˆ By Lab 10, the same code will live behind your own CI, in a container, on a cluster, behind monitoring and security scans, deployed to a real cloud

πŸ€” Think: Why one project for 10 weeks? Because that's how DevOps actually feels at work.


πŸ“ Slide 15 – 🍴 The Fork β†’ Branch β†’ PR Workflow

sequenceDiagram
    participant U as πŸ‘€ You
    participant F as 🍴 Your fork
    participant C as πŸ“š Course repo
    U->>F: git clone
    U->>F: git switch -c feature/lab1
    U->>F: edit, commit, push
    U->>C: open PR (fork:feature/lab1 β†’ course:main)
    C->>U: reviewer comments
    U->>F: fixup commits
    C->>U: βœ… merge / feedback
Loading
  • 1️⃣ Fork the course repo on GitHub or GitLab
  • 2️⃣ Clone your fork locally
  • 3️⃣ Branch per lab: feature/lab1, feature/lab2, …
  • 4️⃣ Commit small, signed, and with a message that reads like a sentence
  • 5️⃣ Push to your fork, never directly to the course repo
  • 6️⃣ Open a PR back to the course repo's main

πŸ“ Slide 16 – πŸ” Signed Commits & the Supply Chain

  • πŸ•΅οΈ Anyone can put any name and email in git config β€” your commit history is unauthenticated by default
  • πŸ’₯ Real consequences: in March 2024, an attacker (account JiaT75) maintained the xz-utils project for two years and slipped in a backdoor that nearly compromised every SSH daemon on Linux
  • πŸͺͺ A signed commit is a cryptographic claim: "this commit really was made by the holder of this key"
  • βœ… Git supports SSH signing since 2.34 (Nov 2021) β€” no GPG keyring, just reuse your SSH key
# βœ… enable SSH signing (one-time)
git config --global gpg.format ssh
git config --global user.signingkey ~/.ssh/id_ed25519.pub
git config --global commit.gpgsign true

πŸ€” Think: Would you merge a PR from a contributor whose commits showed "Unverified" on GitHub?


πŸ“ Slide 17 – πŸ“œ Pull Request Templates

A PR template lives at .github/pull_request_template.md (GitHub) or .gitlab/merge_request_templates/Default.md (GitLab) and auto-fills the PR description.

## Goal
What does this PR accomplish?

## Changes
- bullet
- bullet

## Testing
How did you verify it?

## Checklist
- [ ] Title is a clear sentence
- [ ] Commits are signed
- [ ] Docs / submission file updated
  • βœ… Templates make reviews predictable β€” the same sections every time
  • βœ… They surface what you tested before reviewers have to ask
  • ❌ Keep them short. Long templates get deleted by the author

πŸ“ Slide 18 – 🦊 GitHub or GitLab? Both Are Fine

Feature GitHub GitLab
Hosting github.com gitlab.com or self-hosted CE
Branch protection βœ… βœ…
Code review Pull Request Merge Request
CI GitHub Actions GitLab CI/CD
SSH commit signing βœ… (verified badge) βœ… (verified badge)
Free private repos βœ… βœ…
  • 🚫 Some students lose access to GitHub for reasons outside their control β€” sanctions, account locks, country-of-origin filters
  • βœ… Every lab in this course offers a GitLab path. Lab 3 explicitly mirrors the CI pipeline to .gitlab-ci.yml as a Bonus Task
  • πŸ›οΈ Innopolis itself runs an internal GitLab at gitlab.pg.innopolis.university β€” that's your safety net

πŸ€” Think: The concepts β€” branches, PRs, CI β€” are platform-independent. The tool is replaceable; the discipline isn't.


πŸ“ Slide 19 – πŸ”₯ When DevOps Wasn't There: Real Incidents

πŸ—“οΈ Date 🏒 Who πŸ’₯ What broke πŸŽ“ What it teaches
2012-08-01 Knight Capital Stale code on 1 of 8 servers β†’ $440M in 45 min Automate deploys; never trust manual checklists
2017-01-31 GitLab.com Engineer rm -rf-ed primary DB; 5 of 5 backups broken Test your restores, not just your backups
2017-02-28 AWS S3 us-east-1 Typo in a maintenance command removed too many capacity nodes Blast-radius limits; canaries; reversible commands
2024-07-19 CrowdStrike Faulty kernel-mode update β†’ 8.5M Windows hosts in BSOD loop Staged rollouts; reversible deploys; vendor risk
2024-03 xz-utils backdoor 2-year social-engineering attack on a single maintainer Code provenance; signed commits; SBOMs (Lab 9)

πŸ“ Read: GitLab database incident postmortem (2017) β€” frank, painful, and instructive


πŸ“ Slide 20 – πŸͺΆ Blameless Postmortems

A postmortem is a written record of what happened, why, and what changes. Done well, it's the most valuable artifact of an outage.

πŸ”₯ Blameful βœ… Blameless
"Alice pushed bad code" "The deploy pipeline allowed an untested change to reach prod"
"Bob didn't read the runbook" "The runbook was 41 pages long and last updated in 2019"
Punishment, hidden incidents Learning, surfaced incidents, future-proof fixes
  • πŸ§ͺ The goal is to make the system safer, not to find a person to blame
  • πŸ“š Norms set by John Allspaw at Etsy (~2012) and codified in the Google SRE Workbook, Chapter 9
  • βœ… Every serious lab in this course ends with a submissions/labN.md β€” your mini-postmortem of how the lab went

πŸ’¬ "You will never make a system safer by punishing the humans inside it." β€” Sidney Dekker, The Field Guide to Understanding Human Error


πŸ“ Slide 21 – 🧠 Key Takeaways

  1. πŸ’‘ DevOps is a culture that puts Dev and Ops on the same team with the same goals β€” not a job title
  2. πŸ›€οΈ Three Ways: Flow, Feedback, Continual Learning
  3. πŸ“Š DORA's four metrics make DevOps measurable: deploy frequency, lead time, change failure rate, MTTR
  4. 🌳 Git is the spine of every DevOps workflow β€” fork, branch, PR, sign your commits
  5. 🀝 GitHub or GitLab β€” pick what works for you. Both are first-class in this course
  6. πŸͺΆ Blame systems, not people β€” the postmortem you write after a bad day is the most valuable thing you'll produce that week

πŸ’¬ "It works on my machine" is no longer an acceptable answer.


πŸ“ Slide 22 – πŸš€ What's Next + πŸ“š Resources

  • πŸ“ Next lecture: Version Control deep dive β€” Git's object model, reset/reflog, tagging, modern commands
  • πŸ§ͺ Lab 1: Fork the QuickNotes repo, configure SSH commit signing, add a PR template, open your first PR (GitHub or GitLab), engage with one open-source project as community proof
  • πŸ“– Read this week:
    • πŸ“• The Phoenix Project β€” Kim, Behr & Spafford (2013) β€” Chapters 1-3
    • πŸ“— The DevOps Handbook β€” Kim, Humble, Debois & Willis (2nd ed 2021) β€” Part I
    • πŸ“˜ Pro Git β€” Chacon & Straub β€” Chapters 1-2 (free)
  • πŸŽ₯ Watch: Allspaw & Hammond β€” "10+ Deploys per Day" (2009)
  • πŸ“ Knight Capital SEC filing on the 2012 incident
  • πŸ“ Patrick Debois on the term "DevOps" β€” DevOpsDays origin
graph LR
    Y["πŸ“ You Are Here<br/>DevOps + Git Foundations"] --> N["πŸ“¦ Week 2<br/>Version Control Deep Dive"]
    N --> M["πŸ€– Week 3<br/>CI/CD"]
    M --> K["πŸ’» Week 4<br/>OS & Networking"]
Loading

🎯 Remember: Every lab in this course is a small experiment in shipping software safely. The tools change every five years. The discipline doesn't.