- ποΈ August 1, 2012, 9:30 a.m. ET β Knight Capital, the largest US equities trader, pushes a new release of its order-routing software (SMARS) to 8 production servers
- π¨ One server was missed during the manual deploy β it kept running old, dead code that re-activated a long-disused flag called Power Peg
- πΈ In the first 45 minutes of market open, the rogue server submitted 4 million erroneous orders, accumulating a $440 million loss
- π¦ Knight had to be rescued, then acquired by Getco by December 2012 β gone as an independent firm
- π Lesson: A single bad deploy can sink a 1,000-person company. How you ship is now as important as what you ship
π€ Think: Knight had top-tier traders and top-tier code. What they didn't have was DevOps β automated deploys, kill switches, rollback. Could the same incident hit you tomorrow?
By the end of this lecture you will:
| # | π Outcome |
|---|---|
| 1 | β Explain what DevOps is, who coined the term, and the problem it solves |
| 2 | β Describe the Three Ways and the CALMS framework |
| 3 | β Cite the four DORA metrics and what elite vs low performance looks like |
| 4 | β Use Git's mental model: working tree β index β repository |
| 5 | β Set up signed commits and a PR template to make collaboration safer |
| 6 | β Recognize that GitHub and GitLab are both valid paths for the rest of this course |
graph LR
A["π₯ Knight Capital"] --> B["π DevOps History"]
B --> C["π€οΈ Three Ways<br/>CALMS"]
C --> D["π DORA Metrics"]
D --> E["π³ Git Workflow"]
E --> F["π Signed Commits<br/>PRs"]
F --> G["π Real Incidents<br/>Blameless Postmortems"]
- π Slides 1-4 β Why DevOps exists at all
- π Slides 5-9 β History, frameworks, and how to measure DevOps
- π Slides 10-13 β Git from first principles
- π Slides 14-18 β Your course project, forkβPR, signed commits, templates
- π Slides 19-22 β Real incidents, culture, and what Lab 1 asks of you
Before DevOps had a name, software shops looked like this:
graph LR
D["π¨βπ» Dev<br/>'Ship features fast!'"] -- π§± throws code over --> O["π‘οΈ Ops<br/>'Stop breaking prod!'"]
O -- π§± throws blame back --> D
- π’ Dev got rewarded for new features β pushed code as fast as possible
- π‘οΈ Ops got rewarded for uptime β resisted every release
- π₯ The incentives directly conflicted, so every deploy was a fight
- π’ Releases happened once a quarter, at 2 a.m., under a freeze, with rollback plans written in Word docs
π¬ "It's like a relay race where the runners hate each other." β paraphrasing John Allspaw, ca. 2009
- π¨π¦ August 2008, Toronto β at Agile Conference, Andrew Shafer proposes a session called "Agile Infrastructure". Almost no one shows up β except Patrick Debois
- π§πͺ October 30β31, 2009, Ghent β Debois organizes the first DevOpsDays, named so it would fit a Twitter hashtag
- π¦ The conversation explodes on
#devopsand never stops - πΊ Earlier that year (June 2009 at Velocity), John Allspaw and Paul Hammond of Flickr give the legendary talk "10+ Deploys per Day: Dev and Ops Cooperation" β proof the model works
- π 2013 β Gene Kim's novel The Phoenix Project makes DevOps mainstream literature
π¬ "DevOps was a movement before it was a term." β Patrick Debois
From The Phoenix Project (2013) and The DevOps Handbook (2016):
| # | Way | Mental model | What it asks of you |
|---|---|---|---|
| 1 | π€οΈ Flow | Left-to-right: idea β code β prod | Optimize the whole stream, not the local task |
| 2 | π Feedback | Right-to-left: prod telling Dev what happened | Shorter loops, more signals, no blame |
| 3 | π§ͺ Continual Learning | Experiment + amplify what worked | Blameless postmortems; deliberate practice |
π€ Think: Which "Way" was Knight Capital missing the most?
| Letter | What it means | Concrete signal |
|---|---|---|
| C | Culture | Devs and Ops sit together; nobody says "throw it over the wall" |
| A | Automation | A deploy is a git push, not a 14-step runbook |
| L | Lean | Small batch sizes; finish a thing before starting the next |
| M | Measurement | You can name your deploy frequency and MTTR off the top of your head |
| S | Sharing | Postmortems, blogs, internal demos β knowledge is not hoarded |
Coined as CAMS by John Willis & Damon Edwards in 2010; L for Lean added later by Jez Humble.
π¬ "If you can't measure it, you can't improve it." β Peter Drucker, often invoked by the M
The DevOps Research and Assessment group (DORA, founded ~2014, acquired by Google 2018) studied tens of thousands of teams. Four metrics predict performance:
| Metric | What it captures | Elite teams |
|---|---|---|
| π Deployment frequency | How often you ship to prod | Multiple times per day |
| β±οΈ Lead time for changes | Commit β in production | < 1 hour |
| π Change failure rate | % of deploys that need a fix | 0β15% |
| π οΈ MTTR | Mean time to recover from an incident | < 1 hour |
From 2021, a fifth metric β Reliability β was added to track operational stability.
π Read: Accelerate by Forsgren, Humble & Kim (2018) β the scientific paper-form of these findings
graph LR
L["π Low<br/>once / 6 mo<br/>MTTR weeks"] --> M["πΆ Medium<br/>weekly<br/>MTTR days"]
M --> H["π High<br/>daily<br/>MTTR hours"]
H --> E["ποΈ Elite<br/>on demand<br/>MTTR < 1h"]
- π Low performers and Elite performers differ by ~200Γ on deployment frequency
- π Elite teams also have lower change failure rates β speed and safety go together
- β "Move fast and break things" is not the elite pattern. Elite is move fast and don't break things, because feedback loops are tight
π€ Think: Which bucket does the project you'll graduate to next year sit in? Which would you want it in?
DevOps needs a single source of truth for code, infra, and process. Git is that truth:
- π¦ The Git repository is all of history plus the current state β every laptop has a full copy
- π€ Pull requests turn "review my code" into a collaboration protocol, not a favor
- π Branches let you experiment without breaking the trunk
- π€ CI/CD systems (next week) react to Git events β a push triggers a pipeline, a tag triggers a release
π¬ "Git is the assembly language of collaboration." β folklore, attributed to many
- π§ April 2005 β Linus Torvalds writes Git in roughly 10 days after the Linux kernel loses access to BitKeeper
- π― Design goals: speed, distributed model, integrity (cryptographic hashes), strong support for non-linear development
- π On the name: "I'm an egotistical bastard, and I name all my projects after myself. First Linux, now Git." β Linus
- πͺͺ In November 2021 (Git 2.34), Git gained native SSH commit signing β no GPG keyring required
- π By April 2026 we're on Git 2.49+ β
git switch,git restore,git maintenanceare the modern ergonomics
π Book: Pro Git (Scott Chacon & Ben Straub) β free at git-scm.com/book
graph LR
W["π Working tree"] -- "git add" --> I["π Index<br/>(staging)"]
I -- "git commit" --> R["ποΈ Local repo"]
R -- "git push" --> O["βοΈ Remote"]
O -- "git fetch / pull" --> R
| Stage | What it is | Common command |
|---|---|---|
| π Working tree | Files on disk you can edit | git status |
| π Index | "What I plan to commit" | git add -p |
| ποΈ Local repo | Permanent history on your machine | git commit -s -S |
| βοΈ Remote | Shared truth (GitHub / GitLab) | git push origin feature/x |
π€ Think: Why does Git separate "Index" from "Working tree"? (Hint: ever made a typo just before committing?)
| Strategy | Branch lifetime | Where it shines |
|---|---|---|
| π² Trunk-based | Hours | Continuous delivery, large teams, feature flags |
| π³ GitHub Flow | Days | SaaS apps, this course |
| π΄ GitFlow | Weeks-Months | Versioned products with long support windows |
- β
This course uses GitHub Flow β short-lived
feature/labNbranches, one PR per lab - β Avoid: a
developbranch nobody merges, six-month feature branches that never come home, "let's just push to main on Friday afternoon"
π¬ "The longer your branch lives, the more it hurts to merge." β every senior engineer eventually
Across all 10 labs you'll work on one small Go service: QuickNotes.
graph LR
C["π» curl / browser"] -->|HTTP :8080| S["π’ quicknotes<br/>Go 1.24"]
S --> F["π data/notes.json"]
S -->|GET /metrics| P["π Prometheus<br/>(Week 8)"]
- π’ Endpoints:
GET /notes,GET /notes/{id},POST /notes,DELETE /notes/{id},GET /health,GET /metrics - π οΈ You don't write the app β you package it, ship it, observe it, and harden it
- π By Lab 10, the same code will live behind your own CI, in a container, on a cluster, behind monitoring and security scans, deployed to a real cloud
π€ Think: Why one project for 10 weeks? Because that's how DevOps actually feels at work.
sequenceDiagram
participant U as π€ You
participant F as π΄ Your fork
participant C as π Course repo
U->>F: git clone
U->>F: git switch -c feature/lab1
U->>F: edit, commit, push
U->>C: open PR (fork:feature/lab1 β course:main)
C->>U: reviewer comments
U->>F: fixup commits
C->>U: β
merge / feedback
- 1οΈβ£ Fork the course repo on GitHub or GitLab
- 2οΈβ£ Clone your fork locally
- 3οΈβ£ Branch per lab:
feature/lab1,feature/lab2, β¦ - 4οΈβ£ Commit small, signed, and with a message that reads like a sentence
- 5οΈβ£ Push to your fork, never directly to the course repo
- 6οΈβ£ Open a PR back to the course repo's
main
- π΅οΈ Anyone can put any name and email in
git configβ your commit history is unauthenticated by default - π₯ Real consequences: in March 2024, an attacker (account
JiaT75) maintained the xz-utils project for two years and slipped in a backdoor that nearly compromised every SSH daemon on Linux - πͺͺ A signed commit is a cryptographic claim: "this commit really was made by the holder of this key"
- β Git supports SSH signing since 2.34 (Nov 2021) β no GPG keyring, just reuse your SSH key
# β
enable SSH signing (one-time)
git config --global gpg.format ssh
git config --global user.signingkey ~/.ssh/id_ed25519.pub
git config --global commit.gpgsign trueπ€ Think: Would you merge a PR from a contributor whose commits showed "Unverified" on GitHub?
A PR template lives at .github/pull_request_template.md (GitHub) or .gitlab/merge_request_templates/Default.md (GitLab) and auto-fills the PR description.
## Goal
What does this PR accomplish?
## Changes
- bullet
- bullet
## Testing
How did you verify it?
## Checklist
- [ ] Title is a clear sentence
- [ ] Commits are signed
- [ ] Docs / submission file updated- β Templates make reviews predictable β the same sections every time
- β They surface what you tested before reviewers have to ask
- β Keep them short. Long templates get deleted by the author
| Feature | GitHub | GitLab |
|---|---|---|
| Hosting | github.com | gitlab.com or self-hosted CE |
| Branch protection | β | β |
| Code review | Pull Request | Merge Request |
| CI | GitHub Actions | GitLab CI/CD |
| SSH commit signing | β (verified badge) | β (verified badge) |
| Free private repos | β | β |
- π« Some students lose access to GitHub for reasons outside their control β sanctions, account locks, country-of-origin filters
- β
Every lab in this course offers a GitLab path. Lab 3 explicitly mirrors the CI pipeline to
.gitlab-ci.ymlas a Bonus Task - ποΈ Innopolis itself runs an internal GitLab at
gitlab.pg.innopolis.universityβ that's your safety net
π€ Think: The concepts β branches, PRs, CI β are platform-independent. The tool is replaceable; the discipline isn't.
| ποΈ Date | π’ Who | π₯ What broke | π What it teaches |
|---|---|---|---|
| 2012-08-01 | Knight Capital | Stale code on 1 of 8 servers β $440M in 45 min | Automate deploys; never trust manual checklists |
| 2017-01-31 | GitLab.com | Engineer rm -rf-ed primary DB; 5 of 5 backups broken |
Test your restores, not just your backups |
| 2017-02-28 | AWS S3 us-east-1 | Typo in a maintenance command removed too many capacity nodes | Blast-radius limits; canaries; reversible commands |
| 2024-07-19 | CrowdStrike | Faulty kernel-mode update β 8.5M Windows hosts in BSOD loop | Staged rollouts; reversible deploys; vendor risk |
| 2024-03 | xz-utils backdoor | 2-year social-engineering attack on a single maintainer | Code provenance; signed commits; SBOMs (Lab 9) |
π Read: GitLab database incident postmortem (2017) β frank, painful, and instructive
A postmortem is a written record of what happened, why, and what changes. Done well, it's the most valuable artifact of an outage.
| π₯ Blameful | β Blameless |
|---|---|
| "Alice pushed bad code" | "The deploy pipeline allowed an untested change to reach prod" |
| "Bob didn't read the runbook" | "The runbook was 41 pages long and last updated in 2019" |
| Punishment, hidden incidents | Learning, surfaced incidents, future-proof fixes |
- π§ͺ The goal is to make the system safer, not to find a person to blame
- π Norms set by John Allspaw at Etsy (~2012) and codified in the Google SRE Workbook, Chapter 9
- β
Every serious lab in this course ends with a
submissions/labN.mdβ your mini-postmortem of how the lab went
π¬ "You will never make a system safer by punishing the humans inside it." β Sidney Dekker, The Field Guide to Understanding Human Error
- π‘ DevOps is a culture that puts Dev and Ops on the same team with the same goals β not a job title
- π€οΈ Three Ways: Flow, Feedback, Continual Learning
- π DORA's four metrics make DevOps measurable: deploy frequency, lead time, change failure rate, MTTR
- π³ Git is the spine of every DevOps workflow β fork, branch, PR, sign your commits
- π€ GitHub or GitLab β pick what works for you. Both are first-class in this course
- πͺΆ Blame systems, not people β the postmortem you write after a bad day is the most valuable thing you'll produce that week
π¬ "It works on my machine" is no longer an acceptable answer.
- π Next lecture: Version Control deep dive β Git's object model, reset/reflog, tagging, modern commands
- π§ͺ Lab 1: Fork the QuickNotes repo, configure SSH commit signing, add a PR template, open your first PR (GitHub or GitLab), engage with one open-source project as community proof
- π Read this week:
- π The Phoenix Project β Kim, Behr & Spafford (2013) β Chapters 1-3
- π The DevOps Handbook β Kim, Humble, Debois & Willis (2nd ed 2021) β Part I
- π Pro Git β Chacon & Straub β Chapters 1-2 (free)
- π₯ Watch: Allspaw & Hammond β "10+ Deploys per Day" (2009)
- π Knight Capital SEC filing on the 2012 incident
- π Patrick Debois on the term "DevOps" β DevOpsDays origin
graph LR
Y["π You Are Here<br/>DevOps + Git Foundations"] --> N["π¦ Week 2<br/>Version Control Deep Dive"]
N --> M["π€ Week 3<br/>CI/CD"]
M --> K["π» Week 4<br/>OS & Networking"]
π― Remember: Every lab in this course is a small experiment in shipping software safely. The tools change every five years. The discipline doesn't.