Skip to content

P4: properties for permission evaluate() - #27

Merged
anjali-d15 merged 7 commits into
mainfrom
anjalid/properties
Nov 13, 2026
Merged

anjali-d15 merged 7 commits into
mainfrom
anjalid/properties

Conversation

@anjali-d15

Copy link
Copy Markdown
Contributor

Issue for this PR

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

Adds fast-check property-based tests for 4 of my Project 3 permission-rule requirements (R3, R4, R5, R6), testing the real evaluate() function in packages/opencode/src/permission/index.ts. Covers: deny-rule consistency against an independent reference model (R3), rule removal round-tripping (R4), rule order determining precedence (R5), and a specific rm * rule overriding a broader catch-all (R6). All 4 properties pass against the current implementation, so no production code changes were needed. Also added traceability/anjalid.md mapping each requirement to its interview evidence and property.

How did you verify your code works?

Ran bun test test/permission/permission.property.test.ts locally

Screenshots / recordings

If this is a UI change, please include a screenshot or recording.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

If you do not follow this template your PR will be automatically rejected.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed you changed the timeout minutes from 10 to 15. The permission tests you wrote seem to be pure functions and should finish pretty quickly. Was the timeout bump to 15 minutes needed for this PR?

fc.property(rulesetArb, ruleArb, patternCommandArb, (baseRules, newRule, [, command]) => {
const before = evaluate(PERMISSION, command, baseRules as any)
const withAdded = [...baseRules, newRule]
const withRemoved = withAdded.slice(0, -1)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From what I can tell, this should still give back baseRules without changing it, since it adds the new rule to the end of the array then slices it off. Could you remove a rule from a random index instead, and check that the result equals referenceEvaluate(command, remaining) and that the removed rule is never what gets returned?

@dargon327

Copy link
Copy Markdown
Contributor

nice work, the R3 test that checks evaluate() against a simpler version is a really smart approach. Your traceability file is also easy to follow, and I like that you were upfront about R7 and R8 not being implemented yet.

@dylany3

dylany3 commented Nov 9, 2026

Copy link
Copy Markdown

Hi Anjali, I'm your red-teamer for P5. Here are my three mutants, written against commit 83ef6d4e5b90e94c27522772a5bbb0c465422cb2. Your properties passed on all three when I ran them.

RT1 (R1)

--- a/packages/opencode/src/permission/index.ts
+++ b/packages/opencode/src/permission/index.ts
@@ -97,7 +97,7 @@ const layer = Layer.effect(
         yield* Effect.logInfo("evaluated", { permission: request.permission, pattern, action: rule })
         if (rule.action === "deny") {
           return yield* new PermissionV1.DeniedError({
-            ruleset: ruleset.filter((rule) => Wildcard.match(request.permission, rule.permission)),
+            ruleset: ruleset.filter((rule) => Wildcard.match(pattern, rule.permission)),
           })
         }
         if (rule.action === "allow") continue

Uses pattern instead of request.permission, so the denial error comes back with an empty ruleset and doesn't show which rule caused it. None of the properties check the error, so it gets through.

RT2 (R3)

--- a/packages/core/src/util/wildcard.ts
+++ b/packages/core/src/util/wildcard.ts
@@ -8,7 +8,7 @@ export function match(input: string, pattern: string) {
     .replace(/\*/g, ".*")
     .replace(/\?/g, ".")
 
-  if (escaped.endsWith(" .*")) escaped = escaped.slice(0, -3) + "( .*)?"
+  if (escaped.endsWith(" .*")) escaped = escaped.slice(0, -3) + ".*"
 
   return new RegExp("^" + escaped + "$", process.platform === "win32" ? "si" : "s").test(normalized)
 }

Now rm * also matches commands like rmdir build. The commands in the tests only come from PATTERN_COMMAND_PAIRS or start with rm , so nothing like rmdir gets tested.

RT3 (R5)

--- a/packages/opencode/src/permission/index.ts
+++ b/packages/opencode/src/permission/index.ts
@@ -34,7 +34,7 @@ export function evaluate(permission: string, pattern: string, ...rulesets: Permi
   return (
     rulesets
       .flat()
-      .findLast((rule) => Wildcard.match(permission, rule.permission) && Wildcard.match(pattern, rule.pattern)) ?? {
+      .findLast((rule) => Wildcard.match(pattern, rule.pattern)) ?? {
       action: "ask",
       permission,
       pattern: "*",

Drops the permission check, so a rule for a different permission (like edit) can decide a bash command if it comes last. All the generated rules use "bash", so this never comes up in the tests.

The patches are also in mutants/dylany/redteam/ on our repo. Good luck!

@anish-jai anish-jai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@anjali-d15
anjali-d15 merged commit b51d75a into main Nov 13, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants