Testing Jev: Three Playground Runs

I got early access to Jev after writing about it 2d ago. Here are three tests I ran in the playground, with the actual state, questions, and answers.

Is a Jedi sandwich a sandwich?

State:

{
  "food": "Jedi sandwich",
  "definition": "Put Luke Skywalker in between two slices of toast."
}

Question:

{
  "is_sandwich": {
    "type": "noul",
    "instructions": "Is `food` a sandwich?",
    "criteria": {
      "true": "A sandwich is a food dish where a filling, such as meat, cheese, vegetables, or spread, is placed between structural starch",
      "false": "The food has no bread enclosing a filling or uses only a single slice of bread, or uses a non-bread wrapper such as a tortilla, wafer, or cookie."
    }
  }
}

Answer:

{ "is_sandwich": { "type": "noul", "noul": 0.82 } }

82% true. Jev took the definition literally: a person between two slices of toast fits “filling between structural starch,” so it leans sandwich. It didn’t push back on the premise being absurd. It matched the criteria text against the state, the way the docs say it would.

What color is the sky, with no sky in view

I asked the same question two ways. No state, just the question, so the answer comes from whatever the model already associates with “sky.”

Plain options:

{
  "sky_color": {
    "type": "choice",
    "instructions": "What color is the sky?",
    "criteria": {
      "indigo": "",
      "electric blue": "",
      "baby blue": "",
      "gray": "",
      "lavender": "",
      "salmon": "",
      "seafoam green": ""
    }
  }
}

{
  "sky_color": {
    "type": "choice",
    "choice": "seafoam green",
    "confidence": 0.54,
    "probabilities": {
      "indigo": 0.03,
      "gray": 0.36,
      "seafoam green": 0.61,
      "lavender": 0,
      "baby blue": 0,
      "salmon": 0,
      "electric blue": 0
    }
  }
}

Same options, each with a hex code, an RGB triplet, an HSL triplet, and a short description (I’m showing “indigo” here; the other six options carried the same kind of detail):

{
  "sky_color_with_desc": {
    "type": "choice",
    "instructions": { "object": "sky", "question": "what color is `object`?" },
    "criteria": {
      "indigo": {
        "description": "very dark, deep blue, like fountain-pen ink",
        "hex": "#280868",
        "rgb": { "red": 40, "green": 8, "blue": 104 },
        "hsl": { "hue": 260, "saturation": "86%", "lightness": "22%" }
      }
    }
  }
}

{
  "sky_color_with_desc": {
    "type": "choice",
    "choice": "indigo",
    "confidence": 0.32,
    "probabilities": {
      "indigo": 0.43,
      "gray": 0.2,
      "seafoam green": 0.37,
      "lavender": 0,
      "baby blue": 0,
      "salmon": 0,
      "electric blue": 0
    }
  }
}

Neither run picked “electric blue” or “baby blue,” the two closest to how most people would answer this. The plain version picked “seafoam green” over “gray.” Adding descriptions flipped the top answer to “indigo” and dropped confidence from 0.54 to 0.32.

The only thing that changed between the two calls was how the criteria field described each option — a label for the option, not the question itself. That label moved the answer. Writing criteria for Jev is prompt engineering, same as writing a prompt for any other model. A schema keeps the output well-formed. It doesn’t make the wording of your options neutral.

Who gets credit for the monkey selfie

This one uses the real 2011 case: a wildlife photographer’s camera got loose in a nature reserve, and a macaque named Naruto pressed the shutter and produced a self-portrait.

State:

{
  "scenario": "Naruto is a Celebes crested macaque living in the Tangkoko nature reserve in North Sulawesi, Indonesia. David Slater, a wildlife photographer shooting macaques in the reserve, leaves his camera unattended, and Naruto repeatedly activates the shutter, producing hundreds of images, including a remarkably sharp, grinning self-portrait reminiscent of a human selfie. Slater later processes and publishes the best photographs in a book that names him as the copyright owner. The book states that Naruto took the photographs, with captions like, 'Surely a sign of self-awareness?' Another caption reads, 'Naruto the macaque smiles at itself while pressing the shutter button on a camera.'",
  "subject": "Naruto",
  "human": "Slater",
  "creative_work": "the grinning self-portrait"
}

Questions:

{
  "subject_contribution": {
    "type": "score",
    "instructions": "How much did `subject` contribute to `creative_work`?",
    "criteria": ["None", "Minorly", "Moderately", "Majorly", "Completely"]
  },
  "human_contribution": {
    "type": "score",
    "instructions": "How much did `human` contribute to `creative_work`?",
    "criteria": ["None", "Minorly", "Moderately", "Majorly", "Completely"]
  }
}

Answers:

{
  "subject_contribution": { "type": "score", "score": 3.47, "confidence": 0.56 },
  "human_contribution": { "type": "score", "score": 1.18, "confidence": 0.69 }
}

Naruto scored 3.47 out of 4, between “Majorly” and “Completely.” Slater scored 1.18, just above “Minorly.” That tracks the state: Slater set up the camera and left it, Naruto pressed the shutter and produced the image. Jev split the credit along the facts it was given, in two numbers, with no explanation attached.

Takeaway

The Naruto case shows what the pitch is for: a fact pattern in, a defensible number out, fast. The sky-color case shows the part to watch. The typed schema stops Jev from returning garbage. It doesn’t stop your own wording of the options from quietly changing the answer.

You have to craft the input description carefully, or you won’t get the result you want. Designing the input schema is as core to using Jev as writing the prompt is to using an LLM.