Applied Research Engineer, Evaluation
Build the evaluation backbone that decides whether a model ships, stays shipped, or gets pulled.
The role
You will build the evaluation systems that gate every model we deploy and every change we ship after. You'll work across customer engagements to design eval harnesses, calibration suites, red-team probes, and replay infrastructure that hold up to a regulator's questions.
About Neuralcraft
Evaluation is the deliverable in regulated work. The model is the easier half. We are looking for someone who treats eval engineering as a first-class discipline — not a checklist that runs after training.
You'll partner with the modeling team on every engagement, set the bar a model has to clear, and own the telemetry that tells us a deployed system still works.
What you'll do
- Design evaluation contracts for new engagements — held-out sets, calibration, abstention, behavioral red-team suites — versioned alongside the model and gated in CI.
- Build the replay infrastructure: counterfactual rerunning of historical decisions, calibration drift telemetry, override-rate dashboards segmented by case type.
- Adversarial testing: jailbreak suites, abuse-pattern probes, prompt-injection evaluation, and the playbook for what happens when something passes.
- Work directly with operators and domain experts to capture ground truth and define what "right" looks like for cases that do not have a clean label.
- Publish methodology — internal write-ups for customers, occasional public posts on how we evaluate the systems we ship.
What we're looking for
- 4+ years in ML, with a meaningful chunk of that time on evaluation, alignment, interpretability, or red-teaming.
- Strong Python; comfort with at least one of pytest-style harnesses, statistical testing, or eval frameworks like lm-eval-harness, Inspect, or your own equivalents you've built.
- A mental model for calibration and abstention that goes beyond accuracy. You can argue about what a good eval looks like for a decision that is allowed to say "I don't know."
- Writing voice. Eval results that nobody reads do not count.
Bonus
- Background in stats, econometrics, or experimental design.
- Experience with regulated ML (financial services, healthcare, public sector).
- Prior work on red-teaming, adversarial robustness, or model interpretability.
- Comfort building internal tools (SvelteKit, FastAPI, or similar) when that is the right path to telemetry.
How we hire
- 01 Intro call 30 min with one of the founders.
- 02 Eval conversation 60 min unpacking how you would evaluate a real Neuralcraft engagement, end to end. Bring opinions.
- 03 Paid work trial 1–2 days, paid at market rate. Build a scoped eval harness on a sanitized customer problem.
- 04 Founder-team conversation 90 min with both founders.
- 05 Offer Within 48 hours of the final round.
How to apply
Hit the button below and you'll land on a short form where you can upload your CV and add a few links to work you'd point us at — a paper, a system you shipped, a write-up. No cover letter required.
- 01 Click every link. Open each project, paper, repo, or write-up linked from your CV and confirm it resolves to the right page. Broken links and dead deploys are the most common reason a strong CV gets passed over.
- 02 Verify your contact details. Email, phone number, location, and the best handle to find you on (LinkedIn, GitHub, X, your own site) — make sure each one is current. We will use exactly what's on your CV to reach you.
- 03 Name your file properly.
FirstName-LastName-CV.pdfis enough. PDF only, under 10 MB.
We reply to every application within five working days, including the ones that aren't a fit.