Measure them against the business outcomes they were brought in to move — pipeline, CAC, retention, a successful market entry — not against activity like campaigns launched or content published. Those outputs matter only if they're clearly tied to the metric the engagement was scoped around.
It also helps to check a softer signal: is the in-house team getting sharper and more capable under their direction, or more dependent on them. The former is the sign of a fractional CMO doing the job right.