Collaborative annotation for clinical AI

I was the sole designer for Medal’s clinical annotation platform. Medical experts used it to review and correct AI-extracted data from medical records. I designed per-session versioning, an admin-managed labeling system, and a faster annotation menu, over about two months starting November 2019.

• I shipped a versioning system that let multiple annotators work on the same document independently.
• I killed a notification-based design before build. That avoided a HIPAA risk and engineering work that did not fit the timeline.
• I interviewed three medical coders and one physician. The interviews surfaced the annotation menu as a workflow bottleneck, and my findings put the menu redesign on the roadmap.
• Ciox Health acquired Medal six months after my work ended. The announcement credits consensus review by clinical experts; my system was the tool for that review. The acquisition is context, not my result.

Medal was acquired before long-term adoption could be measured. The results here are what I can verify: what shipped, what I cut before build, and what the research changed.

Medal was a San Francisco biomedical NLP company. Its software extracted structured data from the unstructured parts of medical records: surgical reports, pathology reports, discharge summaries. It installed alongside any EMR system and created a secure, HIPAA-compliant shared record for each patient encounter.

Expert review made the extracted data trustworthy. The acquisition announcement says clinical expert reviewers guided Medal’s NLP approach through consensus. My job was to design the workflow those reviewers used. Ciox Health retrieves more than 100 million medical records each year, and acquired Medal in July 2020 to supply research-grade real-world data to pharma, biotech and government researchers.

Medal’s NLP extracted clinical data from unstructured records. Experts had to review and correct the output before anyone could trust it. Errors in clinical data reach real decisions about real patients.

The existing tool slowed that review down. Labels lived in one long dropdown with a search field and no grouping. Annotators had no way to keep independent versions of their work on the same document. The constraints: a small team, HIPAA rules on all patient data, and about two months.

My first concept was a task list with email notifications. Each notification would link to an updated document version. I killed it for two reasons. We could not build version-linked notifications within the timeline. And HIPAA limited the email content to “An update on [file] made on [date],” which told the reader nothing.

The first sketch: a task list with notifications.

Working out a changelog with post-its, before versioning replaced it.

So I versioned the work instead of pushing updates. Each annotator’s session saves as its own version. Only that annotator and the admin can see it. Annotators work in parallel without conflicts. The admin compares all versions and selects one to show on the patient profile, and can invite specific users to annotate a version.

The tradeoff: no real-time collaboration and no visibility between annotators. For clinical review, that independence has a benefit. Each annotator forms their own judgment instead of anchoring on the first version they see.

Expand

Editor view: each session saves as its own version.

Expand

Admin view: compare all versions, select one, invite others.

Label sets differ by team and record type. I could not design one taxonomy for everyone, so I designed the tool that lets admins build their own. The admin portal requires a category before any label, which forces admins to structure the menu their team will use.

Expand

Labels take custom colors. I worried about contrast, because a misread label has clinical consequences. I added a live preview and forced black or white text based on the chosen color. CSV bulk import was designed but deferred past the MVP; it was implemented later.

Expand

First run: nothing exists until a category does.

Expand

Editing a label: the preview shows the chip in its final contrast.

Expand

Deleting a category asks first, since its labels go with it.

I interviewed three medical coders and one physician. The annotators asked for a faster way to move through the label menu. Search helped, but they still scrolled a long flat list, and the open menu covered document text they needed to read. The menu was not on the roadmap. I recorded a session, presented my findings, and made the case that annotation speed was the product’s core loop. The team prioritized the redesign.

Expand

Exploration: where the menu opens.

Expand

Exploration: hotkeys.

Expand

Exploration: menu length against the document.

I added hotkeys for frequent users and color-coded labels for recognition at a glance. Default colors match Medal’s patient summary conventions: blue for problems, purple for allergies, orange for medications.

Expand

Final rounds of the color-coded menu.

Expand

As shipped: labels colored by category, hotkeys for the frequent ones.

The versioning system, the admin portal and the menu redesign shipped. CSV import followed later. Ciox acquired Medal in July 2020 to power its real-world data business. The announcement credits consensus review by clinical experts. I designed the tool for that review. I cannot claim the acquisition, and I don’t.

The lesson is that I left before the feedback loop closed. Next time I would instrument the work before shipping. Light measures would do: how many admins built custom menus in month one, or median time to apply a label before and after hotkeys. A design’s value should not depend on the designer being in the room. Instrumentation is now part of my design specs.

Why was the menu worth reprioritizing?

Annotation speed was the business. Medal sold expert-corrected data. Every second spent scrolling the dropdown was cost. When interviewees hit the same wall, that was the core loop degrading, not a cosmetic issue.

Was a first solution wrong?

Yes, the notification system. The only HIPAA-compliant email we could send carried no useful information, and we could not build version-linked notifications in the timeline. I reframed the problem: version each session instead of pushing updates. The admin’s side-by-side view answered “what changed” better than email could.

How early was engineering involved?

Early. The notification concept died on feasibility before I invested in high-fidelity work. With four engineers and a short timeline, designing over the wall was not an option.

What changed between design and ship?

CSV bulk import was deferred. Manual label creation shipped first, and import followed.