Long lists: value sets

A handful of allergens fits in a contract. Every disease, gene, industry code or product category does not. A value set is that long list, imported from a CSV or JSON file: each entry has a code and a label, and can have synonyms, a definition, codes in other standards, and parents. An entry can have two parents, so the hierarchy is a graph, not only a tree.

code,label,synonyms,parents,code:icd10
lung,Lung cancer,,,C34
nsclc,Non-small cell lung cancer,NSCLC,lung,
sclc,Small cell lung cancer,SCLC,lung,

Every import is checked before anything is saved. Duplicate codes, parents that don't exist and loops are refused, with the loop named. An entry can never vanish from a later version: mark it deprecated instead, with what replaces it, so a stored value never points at nothing. A term two entries share is shown as a warning, and so is a short term in capitals like “MET”. The check says how many entries the new version adds, changes and deprecates. Only then do you import it. A version never changes afterwards.

1. 1

   Point a field at it

   In the field editor, choose a value set instead of your own list. The field is pinned to a version, and can be narrowed to some branches: only cancers, from a list of every disease.

2. 2

   Rules, longest first

   Every name and synonym is matched as whole words, the longest first, so “non-small cell lung cancer” is never also “lung cancer”. A short term in capitals only matches in capitals, so the gene MET isn't the word “met”. A term two entries share never settles anything.

3. 3

   The model sees a shortlist

   What the rules don't settle goes to the model with about twenty candidate entries: what the rules found, what the record holds, and the entries sharing most words with it. Its answer can only be one of them. A record close to nothing goes to a person without a model call.

4. 4

   People search it

   In review, when curating and when labelling an answer key, you search the set by name, synonym or code. Codes are shown with their labels.

Serving along the hierarchy. A caller names entries by code or by exact name; a name that isn't an entry, or that two entries share, is refused rather than guessed. A filter on an entry matches everything under it. An exclusion leaves out everything under it, and also every record tagged only with something above it: when a caller excludes cashew, a dish tagged just “tree nuts” is left out, because it isn't known to be cashew-free. Results carry labels, the name of each code in them, and find_values looks entries up (an MCP tool, and POST /v1/tools/<name>/values).

1. 1

   Upgrade a field, impact first

   When the set gets a new version, a field stays on its own until you move it. Moving shows first what happens to every stored value: an entry replaced by one other is re-pointed on every record, each with a receipt; one split into several goes to a person; one retired with no replacement is kept. Then the next run decides against the new version.

2. 2

   Teach it your words

   Choosing an entry in the review queue, you can say which words in the record mean it (“HER-2/neu”, “kaju”). They are saved on that field, not in the set, and outrank the set's own words there, so they can settle a term the set shares: in your records, “MS” may only ever mean one thing. The next run settles them by rules.

3. 3

   From a list to a value set

   A field's own list can become a value set in one step. Each value becomes an entry whose code is the value itself, so nothing stored, served or pinned on a key changes. A list whose values have words that don't count can't be turned yet: a value set can't hold them.