Shadow AI in healthcare

If you’re building in clinical AI, one of the strongest market signals you can get is that clinicians are already using generative AI tools every day - often outside formal approval.

Offcall’s 2025 Physicians AI Report (based on self-reported survey data) suggests that 67% of clinicians use AI daily in their practice. 84% say AI makes them better at their job, 81% feel dissatisfied with their employer's AI adoption speed, 71% have little to no influence on which AI tools get used and 48% say employer communication about AI is poor.

What this points to is a clear tension between high market readiness and low trust infrastructure.

Why this matters for your product strategy

We often treat shadow AI as a hospital governance issue. In reality, it can help shape your product-market fit.

Clinicians are adopting tools because they reduce friction in:

• documentation and admin load;

• information synthesis; and

• cognitive burden in complex decision environments.

That means your competitive edge lies in:

• workflow integration;

• low friction user experience;

• safe-by-design guardrails; and

• auditability.

If your tool increases clicks, adds extra steps or creates uncertainty about accountability, it is likely to be bypassed.

The hard truth: accountability doesn’t disappear just because AI is involved

A generative model is not a legal actor. It cannot hold professional responsibility. When something goes wrong, scrutiny lands on humans and organisations: clinicians, employers and, in some cases, vendors, depending on claims, intended use and contractual allocation of responsibilities.

For AI companies, the key question are:

• What are you representing the tool does?

• Where does clinical decision making sit?

• What evidence supports performance in the intended setting?

• Can you demonstrate control: logging, versioning, change management and incident response?

Regulatory classification is not academic - it is a roadmap constraint

A recurring market mistake is allowing “helpful” features to drift into regulated clinical functionality without acknowledging the consequences.

If your system is used to inform diagnosis, triage, treatment decisions or clinical management, you are likely to be in medical device territory - depending on your jurisdiction and the specifics of intended use, claims, and implementation. Even where classification is nuanced, procurement teams increasingly behave as if the bar is rising: they want a clinical safety case, clear governance and evidence that is relevant to real workflows.

A practical way to think about this:

• If your tool is administrative (drafting letters, summarising non-identifiable text, patient-friendly explanations), your compliance looks one way.

• If your tool is clinical (recommending, prioritising, interpreting, deciding), your compliance must look very different: risk management, validation, change control, post-market monitoring and stronger controls around data and outputs.

Clinician behaviour research points to a product-design opportunity

A useful insight from observational research on clinicians using GPT-4 in vignette-based clinical reasoning is that “paste more context” is not a reliable path to better performance. Clinicians used a range of input styles (copy-paste, selective paste, summarise, search-like prompts), and no single style consistently improved reasoning scores.

Founders should read that as a UX and safety design cue:

• Your differentiator is not “more context.”

• It is structured relevance, interrogation workflows, and safe integration.

In practice, that means building tools that help users:

• surface what matters (not everything),

• challenge the model (not accept it),

• and create traceability (not ambiguity).

Governing for adoption: make the safe path the easy path

Hospitals don’t just buy technology; they buy governance. Clinicians don’t just want rules; they want tools that work.

The winning pattern is to reduce workarounds by designing “approved” pathways that are:

• low friction,

• embedded into clinical systems,

• and transparent in how outputs are generated, constrained, and logged.

This is where companies can lead rather than react:

• Provide procurement-ready documentation (security pack, DPIA support posture, audit logging, model update policy).

• Provide a clear responsibility matrix (what the tool does/doesn’t do; what the clinician must do).

• Provide change control and monitoring commitments (how updates happen; how drift is detected; how incidents are handled).

What sophisticated buyers will ask you in the room

If you are selling into healthcare, expect these questions—often from legal, IG, clinical safety, and procurement together:

1. Intended use and claims: what exactly are you saying the tool does?

2. Data flows: what data is processed, where, by whom, and with what safeguards?

3. Auditability: can we reconstruct what happened for any output?

4. Safety management: how do you test, monitor, and handle incidents?

5. Model updates: how do you control and communicate changes?

6. Liability and responsibility: what sits with you vs the hospital vs clinicians?

7. Evidence: what supports performance in our population and workflow?

8. Human oversight: what does “human-in-the-loop” actually mean operationally?

A founder who can answer these cleanly accelerates adoption; a founder who can’t will stall.

Where outside senior legal counsel creates leverage

This is precisely the gap for many clinical AI founders: you don’t need a large legal team but you do need senior judgement at the intersection of product, regulation, clinical safety and commercial contracting.

Fractional senior legal support typically unlocks three things:

• Speed: procurement cycles move faster when you have coherent risk narratives and buyer-ready documentation.

• Defensibility: you avoid accidental over-claims, misclassification risk and governance gaps that create downstream exposure.

• Scale: you standardise contracting so that each deployment doesn’t reinvent the wheel.

Bottom line

Shadow AI is not just a compliance headache. It is evidence that clinicians are ready for tools that reduce friction—andevidence that healthcare systems will punish solutions that are hard to govern.

The winners in clinical AI will be those who treat legal, regulatory, and clinical safety not as blockers, but as product features: the architecture of trust that makes adoption sustainable.