AI Scribe Feature - Usage Guide

OverviewCopy Anchor Link

The AI Scribe is an intelligent assistant that provides real-time transcription and AI-powered clinical suggestions during patient encounters. It integrates seamlessly with the Care Plan workflow to enhance documentation efficiency and clinical decision-making.

Status: Production-ready with real backend API integration (December 2025)

FeaturesCopy Anchor Link

🎤 Audio Recording & TranscriptionCopy Anchor Link

• Live Recording: Real-time audio capture with visual indicators via Deepgram SDK
• Voice Activity Detection: Dynamic audio level monitoring
• Speaker Identification: Automatic detection of Doctor vs. Patient speech (diarization)
• Pause/Resume: Control recording flow during encounters
• Confidence Scoring: AI confidence levels for transcription accuracy
• Medical Terminology: Enhanced recognition of medical terms (nova-2-medical model)
• Multi-Language Support: 21 languages including Indian regional languages

🤖 AI-Powered SuggestionsCopy Anchor Link

• Context-Aware: Suggestions based on chief complaint and transcription
• 10 Suggestion Types: Vitals, complaint, diagnosis, medication, test, treatment plan, follow-up, note, advice, diet
• Confidence Scoring: AI confidence levels (0.0-1.0)
• Accept/Reject: Easy approval workflow for suggestions with backend persistence
• Auto-Integration: Accepted suggestions can populate relevant encounter sections
• Multiple AI Providers: OpenAI, Anthropic, Google (configurable per clinic)

🌍 Geography-Based Provider SelectionCopy Anchor Link

The AI Scribe automatically selects the best transcription provider based on clinic location:

• India (IN): Deepgram with medical model
• USA (US): Deepgram
• Other regions: Configurable with fallback support

How to UseCopy Anchor Link

Accessing AI ScribeCopy Anchor Link

1. Open an Encounter: Navigate to any active patient encounter
2. Toggle AI Scribe: Click the "AI Scribe" button in the top-right corner
3. Sidebar Opens: The AI Scribe sidebar appears on the right side

Recording ProcessCopy Anchor Link

1. Start Recording: Click the red "Start Recording" button
2. Monitor Status: Watch the live duration counter and voice activity indicator
3. Pause/Resume: Use pause button to temporarily stop recording
4. Stop Recording: Click the square stop button when finished

AI Suggestions WorkflowCopy Anchor Link

1. Auto-Generation: Suggestions appear automatically during recording
2. Manual Generation: Click the wand icon to generate suggestions anytime
3. Review Suggestions: Each suggestion shows type, confidence, and reasoning
4. Accept/Reject: Use checkmark (accept) or X (reject) buttons
5. Integration: Accepted suggestions automatically update the encounter

Real-Time TranscriptionCopy Anchor Link

• Live Updates: Text appears in real-time as you speak
• Speaker Labels: Different colors for Doctor, Patient, and Unknown speakers
• Timestamps: Each segment includes time of utterance
• Confidence Scores: Visual indicators for transcription accuracy
• Auto-Scroll: Automatically scrolls to show latest transcription

ArchitectureCopy Anchor Link

Real-Time Transcription FlowCopy Anchor Link

1. Audio Capture: Browser MediaStream API captures microphone audio
2. WebSocket Connection: Deepgram SDK establishes live connection with temporary token
3. Streaming: Audio chunks sent to Deepgram in real-time
4. Transcription: Deepgram returns transcription segments with confidence scores
5. Batch Upload: Segments batched every 3 seconds and uploaded to backend
6. AI Processing: Backend generates suggestions based on accumulated transcript

Component InterfaceCopy Anchor Link

interface AIScribeSidebarProps {
  isOpen: boolean
  appointmentId: string        // Required - links to appointment
  countryCode?: string         // Optional - for provider selection
  onSuggestionAccept?: (suggestion: AISuggestion) => void
  onSuggestionReject?: (suggestionId: string) => void
  onTranscriptionUpdate?: (transcription: string) => void
  onStatusChange?: (status: AIScribeStatus) => void
  className?: string
}

State ManagementCopy Anchor Link

• Zustand Store: aiScribeStore manages session, recording, and transcription state
• Custom Hooks:
  • useTranscription(appointmentId, countryCode) - Main transcription hook
  • useAiSuggestions(appointmentId) - Suggestion management
  • useAiScribeSettings() - Clinic settings

AI Suggestion TypesCopy Anchor Link

The backend supports 10 suggestion types:

Integration PointsCopy Anchor Link

With Encounter TabsCopy Anchor Link

• Observations Tab: Shows AI Scribe active status
• Auto-Save: Transcription triggers auto-save functionality
• Field Population: Accepted suggestions update relevant fields

With Care Plan WorkflowCopy Anchor Link

• Medication Integration: Ready for Phase 5 medication workflow
• Investigation Integration: Ready for Phase 6 investigations workflow
• Clinical Assessment: Directly updates observation notes

Technical FeaturesCopy Anchor Link

Performance OptimizationsCopy Anchor Link

• Lazy Loading: Components load only when needed
• Memory Management: Efficient state management for long recordings
• Auto-Cleanup: Automatic resource cleanup on component unmount

UI/UX EnhancementsCopy Anchor Link

• Responsive Design: Works on various screen sizes
• Visual Feedback: Clear indicators for all states and actions
• Keyboard Shortcuts: Space bar to start/stop recording (planned)
• Accessibility: Screen reader compatible, keyboard navigation

Data HandlingCopy Anchor Link

• Backend Persistence: Transcription segments stored in database via backend API
• Session Management: Sessions linked to appointments with full audit trail
• HIPAA Compliance: Audio not stored, only text transcripts
• Token Security: Temporary tokens issued by backend (configurable TTL 1-3600s)
• Batch Upload: Segments uploaded every 3 seconds for efficient API usage

Testing ChecklistCopy Anchor Link

• Open AI Scribe from encounter page
• Start recording and observe live duration counter
• Watch voice activity detection indicator
• See transcription segments appear with speaker labels
• Wait for auto-generated AI suggestions
• Accept an AI suggestion and verify it updates the encounter
• Reject an AI suggestion and verify it's marked as rejected
• Manually generate more suggestions using the wand icon
• Pause and resume recording
• Clear transcription and start fresh
• Test with different chief complaints to see scenario adaptation

Development NotesCopy Anchor Link

File Locations (Feature Folder Structure)Copy Anchor Link

apps/cms/src/features/aiScribe/
├── components/
│   └── AIScribeSidebar.tsx     # Main sidebar component
├── hooks/
│   ├── useTranscription.ts     # Transcription + audio capture
│   ├── useAiSuggestions.ts     # Suggestion management
│   ├── useAiScribeSettings.ts  # Clinic settings
│   └── useAudioCapture.ts      # Audio capture utilities
├── services/
│   ├── api/                    # Backend API services
│   │   ├── transcriptionSessionService.ts
│   │   ├── aiSuggestionService.ts
│   │   ├── clinicSettingsService.ts
│   │   └── transcriptionTokenService.ts
│   └── transcription/          # Transcription providers
│       ├── TranscriptionProvider.interface.ts
│       ├── TranscriptionProviderFactory.ts
│       └── providers/
│           └── DeepgramAdapter.ts
├── store/
│   └── aiScribeStore.ts        # Zustand state management
├── types/
│   └── index.ts                # TypeScript types
└── utils/
    └── errorHandler.ts         # Error handling utilities

Parent Component IntegrationCopy Anchor Link

• PrescriptionPage: apps/cms/src/features/prescription/pages/PrescriptionPage.tsx (main clinical documentation)
• PatientSidebar: apps/cms/src/features/prescription/components/PatientSidebar.tsx

Note: Encounters navigate directly to PrescriptionPage for clinical documentation. AI Scribe integrates with the prescription workflow.

Backend DocumentationCopy Anchor Link

• Complete implementation: backend/docs/04-features/ai-scribe/AI_Scribe_Complete_Implementation.md
• API reference: backend/docs/04-features/ai-scribe/API_Reference.md

Future Enhancements (Planned)Copy Anchor Link

• Settings Modal: UI for clinic-level AI Scribe configuration
• Voice Commands: "Add medication", "Order test" voice commands
• Export Functionality: Export transcripts to different formats
• Offline Mode: Local speech processing capability
• Additional Providers: AssemblyAI, Google STT, Azure integration
• Real-Time Collaboration: Pusher-based multi-user transcript viewing

TroubleshootingCopy Anchor Link

Common IssuesCopy Anchor Link

1. No Transcription

   • Ensure microphone permission is granted
   • Check that recording is started and not paused
   • Verify Deepgram connection status (connected indicator)
   • Check browser console for WebSocket errors

2. Connection Failed

   • Backend token service may be unavailable
   • Check network connectivity
   • Verify backend AI Scribe endpoints are accessible
   • Token may have expired - retry will request new token

3. No AI Suggestions

   • Wait for sufficient transcript content (minimum context needed)
   • Click wand icon to manually trigger generation
   • Check that AI suggestions are enabled in clinic settings
   • Verify backend AI provider configuration

4. Speaker Detection Not Working

   • Speaker diarization requires speakerDetection option enabled
   • Works best with clear audio and distinct speakers
   • May not work well with overlapping speech

5. Performance Issues

   • Long recording sessions may accumulate many segments
   • Use pause during silent periods to reduce data
   • Clear transcription to start fresh if needed

Error MessagesCopy Anchor Link

Browser CompatibilityCopy Anchor Link

• Recommended: Chrome 90+, Firefox 88+, Safari 14+, Edge 90+
• Required APIs: MediaDevices, WebSocket, Web Audio API
• Mobile browsers supported but optimized for desktop

Related DocsCopy Anchor Link

• Feature access & gating (developer) — facility/practitioner gating, resolver API, override scope.
• Product access model — bundled vs. self-serve, what admins control.
• QA guide — enablement recipes and regression checklist.

Last Updated: December 2025