AI Scribe Frontend Integration Plan

OverviewCopy Anchor Link

Integrate the AI Scribe frontend with backend APIs and transcription service(s), with support for geography-based service switching (India, UAE, USA). All new code will be namespaced in the aiScribe feature folder.

Implementation StatusCopy Anchor Link

✅ Completed (Phases 1-5.2)Copy Anchor Link

Phase 1: Architecture & Infrastructure SetupCopy Anchor Link

• ✅ Feature folder structure created
• ✅ Dependencies installed (@deepgram/sdk)
• ✅ TranscriptionProvider interface created

Phase 2: Backend API Integration LayerCopy Anchor Link

• ✅ TranscriptionSessionService (5 methods)
• ✅ AISuggestionService (6 methods)
• ✅ ClinicSettingsService (2 methods)
• ✅ TranscriptionTokenService (3 methods + helpers)

Phase 3: Transcription Service IntegrationCopy Anchor Link

• ✅ DeepgramAdapter fully implemented
• ✅ TranscriptionProviderFactory with geography routing
• ✅ useAudioCapture hook (reuses teleconsultation infrastructure)

Phase 4: State ManagementCopy Anchor Link

• ✅ aiScribeStore with 20+ actions
• ✅ useTranscription hook (main integration hook)
• ✅ useAiSuggestions hook
• ✅ useAiScribeSettings hook

Phase 5: Component UpdatesCopy Anchor Link

• ✅ 5.1: AIScribeSidebar moved to feature folder
• ✅ 5.2: Replace Mock Data with Real Integration (December 2025)
  • ✅ Removed simulateTranscription() function
  • ✅ Removed mock AI suggestion generation
  • ✅ Added real hooks (useTranscription, useAiSuggestions)
  • ✅ Updated all recording handlers (start, stop, pause, resume)
  • ✅ Added batch segment upload effect (3-second intervals)
  • ✅ Updated suggestion handlers (accept/reject with API calls)
  • ✅ Added loading states and error handling
  • ✅ Added connection state monitoring (disconnected, connecting, connected, reconnecting, error)
  • ✅ Updated interface to require appointmentId prop
• ✅ Updated Parent Component Imports (December 2025)
  • ✅ PatientSidebar.tsx - passes appointmentId, uses AISuggestion type
  • ✅ EncounterPage.tsx - passes appointmentId, updated suggestion handlers
  • ✅ EncounterEditPage.tsx - passes appointmentId, updated suggestion handlers

⏳ Pending WorkCopy Anchor Link

Phase 5: Component Updates (Remaining)Copy Anchor Link

• ❌ 5.3: Create Settings Modal
  • Create AIScribeSettingsModal.tsx component
  • Master enable/disable toggle
  • Language selection (13 languages)
  • Feature toggles (medical terminology, speaker detection, AI suggestions)
  • AI provider/model selection
  • Confidence threshold slider
  • Add settings button to AIScribeSidebar header

Phase 6: Error Handling & User FeedbackCopy Anchor Link

• ❌ 6.1: Error Handler Utility

  • Create utils/errorHandler.ts
  • Error categorization
  • Toast notifications
  • Retry mechanisms

• ❌ 6.2: Loading States

  • Implement loading UI in components

Phase 7: Advanced Features (Optional)Copy Anchor Link

• ❌ Transcript history view
• ❌ Real-time collaboration (Pusher)
• ❌ Suggestion modification UI
• ❌ Confidence score filtering

Phase 8: Testing & Quality AssuranceCopy Anchor Link

• ❌ Unit tests
• ❌ Integration tests
• ❌ Manual testing checklist

Priority Order for Remaining WorkCopy Anchor Link

1. Update AIScribeSidebar Component (Phase 5.2) - ✅ Complete
2. Find and Update Parent Component Imports - ✅ Complete
3. Create Error Handler Utility (Phase 6.1) - Essential for UX
4. Create Settings Modal (Phase 5.3) - Important for customization
5. Manual Testing - Validate end-to-end functionality

Phase 1: Architecture & Infrastructure SetupCopy Anchor Link

1.1 Create Feature Folder StructureCopy Anchor Link

frontend/apps/cms/src/features/aiScribe/
├── components/          # AI Scribe specific components
├── hooks/              # Custom hooks for AI Scribe
├── services/           # API services and transcription providers
│   ├── api/           # Backend API services
│   └── transcription/ # Transcription service adapters
├── store/             # Zustand store for AI Scribe state
├── types/             # TypeScript types and interfaces
└── utils/             # Utility functions

1.2 Install DependenciesCopy Anchor Link

• Add transcription SDK(s) based on geography requirements (Deepgram for now, extensible for others)
• No other new dependencies needed (reusing existing audio infrastructure)

1.3 Create Transcription Provider AbstractionCopy Anchor Link

File: frontend/apps/cms/src/features/aiScribe/services/transcription/TranscriptionProvider.interface.ts

Create provider-agnostic interface supporting:

• Multiple providers (Deepgram, AssemblyAI, Google STT, Azure, etc.)
• Geography-based routing (India → Provider A, UAE → Provider B, USA → Provider C)
• Common interface: connect(), disconnect(), send(), onTranscript(), onError()
• Provider selection based on clinic country/region settings

Phase 2: Backend API Integration LayerCopy Anchor Link

2.1 Transcription Session ServiceCopy Anchor Link

File: frontend/apps/cms/src/features/aiScribe/services/api/transcriptionSessionService.ts

Methods:

• createSession(appointmentId, options) → POST /api/v1/appointments/:id/transcription_sessions
• stopSession(sessionId) → PATCH /api/v1/transcription_sessions/:id/stop
• addSegments(sessionId, segments[]) → POST /api/v1/transcription_sessions/:id/add_segment
• getTranscript(sessionId) → GET /api/v1/transcription_sessions/:id/transcript
• getSession(sessionId) → GET /api/v1/transcription_sessions/:id

2.2 AI Suggestion ServiceCopy Anchor Link

File: frontend/apps/cms/src/features/aiScribe/services/api/aiSuggestionService.ts

Methods:

• generateSuggestions(appointmentId, options) → POST /api/v1/appointments/:id/generate_ai_suggestions
• listSuggestions(appointmentId) → GET /api/v1/appointments/:id/ai_suggestions
• acceptSuggestion(suggestionId, applyToCareplan) → PATCH /api/v1/ai_suggestions/:id/accept
• rejectSuggestion(suggestionId) → PATCH /api/v1/ai_suggestions/:id/reject
• modifySuggestion(suggestionId, content) → PATCH /api/v1/ai_suggestions/:id

2.3 Clinic Settings ServiceCopy Anchor Link

File: frontend/apps/cms/src/features/aiScribe/services/api/clinicSettingsService.ts

Methods:

• getSettings(clinicId) → GET /api/v1/clinic_ai_scribe_settings
• saveSettings(clinicId, settings) → POST /api/v1/clinic_ai_scribe_settings

2.4 Transcription Token ServiceCopy Anchor Link

File: frontend/apps/cms/src/features/aiScribe/services/api/transcriptionTokenService.ts

Methods:

• issueToken(sessionId, provider) → POST /api/v1/ai_scribe/issue_token
• Handle token refresh logic before expiration
• Support multiple provider token formats

Phase 3: Transcription Service IntegrationCopy Anchor Link

3.1 Provider Adapter PatternCopy Anchor Link

Files:

• frontend/apps/cms/src/features/aiScribe/services/transcription/providers/DeepgramAdapter.ts
• frontend/apps/cms/src/features/aiScribe/services/transcription/providers/AssemblyAIAdapter.ts (placeholder)
• frontend/apps/cms/src/features/aiScribe/services/transcription/providers/GoogleSTTAdapter.ts (placeholder)

Each adapter implements common interface:

interface TranscriptionAdapter {
  connect(token: string, options: TranscriptionOptions): Promise<void>
  disconnect(): Promise<void>
  send(audioData: ArrayBuffer): void
  pause(): void
  resume(): void
  onSegment(callback: (segment: TranscriptionSegment) => void): void
  onError(callback: (error: Error) => void): void
  getConnectionState(): ConnectionState
}

3.2 Provider FactoryCopy Anchor Link

File: frontend/apps/cms/src/features/aiScribe/services/transcription/TranscriptionProviderFactory.ts

Features:

• Select provider based on clinic geography (country_code)
• Fallback provider logic if primary fails
• Configuration mapping: {IN: 'deepgram', AE: 'assemblyai', US: 'deepgram'}
• Extensible for future provider additions

3.3 Audio Capture IntegrationCopy Anchor Link

File: frontend/apps/cms/src/features/aiScribe/hooks/useAudioCapture.ts

Reuses existing teleconsultation code:

• Permission request flow from PreCallSetupEnhanced.tsx
• Device selection using useVideoProvider.switchMicrophone()
• Stream cleanup via mediaStreamCleanup
• Audio constraints (echo cancellation, noise suppression)

New additions:

• Audio level visualization using Web Audio API
• Chunking audio for streaming to transcription service
• Recording state management (recording, paused, stopped)

Phase 4: State ManagementCopy Anchor Link

4.1 AI Scribe StoreCopy Anchor Link

File: frontend/apps/cms/src/features/aiScribe/store/aiScribeStore.ts

State:

{
  // Session state
  session: TranscriptionSession | null
  sessionStatus: 'idle' | 'creating' | 'active' | 'paused' | 'stopping' | 'stopped'

  // Recording state
  isRecording: boolean
  isPaused: boolean
  duration: number
  audioLevel: number

  // Transcription state
  segments: TranscriptionSegment[]
  fullTranscript: string

  // AI Suggestions state
  suggestions: AISuggestion[]
  isGeneratingSuggestions: boolean

  // Settings
  settings: ClinicAiScribeSetting | null

  // Error handling
  error: Error | null

  // Provider state
  provider: TranscriptionAdapter | null
  connectionState: ConnectionState
}

Actions:

• Session lifecycle: startSession(), stopSession(), pauseSession(), resumeSession()
• Segment management: addSegment(), batchUploadSegments()
• Suggestion management: generateSuggestions(), acceptSuggestion(), rejectSuggestion()
• Settings: loadSettings(), updateSettings()

4.2 Custom HooksCopy Anchor Link

Files:

• frontend/apps/cms/src/features/aiScribe/hooks/useTranscription.ts - Main hook combining audio capture + transcription
• frontend/apps/cms/src/features/aiScribe/hooks/useAiSuggestions.ts - Suggestion management
• frontend/apps/cms/src/features/aiScribe/hooks/useAiScribeSettings.ts - Settings with caching

Phase 5: Component UpdatesCopy Anchor Link

5.1 Move AIScribeSidebar to Feature FolderCopy Anchor Link

From: frontend/apps/cms/src/components/care-plan/AIScribeSidebar.tsx To: frontend/apps/cms/src/features/aiScribe/components/AIScribeSidebar.tsx

Update import paths in parent components.

5.2 Replace Mock Data with Real IntegrationCopy Anchor Link

Changes to AIScribeSidebar.tsx:

1. Remove simulateTranscription() function
2. Remove mock AI suggestion generation
3. Add hooks:
   • useTranscription() for recording and real-time transcription
   • useAiSuggestions() for suggestion management
   • useAiScribeSettings() for clinic settings
4. Update recording handlers:
   • handleStartRecording() - Create session + connect to transcription provider
   • handleStopRecording() - Disconnect + stop session + generate suggestions
   • handlePauseRecording() - Pause audio stream + pause transcription
5. Add batch segment upload effect (every 3-5 seconds)
6. Update suggestion handlers to call real APIs
7. Add loading states and error handling
8. Add user confirmation modal for "Apply to Careplan"

5.3 Create Settings ModalCopy Anchor Link

File: frontend/apps/cms/src/features/aiScribe/components/AIScribeSettingsModal.tsx

UI Features:

• Master enable/disable toggle
• Language selection (13 supported languages)
• Medical terminology toggle
• Speaker detection toggle
• AI suggestions toggle
• AI provider selection (OpenAI, Anthropic, Google, Mock)
• AI model selection (conditional on provider)
• Confidence threshold slider (0.0-1.0)
• Save/Cancel actions

Add settings button to AIScribeSidebar header.

Phase 6: Error Handling & User FeedbackCopy Anchor Link

6.1 Error Types & HandlingCopy Anchor Link

File: frontend/apps/cms/src/features/aiScribe/utils/errorHandler.ts

Error Categories:

• Microphone permission denied
• Transcription service connection failed
• Network errors during API calls
• Session creation/stop failures
• Token expiration
• Provider-specific errors

User Feedback:

• Toast notifications for errors
• Inline error messages in sidebar
• Retry mechanisms with exponential backoff
• Fallback to alternative provider if available

6.2 Loading StatesCopy Anchor Link

• Creating session spinner
• Connecting to transcription service indicator
• Generating suggestions loader
• Accepting/rejecting suggestion feedback
• Settings save feedback

Phase 7: Advanced Features (Optional)Copy Anchor Link

7.1 Transcript History ViewCopy Anchor Link

• List previous transcription sessions for appointment
• View past transcripts with metadata
• Display session duration, language, created by

7.2 Real-Time Collaboration (Pusher)Copy Anchor Link

• Subscribe to appointment-specific channels
• Show when other clinicians viewing same transcript
• Real-time suggestion updates across views

7.3 Suggestion Modification UICopy Anchor Link

• Edit suggestion content before accepting
• Track modification history
• Show "Modified" badge

7.4 Confidence Score FilteringCopy Anchor Link

• Filter suggestions below threshold
• Color-coded confidence scores
• Manual threshold adjustment in UI

Phase 8: Testing & Quality AssuranceCopy Anchor Link

8.1 Unit TestsCopy Anchor Link

• Test each API service with mocked responses
• Test custom hooks with React Testing Library
• Test store actions and state updates
• Test provider adapters with mock WebSocket

8.2 Integration TestsCopy Anchor Link

• Test full recording workflow
• Test suggestion generation and acceptance
• Test settings persistence
• Test error scenarios and recovery

8.3 Manual Testing ChecklistCopy Anchor Link

• Microphone access in different browsers (Chrome, Firefox, Safari)
• Audio level visualization accuracy
• Pause/resume maintains state
• Provider connection recovery on failure
• Token refresh before expiration
• Segment batch upload timing
• Transcript persistence after stop
• AI suggestions match transcript
• Accept/reject updates UI correctly
• Careplan updates on suggestion acceptance
• Settings persist across sessions
• All 10 suggestion types render correctly
• Error messages are user-friendly
• Loading states provide clear feedback
• Geography-based provider selection works
• Provider fallback on failure

Implementation OrderCopy Anchor Link

Critical Path:

1. Phase 1 - Architecture setup (folder structure, interfaces)
2. Phase 2 - Backend API services
3. Phase 3.1 - First provider adapter (Deepgram)
4. Phase 3.2 - Provider factory with geography routing
5. Phase 3.3 - Audio capture (reusing teleconsultation code)
6. Phase 4 - State management (store + hooks)
7. Phase 5.1-5.2 - Component updates (replace mocks)
8. Phase 6 - Error handling & feedback
9. Phase 5.3 - Settings modal
10. Phase 8 - Testing

Can be done in parallel:

• Phase 2 API services (all independent)
• Phase 3.1 provider adapters (if adding multiple)
• Phase 5.3 settings modal (after Phase 2.3)
• Phase 7 optional features (after core complete)

Key Technical DecisionsCopy Anchor Link

1. Geography-Based Provider SelectionCopy Anchor Link

• Clinic settings include country_code field
• Provider factory maps country to preferred service:

  Copy Text

  const PROVIDER_MAP = {
    IN: 'deepgram',      // India
    AE: 'assemblyai',    // UAE (placeholder)
    US: 'deepgram',      // USA
    // ... extensible
  }

• Fallback cascade if primary provider fails

2. Reuse Teleconsultation Audio InfrastructureCopy Anchor Link

• Permission flow from PreCallSetupEnhanced.tsx
• Device management from useVideoProvider
• Stream cleanup from mediaStreamCleanup
• Audio constraints (echo cancellation, noise suppression)
• No need to rebuild microphone access

3. Provider Adapter PatternCopy Anchor Link

• All transcription services implement common interface
• Easy to add new providers without changing consumer code
• Provider-specific configuration isolated in adapters
• Supports different auth mechanisms (JWT, API key, OAuth)

4. Feature Folder OrganizationCopy Anchor Link

• All AI Scribe code in features/aiScribe/
• Clear separation from other features
• Easier to maintain and extend
• Better code discoverability

5. Token SecurityCopy Anchor Link

• Master API keys never exposed to frontend
• Temporary tokens with short TTL (configurable 1-3600s)
• Token refresh logic before expiration
• Audit trail on backend for all token issuance

Existing Teleconsultation Audio InfrastructureCopy Anchor Link

Available for ReuseCopy Anchor Link

1. Microphone Access & PermissionsCopy Anchor Link

File: frontend/apps/cms/src/components/teleconsultation/ui/PreCallSetupEnhanced.tsx

Features:

• Permission request using navigator.mediaDevices.getUserMedia()
• Device enumeration with navigator.mediaDevices.enumerateDevices()
• Audio constraints with echo cancellation and noise suppression
• Comprehensive error handling for permission errors
• MediaStream lifecycle management

2. Device ManagementCopy Anchor Link

File: frontend/apps/cms/src/hooks/useVideoProvider.ts

Features:

• switchMicrophone(deviceId) for device selection
• getDevices() returns all available MediaDeviceInfo[]
• Audio/video toggle functionality

3. Stream CleanupCopy Anchor Link

File: frontend/apps/cms/src/utils/mediaStreamCleanup.ts

Features:

• Singleton manager for MediaStream tracking
• Prevents camera icon remaining in URL bar
• Global cleanup on route changes and page unload
• Methods: registerStream(), stopStream(), stopAllStreams()

4. Audio Level MonitoringCopy Anchor Link

File: frontend/apps/cms/src/services/video/core/VideoProvider.interface.ts

Features:

• Performance metrics include audio.inputLevel and audio.outputLevel
• Audio codec information tracking
• Echo cancellation and noise suppression status

What Needs to Be BuiltCopy Anchor Link

• Audio level visualization UI (waveform/bars)
• Audio recording to blob/file for transcription
• Audio chunking for streaming APIs
• Integration with transcription service WebSocket
• Local state management for AI Scribe

Files to CreateCopy Anchor Link

API Services (4 files):

1. frontend/apps/cms/src/features/aiScribe/services/api/transcriptionSessionService.ts
2. frontend/apps/cms/src/features/aiScribe/services/api/aiSuggestionService.ts
3. frontend/apps/cms/src/features/aiScribe/services/api/clinicSettingsService.ts
4. frontend/apps/cms/src/features/aiScribe/services/api/transcriptionTokenService.ts

Transcription Services (4 files): 5. frontend/apps/cms/src/features/aiScribe/services/transcription/TranscriptionProvider.interface.ts 6. frontend/apps/cms/src/features/aiScribe/services/transcription/TranscriptionProviderFactory.ts 7. frontend/apps/cms/src/features/aiScribe/services/transcription/providers/DeepgramAdapter.ts 8. frontend/apps/cms/src/features/aiScribe/services/transcription/providers/index.ts (exports)

State Management (1 file): 9. frontend/apps/cms/src/features/aiScribe/store/aiScribeStore.ts

Custom Hooks (4 files): 10. frontend/apps/cms/src/features/aiScribe/hooks/useAudioCapture.ts 11. frontend/apps/cms/src/features/aiScribe/hooks/useTranscription.ts 12. frontend/apps/cms/src/features/aiScribe/hooks/useAiSuggestions.ts 13. frontend/apps/cms/src/features/aiScribe/hooks/useAiScribeSettings.ts

Components (2 files): 14. frontend/apps/cms/src/features/aiScribe/components/AIScribeSettingsModal.tsx 15. Move & update frontend/apps/cms/src/features/aiScribe/components/AIScribeSidebar.tsx

Types (1 file): 16. frontend/apps/cms/src/features/aiScribe/types/index.ts

Utils (1 file): 17. frontend/apps/cms/src/features/aiScribe/utils/errorHandler.ts

Total: 17 new files + 1 moved file

Files to ModifyCopy Anchor Link

1. frontend/apps/cms/package.json - Add transcription SDK dependency
2. Parent components importing AIScribeSidebar - Update import path
3. Any existing type definitions that need extension

Backend API ReferenceCopy Anchor Link

Transcription Session EndpointsCopy Anchor Link

POST   /api/v1/appointments/:id/transcription_sessions
GET    /api/v1/transcription_sessions/:id
PATCH  /api/v1/transcription_sessions/:id/stop
POST   /api/v1/transcription_sessions/:id/add_segment
GET    /api/v1/transcription_sessions/:id/transcript

AI Suggestion EndpointsCopy Anchor Link

GET    /api/v1/appointments/:id/ai_suggestions
POST   /api/v1/appointments/:id/generate_ai_suggestions
GET    /api/v1/ai_suggestions/:id
PATCH  /api/v1/ai_suggestions/:id
PATCH  /api/v1/ai_suggestions/:id/accept
PATCH  /api/v1/ai_suggestions/:id/reject

Settings EndpointsCopy Anchor Link

GET    /api/v1/clinic_ai_scribe_settings
POST   /api/v1/clinic_ai_scribe_settings

Token EndpointCopy Anchor Link

POST   /api/v1/ai_scribe/issue_token

Data ModelsCopy Anchor Link

TranscriptionSessionCopy Anchor Link

{
  id: string
  appointment_id: string
  session_id: string
  status: 'active' | 'paused' | 'completed' | 'failed'
  language: string
  medical_terminology: boolean
  metadata: {
    clinic_id: string
    doctor_id: string
    patient_id: string
    created_by_user_id: string
  }
  started_at: string
  ended_at: string | null
}

TranscriptionSegmentCopy Anchor Link

{
  id: string
  transcription_session_id: string
  content: string
  confidence_score: number  // 0.0 - 1.0
  speaker?: string          // "Doctor", "Patient", etc.
  start_time_ms: number
  end_time_ms: number
  sequence_number: number
  metadata?: {
    source: string
    created_by: string
  }
}

AISuggestionCopy Anchor Link

{
  id: string
  appointment_id: string
  transcription_session_id?: string
  suggestion_type: 'vitals' | 'complaint' | 'diagnosis' | 'medication' |
                   'test' | 'treatment_plan' | 'follow_up' | 'note' |
                   'advice' | 'diet'
  content: any  // JSON structure varies by type
  confidence_score?: number
  reasoning: string
  status: 'pending' | 'accepted' | 'rejected' | 'modified'
  processed_at?: string
  processed_by_user_id?: string
}

ClinicAiScribeSettingCopy Anchor Link

{
  id: string
  clinic_id: string
  enabled: boolean
  default_language: string  // 13 supported languages
  medical_terminology_enabled: boolean
  speaker_detection_enabled: boolean
  ai_suggestions_enabled: boolean
  ai_provider: 'openai' | 'anthropic' | 'google' | 'mock'
  ai_model: string
  confidence_threshold: number  // 0.0 - 1.0
  custom_settings?: Record<string, any>
}

Success CriteriaCopy Anchor Link

• ✅ Real-time transcription working with geography-based provider selection
• ✅ Transcription segments persist to backend database
• ✅ AI suggestions generate from actual transcript
• ✅ Accept/reject suggestions update backend and careplan
• ✅ Settings modal allows clinic configuration
• ✅ All error scenarios handled gracefully
• ✅ Microphone permissions and device selection work
• ✅ Audio level visualization displays correctly
• ✅ Pause/resume maintains accurate state
• ✅ Provider can be switched based on geography
• ✅ Code properly organized in feature folder
• ✅ No breaking changes to existing teleconsultation functionality
• ✅ All 10 suggestion types render correctly
• ✅ Token refresh works before expiration
• ✅ Batch segment upload operates on 3-5 second intervals
• ✅ Full transcript persists after session stop
• ✅ Loading states provide clear user feedback
• ✅ Error messages are user-friendly and actionable