Kulcare Docs
General08 troubleshooting

Troubleshooting Guide

This directory contains comprehensive troubleshooting guides for common issues encountered in the Kulcare API backend system.

Quick Reference

Emergency Issues

Common Problems

Development Issues

System Health Monitoring

Health Check Endpoints

The system provides several health check endpoints for monitoring:

# Main application health check
GET /health

# Database connectivity check  
GET /health/database

# External services health check
GET /health/services

# Background jobs health check
GET /health/jobs

Key Metrics to Monitor

Application Metrics

  • Response time averages
  • Error rates by endpoint
  • Active user sessions
  • Database connection pool usage

Infrastructure Metrics

  • CPU and memory utilization
  • Disk space usage
  • Network connectivity
  • Load balancer status

Business Metrics

  • Appointment creation rates
  • SMS/WhatsApp delivery rates
  • Payment processing success
  • User registration trends

Diagnostic Tools

Built-in Debugging Tools

Rails Console Commands

# Check system status
User.connection.active?                    # Database connectivity
Sidekiq::Stats.new.processed              # Background job stats
MsgLog.where('created_at > ?', 1.hour.ago).count  # Recent message count

# Check service integrations
begin
  HTTParty.get('https://api.msg91.com/api/v5/flow/status')
rescue => e
  puts "MSG91 API Error: #{e.message}"
end

Log Analysis

# Check recent errors
tail -f log/production.log | grep ERROR

# Monitor API requests
tail -f log/production.log | grep "Started GET\|Started POST"

# Check background job errors
tail -f log/sidekiq.log | grep ERROR

Performance Monitoring

Database Query Analysis

# Slow query identification
ActiveRecord::Base.logger.level = 0  # Enable query logging

# Find N+1 queries
gem 'bullet' # Add to Gemfile for N+1 detection

Memory Usage Monitoring

# Check memory usage
ps aux | grep puma
free -m

# Monitor memory leaks
cat /proc/$(pgrep puma)/status | grep VmRSS

Common Error Patterns

Authentication Errors

JWT Token Issues

# Symptoms
"JWT::DecodeError: Invalid token"
"Token expired"

# Diagnosis
decoded_token = TokenIssuer.decode_token(token)
puts "Token expires at: #{Time.at(decoded_token['exp'])}" if decoded_token

# Solutions
1. Check token expiration settings
2. Verify secret key configuration
3. Ensure proper token refresh logic

Session Management Problems

# Symptoms
"User session not found"
"Authentication required"

# Diagnosis
user = User.find_by(id: user_id)
puts "User verification status: #{user&.verified?}"
puts "Active devices: #{user&.devices&.count}"

# Solutions
1. Verify user verification status
2. Check device registration
3. Review session timeout settings

Database Issues

Connection Pool Exhaustion

# Symptoms
"ActiveRecord::ConnectionTimeoutError"
"could not obtain a connection from the pool"

# Diagnosis
ActiveRecord::Base.connection_pool.stat

# Solutions
1. Increase pool size in database.yml
2. Check for long-running queries
3. Review connection leak issues

Migration Problems

# Symptoms
"PG::DuplicateColumn: ERROR: column already exists"
"PG::UndefinedTable: ERROR: relation does not exist"

# Diagnosis
ActiveRecord::Base.connection.tables
ActiveRecord::SchemaMigration.all

# Solutions
1. Check migration status: rake db:migrate:status
2. Run pending migrations: rake db:migrate
3. Review migration files for conflicts

Background Job Failures

Sidekiq Job Failures

# Symptoms
Jobs stuck in retry queue
High memory usage in Sidekiq processes

# Diagnosis
require 'sidekiq/api'
stats = Sidekiq::Stats.new
puts "Failed jobs: #{stats.failed}"
puts "Retry jobs: #{stats.retry_size}"

# Solutions
1. Check job arguments and dependencies
2. Review memory usage and job complexity
3. Implement proper error handling

External Service Issues

SMS Delivery Failures

# Symptoms
"SMS not delivered"
"MSG91 API timeout"

# Diagnosis
recent_sms = MsgLog.where('created_at > ?', 1.hour.ago)
puts "Failed SMS: #{recent_sms.where(msg_status: 'FAILED').count}"

# Solutions
1. Check API credentials and quota
2. Verify phone number format
3. Review DLT template compliance

Emergency Response Procedures

System Outage Response

Immediate Actions (First 15 minutes)

  1. Assess Impact: Determine scope of outage
  2. Check Dependencies: Verify database, Redis, external services
  3. Review Logs: Identify error patterns and root cause
  4. Implement Quick Fixes: Apply immediate workarounds

Short-term Recovery (15-60 minutes)

  1. Scale Resources: Add server capacity if needed
  2. Database Recovery: Restore from backup if corrupted
  3. Service Restart: Restart application services
  4. Monitor Recovery: Track system restoration

Post-Incident (After resolution)

  1. Root Cause Analysis: Detailed investigation
  2. Documentation: Update troubleshooting guides
  3. Prevention: Implement preventive measures
  4. Communication: Notify stakeholders of resolution

Data Recovery Procedures

Database Backup Restoration

# Stop application servers
sudo systemctl stop puma
sudo systemctl stop sidekiq

# Restore database from backup
pg_restore --clean --no-acl --no-owner -h localhost -U kulcare -d kulcare_production backup_file.dump

# Restart services
sudo systemctl start puma
sudo systemctl start sidekiq

File Recovery

# Restore uploaded files from S3 backup
aws s3 sync s3://kulcare-backups/files/ /app/public/uploads/

# Fix file permissions
chown -R app:app /app/public/uploads/
chmod -R 644 /app/public/uploads/

Preventive Measures

Monitoring Setup

Application Monitoring

  • Set up Sentry for error tracking
  • Configure New Relic for performance monitoring
  • Implement custom health check endpoints
  • Set up log aggregation (ELK stack)

Infrastructure Monitoring

  • Monitor server resources (CPU, RAM, disk)
  • Track database performance metrics
  • Monitor external service response times
  • Set up uptime monitoring (Pingdom, UptimeRobot)

Backup Strategies

Database Backups

# Daily database backup cron job
0 2 * * * pg_dump -U kulcare kulcare_production | gzip > /backups/kulcare_$(date +\%Y\%m\%d).sql.gz

File Backups

# Weekly file backup to S3
0 1 * * 0 aws s3 sync /app/public/uploads/ s3://kulcare-backups/files/

Code Quality Measures

Testing Coverage

  • Maintain >80% test coverage
  • Include integration tests for critical paths
  • Test error scenarios and edge cases
  • Regular load testing

Code Review Process

  • Mandatory peer review for all changes
  • Security review for authentication/payment code
  • Performance review for database-heavy features
  • Documentation updates with code changes

Escalation Procedures

Issue Severity Levels

P0 - Critical (Complete Outage)

  • Response Time: Immediate (< 5 minutes)
  • Examples: System down, data loss, security breach
  • Escalation: Notify all on-call team members immediately

P1 - High (Significant Impact)

  • Response Time: < 30 minutes
  • Examples: Feature down, slow performance, payment issues
  • Escalation: Notify primary on-call, CC team lead

P2 - Medium (Limited Impact)

  • Response Time: < 2 hours
  • Examples: Minor feature issues, integration warnings
  • Escalation: Create ticket, notify during business hours

P3 - Low (Minimal Impact)

  • Response Time: Next business day
  • Examples: Cosmetic issues, enhancement requests
  • Escalation: Regular ticket queue

Contact Information

On-Call Rotation

Primary: dev-oncall@kulcare.com
Secondary: devops-oncall@kulcare.com
Escalation: engineering-leads@kulcare.com

External Vendors

AWS Support: [Account-specific support cases]
Database Specialist: [External DBA contact]
Security Team: security@kulcare.com

Documentation Standards

Incident Documentation

Each incident should be documented with:

  • Timeline: Detailed sequence of events
  • Root Cause: Technical analysis of failure
  • Impact Assessment: Business and user impact
  • Resolution Steps: Detailed recovery actions
  • Prevention Plan: Measures to prevent recurrence

Runbook Updates

  • Update troubleshooting guides after each incident
  • Include new error patterns and solutions
  • Review and test recovery procedures quarterly
  • Keep contact information current

Integration Failures Troubleshooting

Previous Page

Parts System - Deep Dive

Next Page

On this page

Quick ReferenceEmergency IssuesCommon ProblemsDevelopment IssuesSystem Health MonitoringHealth Check EndpointsKey Metrics to MonitorApplication MetricsInfrastructure MetricsBusiness MetricsDiagnostic ToolsBuilt-in Debugging ToolsRails Console CommandsLog AnalysisPerformance MonitoringDatabase Query AnalysisMemory Usage MonitoringCommon Error PatternsAuthentication ErrorsJWT Token IssuesSession Management ProblemsDatabase IssuesConnection Pool ExhaustionMigration ProblemsBackground Job FailuresSidekiq Job FailuresExternal Service IssuesSMS Delivery FailuresEmergency Response ProceduresSystem Outage ResponseImmediate Actions (First 15 minutes)Short-term Recovery (15-60 minutes)Post-Incident (After resolution)Data Recovery ProceduresDatabase Backup RestorationFile RecoveryPreventive MeasuresMonitoring SetupApplication MonitoringInfrastructure MonitoringBackup StrategiesDatabase BackupsFile BackupsCode Quality MeasuresTesting CoverageCode Review ProcessEscalation ProceduresIssue Severity LevelsP0 - Critical (Complete Outage)P1 - High (Significant Impact)P2 - Medium (Limited Impact)P3 - Low (Minimal Impact)Contact InformationOn-Call RotationExternal VendorsDocumentation StandardsIncident DocumentationRunbook UpdatesRelated Documentation