Hackathon Project
2nd Place

SeeForMe

AI-powered visual assistance platform that provides real-time environmental navigation, image description, and color blindness tools for visually impaired users using YOLO v11 and LLM integration.

React
Python
Flask
YOLO
Computer Vision
AI
Accessibility
Healthcare
Socket.IO
OpenCV
TailwindCSS
Web Speech API
GROQ
Gemini API
Real-time
Mobile
Duration: 1 month
Completed: 7/13/2024
Team Size: 3

Team Members:

Taiwo Ayomide
Omidire Ifeoluwa
View SourceLive Demo
SeeForMe

Project Overview

An AI-powered visual assistance platform that provides real-time environmental navigation, image description, and color blindness tools for visually impaired users using YOLO v11 and LLM integration.

Key Features

  • Real-time environmental navigation with custom-trained YOLO v11 object detection
  • Instant image description using Gemini API with voice output
  • Color blindness simulation tools for multiple vision deficiencies (Protanopia, Deuteranopia, Tritanopia, Achromatopsia)
  • Voice-controlled navigation and commands via Web Speech API
  • Hands-free keyboard navigation shortcuts for accessibility
  • Live camera streaming with Socket.IO for real-time guidance
  • Context-aware navigation instructions powered by GROQ LLM
  • Cross-platform compatibility (web/mobile) with responsive design
  • Open-source codebase focused on accessibility and inclusivity

Challenges & Solutions

  • AI Model Integration: Successfully combined YOLO v11 object detection with LLM reasoning to provide contextual, actionable navigation instructions rather than simple object identification
  • Real-time Performance: Implemented efficient Socket.IO streaming to handle live camera feeds while maintaining low latency for critical navigation guidance
  • Cross-platform Accessibility: Addressed platform-specific limitations (iOS voice restrictions) while ensuring consistent user experience across devices
  • Custom YOLO Training: Developed specialized object detection model trained on environmental obstacles and navigation-relevant objects for improved accuracy
  • Voice Interface Design: Created intuitive voice command system with clear audio feedback, essential for users who cannot see visual confirmations
  • Accessibility-First UI: Designed interface following WCAG guidelines with high contrast, screen reader compatibility, and keyboard-only navigation support

System Architecture

Architecture Overview

SeeForMe employs a sophisticated multi-tier architecture that seamlessly integrates computer vision, natural language processing, and real-time communication to deliver comprehensive visual assistance.

Frontend Architecture (React + Vite)

The frontend is built using React with Vite for optimal performance and development experience, featuring:

  • Component Structure:

    • Main Dashboard with feature selection
    • Camera Interface for real-time navigation
    • Image Upload/Capture components
    • Color Blindness Simulation tools
    • Voice Command Interface
    • Accessibility Navigation components

  • Key Technologies:

    • TailwindCSS: Utility-first styling with accessibility-focused design
    • Socket.IO Client: Real-time bidirectional communication
    • Web Speech API: Voice recognition and text-to-speech functionality
    • Gemini API Integration: Direct frontend image description processing
    • Responsive Design: Mobile-first approach with cross-device compatibility

  • Accessibility Features:

    • ARIA labels and semantic HTML structure
    • High contrast color schemes
    • Keyboard navigation support
    • Screen reader compatibility
    • Voice command integration

Backend Architecture (Python + Flask)

The backend serves as the AI processing engine, handling complex computer vision and natural language tasks:

  • Core Framework:

    • Flask: Lightweight web framework for API endpoints
    • Flask-SocketIO: Real-time WebSocket communication
    • OpenCV: Computer vision and image processing
    • Ultralytics YOLO v11: Custom-trained object detection model

  • AI Pipeline:

    • YOLO v11 Object Detection: Custom-trained model for environmental object recognition
    • GROQ LLM Integration: Context-aware instruction generation
    • Real-time Processing: Efficient frame processing for live camera feeds
    • Multi-threaded Processing: Concurrent handling of multiple user sessions

Real-Time Communication Flow

Frontend Camera → Socket.IO → Backend Processing → YOLO Detection → LLM Context → Voice Instruction → Frontend Audio Output

AI Model Integration

YOLO v11 Custom Training:

  • Trained on environmental obstacles and navigation-relevant objects
  • Optimized for real-time inference on mobile devices
  • Custom classes for accessibility-specific scenarios

LLM Context Processing:

  • GROQ API integration for natural language instruction generation
  • Context-aware responses based on detected objects and user location
  • Concise, actionable guidance optimized for audio delivery

Gemini API Integration:

  • Frontend-direct image description processing
  • Multi-modal understanding for complex scene analysis
  • Real-time image-to-text conversion with accessibility focus

Data Flow Architecture

  1. Real-time Navigation:

    • Camera stream → Frontend capture → Socket.IO transmission
    • Backend YOLO processing → Object detection → LLM context generation
    • Instruction delivery → Text-to-speech → User audio feedback

  2. Image Description:

    • Image capture/upload → Gemini API processing
    • Description generation → Voice output → User feedback

  3. Color Blindness Simulation:

    • Image processing → Frontend color transformation
    • Multiple deficiency type simulation → Visual comparison display

Security & Performance

  • Secure Communication: HTTPS/WSS protocols for all data transmission
  • Efficient Processing: Optimized YOLO inference for mobile devices
  • Error Handling: Comprehensive fallback mechanisms for AI service failures
  • Privacy-First: No permanent storage of user images or personal data

Deployment Architecture

  • Frontend: Static deployment optimized for CDN delivery
  • Backend: Containerized Flask application with auto-scaling capabilities
  • AI Models: Optimized model serving with GPU acceleration support
  • Real-time Infrastructure: WebSocket connection management with session handling