DeepGIS-XR

Demo clip:

Advanced Geospatial Visualization Platform with AI-Powered Analysis

DeepGIS-XR is a comprehensive geospatial visualization and analysis platform that combines advanced 3D mapping, AI-powered image analysis, and adaptive sampling systems for Earth and lunar exploration.

🌟 Key Features

🤖 AI-Powered Viewport Analysis

Segment Anything Model (SAM): Universal image segmentation with no training required
- Three model sizes: Base (375MB), Large (1.2GB), Huge (2.4GB)
- Automatic region detection and boundary identification
- GeoJSON export with polygon simplification
YOLOv8 Detection: Ultra-fast real-time object detection
- Multiple model sizes: Nano, Small, Medium, Large, XLarge
- 80 COCO object categories
- Class filtering support
Grounding DINO: Open-vocabulary text-based object detection
- Detect ANY object by describing it in natural language
- Text prompts like: "rock . boulder . crater . debris"
- Supports remote API deployment for GPU acceleration
- Ideal for domain-specific detection (geology, archaeology, agriculture)
Zero-Shot Object Detection: Pre-trained COCO model for 80 object categories
- Detects: person, car, bicycle, truck, bus, animals, and more
- Confidence-based filtering
- Class-labeled visualizations
Mask2Former: State-of-the-art instance segmentation
- More accurate than Zero-Shot for complex scenes
- Pre-trained on COCO dataset

🌍 World Sampler - Adaptive Geospatial Sampling

Intelligent Spatial Sampling: Probabilistic framework for location sampling
Adaptive Learning: Updates distribution based on feedback and rewards
Survey Mode: Cycle through sampled points with automatic navigation
Spatial Queries: Efficient region-based queries and statistics
Multiple Initialization Strategies: Uniform, Gaussian, Gaussian mixture, custom

🌙 Moon Viewer

Lunar Visualization: Full Moon globe with LROC QuickMap imagery
Apollo Landing Sites: Historical mission locations
Aviation-Style Navigation: Heading dial, attitude indicator, sun/moon info
LOLA Terrain: High-resolution lunar elevation data
Lunar Digital Twin: Navigational decision support system

🌤️ Weather Stations Integration

NWS Weather Stations: Real-time weather data from National Weather Service API
Multi-State Support: Quick load stations from California, Arizona, Colorado, and Nevada
Interactive Display: Temperature labels, weather icons, and detailed popups
Auto-Update: Automatic refresh every 15 minutes for current conditions
HUD Integration: Weather stations accessible via bottom toolbar layer button
21 Default Stations: Pre-configured stations across four western US states

🗺️ Advanced Geospatial Features

3D Globe Visualization: CesiumJS-powered Earth and Moon globes
3D Buildings Layer: OpenStreetMap buildings with worldwide coverage
- Toggle via View panel checkbox or press B key
- Free and open data source (ODbL license)
- Smart loading: loads once, toggles visibility thereafter
Multi-Layer Support: Raster and vector layer management
Tile Server Integration: Custom tile server for large datasets
3D Model Support: GLB/GLTF model loading and visualization
Coordinate Systems: Support for multiple projections and ellipsolds
Drone Navigation: Fly mode and orbit mode for automated camera movement
Measurement Tools: Distance, area, and height measurement capabilities

🔗 Experience URL Sharing

Shareable URLs: Generate URLs that capture complete camera state
Camera Parameters: Position (lon, lat, alt), orientation (heading, pitch, roll)
View Mode Preservation: Remembers 2D, 3D, or Columbus view mode
Drone State Capture: Includes fly distance, speeds, orbit settings
Active Mode Restoration: Restores takeoff, landing, fly, and orbit modes
QR Code Generation: Share experiences via QR codes
Keyboard Shortcuts: Press S to share current view

🚀 Quick Start

Prerequisites

Docker and Docker Compose
NVIDIA GPU (optional, for AI features)
Python 3.8+ (for local development)

Installation

Clone the repository

git clone https://github.com/Earth-Innovation-Hub/deepgis-xr.git
cd deepgis-xr

Start with Docker Compose
```
docker-compose up -d
```
Access the application
- Main application: http://localhost:8060
- Tile server: http://localhost:8091

Local Development Setup

Install dependencies
```
pip install -r requirements.txt
```
Run migrations
```
python manage.py migrate
```
Start development server
```
python manage.py runserver
```

🏗️ Architecture

Technology Stack

Backend:

Django 4.0+ (Python web framework)
Django REST Framework (API endpoints)
PostgreSQL/SQLite (database)
Celery (async tasks, optional)

Frontend:

CesiumJS (3D globe visualization)
JavaScript ES6+ (modern frontend)
Bootstrap (UI components)

AI/ML:

Segment Anything Model (SAM) - Meta AI
Grounding DINO - Open-vocabulary detection (IDEA Research)
YOLOv8 - Real-time object detection (Ultralytics)
Zero-Shot Detection (Mask R-CNN) - COCO pre-trained
Mask2Former - Instance segmentation
PyTorch (deep learning framework)

Infrastructure:

Docker & Docker Compose (containerization)
Nginx (reverse proxy, optional)
TileServer GL (tile serving)

Project Structure

deepgis-xr/
├── deepgis_xr/              # Django project
│   ├── apps/
│   │   ├── web/             # Web application
│   │   │   ├── world_sampler_api.py  # Sampling & AI APIs
│   │   │   └── views.py     # View handlers
│   │   ├── core/            # Core models
│   │   └── ml/              # ML models
│   └── settings.py          # Django settings
├── staticfiles/             # Static assets
│   └── web/
│       └── js/
│           ├── main.js              # Main application entry
│           ├── world-sampler-ui.js  # World sampler UI logic
│           ├── widgets/
│           │   └── weather-stations.js  # Weather stations widget
│           └── utils/
│               └── nws-weather-stations.js  # NWS API integration
├── docker-compose.yml       # Docker configuration
├── requirements.txt        # Python dependencies
└── README.md              # This file

🔌 API Endpoints

World Sampler API

POST /webclient/sampler/initialize - Initialize new sampler
POST /webclient/sampler/sample - Get sample locations
POST /webclient/sampler/update - Update distribution
GET /webclient/sampler/query - Query spatial region
GET /webclient/sampler/statistics - Get distribution stats
POST /webclient/sampler/reset - Reset sampler
GET /webclient/sampler/history - View sample history

AI Analysis API

POST /webclient/sampler/analyze-viewport - Analyze viewport with AI
- Parameters:
  - model_type: 'sam', 'yolov8', 'grounding_dino', 'zero_shot', or 'mask2former'
  - image: Base64-encoded viewport image
  - location: Camera position metadata
  - SAM options:
    - sam_model: 'vit_b', 'vit_l', or 'vit_h'
    - min_area: Minimum segment area in pixels
  - YOLOv8 options:
    - yolo_model: 'yolov8n', 'yolov8s', 'yolov8m', 'yolov8l', 'yolov8x'
    - confidence_threshold: 0.0-1.0
    - class_filter: Comma-separated class names (e.g., "person,car,truck")
  - Grounding DINO options:
    - text_prompt: Dot-separated object descriptions (e.g., "rock . boulder . crater")
    - box_threshold: Detection confidence threshold (default: 0.3)
    - text_threshold: Text matching threshold (default: 0.25)
  - Zero-Shot/Mask2Former options:
    - confidence_threshold: 0.0-1.0
- Returns: GeoJSON with segments/detections, metadata, saved file paths

Labeling API

POST /label/semi-supervised/api/generate-labels/ - Generate assisted labels
POST /label/semi-supervised/api/save-labels/ - Save labels
GET /label/semi-supervised/api/get-images/ - Get label images

🧠 AI Viewport Analysis Architecture

The AI Viewport Analysis system supports multiple detection models, including remote API deployment for GPU-intensive models like Grounding DINO.

System Flow

┌─────────────────────────────────────────────────────────────────────┐
│                         DeepGIS-XR Frontend                         │
│  ┌──────────────────────────────────────────────────────────────┐  │
│  │  AI Viewport Analysis Panel                                   │  │
│  │  ┌────────────────────────────────────────────────────────┐  │  │
│  │  │ Analysis Type: [Grounding DINO (Open Vocab) ▼]         │  │  │
│  │  │ Text Prompt:   [rock . boulder . crater . debris    ]  │  │  │
│  │  │ Box Threshold: [═══════●═══] 0.30                      │  │  │
│  │  │ Text Threshold:[══════●════] 0.25                      │  │  │
│  │  │ [  🧠 Analyze Viewport  ]                              │  │  │
│  │  └────────────────────────────────────────────────────────┘  │  │
│  └──────────────────────────────────────────────────────────────┘  │
└──────────────────────────────┬──────────────────────────────────────┘
                               │ POST /webclient/sampler/analyze-viewport
                               │ {image, location, model_type, text_prompt, ...}
                               ▼
┌─────────────────────────────────────────────────────────────────────┐
│                    DeepGIS-XR Django Backend                        │
│  world_sampler_api.py::analyze_viewport()                          │
│     ├── model_type == 'sam'           → Local SAM inference        │
│     ├── model_type == 'yolov8'        → Local YOLOv8 inference     │
│     ├── model_type == 'grounding_dino'→ Remote API call ──────┐    │
│     ├── model_type == 'zero_shot'     → Local Mask R-CNN      │    │
│     └── model_type == 'mask2former'   → Local Mask2Former     │    │
└──────────────────────────────┬────────────────────────────────│────┘
                               │                                │
                               ▼                                ▼
┌──────────────────────────────────────┐  ┌───────────────────────────┐
│     Local GPU/CPU Processing         │  │  Remote Grounding DINO    │
│  ┌────────────────────────────────┐  │  │  API Server               │
│  │ • SAM (vit_b, vit_l, vit_h)   │  │  │  ┌─────────────────────┐  │
│  │ • YOLOv8 (n, s, m, l, x)      │  │  │  │ POST /predict       │  │
│  │ • Mask R-CNN (COCO)           │  │  │  │ POST /predict_batch │  │
│  │ • Mask2Former (COCO)          │  │  │  │ GET  /health        │  │
│  └────────────────────────────────┘  │  │  └─────────────────────┘  │
└──────────────────────────────────────┘  └───────────────────────────┘

Remote AI APIs

GPU-accelerated AI services on dedicated server for open-vocabulary detection and segmentation.

Grounding DINO (port 5000): Text-based detection

Grounded-SAM-2 (port 5001): Detection + high-quality segmentation

# Grounding DINO - Detection only
curl -X POST http://192.168.0.232:5000/api/predict \
    -F "file=@image.jpg" -F "text_prompt=rock . boulder . crater"

# Grounded-SAM-2 - Detection + Segmentation
curl -X POST http://192.168.0.232:5001/detect \
    -F "image=@image.jpg" -F "text_prompt=rock . boulder . crater"

# Python client
./grounding_dino_api_client.py --image viewport.jpg --prompt "rock . boulder"

Example Prompts: Geology: "rock . boulder . crater" | Urban: "building . car . tree" | Wildlife: "animal . bird . nest"

🎯 Usage Examples

AI Viewport Analysis

Navigate to DeepGIS Search (/label/3d/search/)
Open AI Viewport Analysis panel (brain icon in HUD)
Select analysis type:
- SAM: Universal segmentation (all regions)
- YOLOv8: Fast real-time detection (80 COCO categories)
- Grounding DINO: Open-vocabulary detection (describe any object)
- Zero-Shot: Pre-trained COCO detection
- Mask2Former: High-accuracy instance segmentation
Configure parameters:
- SAM: Model size (Base/Large/Huge), minimum segment area
- YOLOv8: Model size (Nano to XLarge), confidence, class filter
- Grounding DINO: Text prompt (e.g., "rock . crater . boulder"), thresholds
- Zero-Shot/Mask2Former: Confidence threshold
Click "Analyze Viewport"
View results on map with color-coded polygons and labels

World Sampler

Initialize sampler with desired strategy
Sample locations based on adaptive distribution
Navigate to samples using survey mode
Update distribution based on feedback
Query regions for spatial analysis

Moon Viewer

Navigate to Moon Viewer (/label/3d/moon/)
Explore lunar surface with LROC imagery
View Apollo landing sites and historical locations
Use navigation widgets for precise control
Adjust camera with aviation-style controls

3D Buildings Layer

Navigate to DeepGIS Search (/label/3d/search/)
Enable buildings:
- Click "View" button in HUD toolbar
- Check "3D Buildings (OSM)" checkbox
- Or press B key to toggle instantly
Best viewed in 3D mode (press V to switch to 3D)
Zoom to urban areas to see detailed building models
Coverage: Worldwide, based on OpenStreetMap data quality

Weather Stations

Navigate to DeepGIS Search (/label/3d/search/)
Click "Weather" button in the bottom HUD toolbar
Toggle "Show Weather Stations" to enable
Load stations:
- Click "All States" to load all 21 stations (CA, AZ, CO, NV)
- Or click individual state buttons (CA, AZ, CO, NV) for specific regions
View weather data: Click on station markers for detailed information
Auto-update: Stations refresh every 15 minutes automatically

Experience URL Sharing

Navigate to any view in DeepGIS Search
Configure your experience:
- Set camera position and orientation
- Choose view mode (2D/3D/Columbus) - press V to toggle
- Enable drone modes (fly, orbit, takeoff, landing)
Share your view:
- Press S or click the Share button
- URL is automatically copied to clipboard
Generate QR Code: Click QR button to display scannable code

URL includes:

Parameter	Description
`lon`, `lat`, `alt`	Camera position
`heading`, `pitch`, `roll`	Camera orientation
`viewMode`	2D, 3D, or Columbus
`flyDist`, `hSpeed`, `vSpeed`	Drone fly settings
`orbRadius`, `orbPitch`, `orbYaw`	Orbit settings
`orbiting`, `flying`, `takeoff`, `landing`	Active mode flags

Keyboard Shortcuts

Press H to view all shortcuts in-app. Key shortcuts include:

Key	Action
`B`	Toggle 3D Buildings
`V`	Toggle View Mode (2D/3D/Columbus)
`F`	Toggle Full Screen
`H`	Show Keyboard Shortcuts Help
`S`	Share Current View
`Q`	Toggle QR Code
`T`	Hide/Show Toolbars
`W`	Toggle Wireframe
`D`	Drone Fly Forward
`U`	Takeoff (Up)
`L`	Land
`O`	Start Orbit
`P`	Pause/Stop Orbit
`J`	Toggle Virtual Joysticks
`↑` `↓` `←` `→`	Camera Perspectives (N/S/W/E)
`ESC`	Stop Orbit / Close Panels

🔧 Configuration

Environment Variables

DEBUG=True
DJANGO_SETTINGS_MODULE=deepgis_xr.settings
NVIDIA_VISIBLE_DEVICES=all  # For GPU support

# Remote AI Services (optional - defaults in docker-compose.yml)
# GROUNDING_DINO_API_URL=http://192.168.0.232:5000

Docker Configuration

The docker-compose.yml includes:

Web service: Django application with GPU support
Tile server: MapTiler TileServer GL for tile serving
Volume mounts:
- dreams_laboratory/scripts - ML model scripts
- deepgis_results - AI analysis results (shared with host)

GPU Support

To enable GPU for AI features:

Install NVIDIA Docker runtime
Uncomment GPU configuration in docker-compose.yml
Ensure NVIDIA_VISIBLE_DEVICES=all is set

📊 Recent Updates (December 2025)

✅ 3D Buildings Layer: OpenStreetMap buildings worldwide; toggleable via UI or B key; free and open data (ODbL)
✅ Experience URL Sharing: Complete camera state sharing via URL; supports takeoff/landing/fly/orbit modes; QR code generation
✅ Grounding DINO: Open-vocabulary detection with text prompts; remote API architecture for GPU servers
✅ Weather Stations: NWS integration with 21 stations across CA, AZ, CO, NV; HUD toolbar integration; auto-update every 15 min
✅ UI/UX: HUD toolbar with floating panels; aviation-style navigation widgets; drone fly/orbit modes
✅ AI/ML: YOLOv8 and Mask2Former integration; SAM optimization; clean viewport capture
✅ Performance: Memory optimization; improved error handling; duplicate entity prevention
✅ View Mode Switching: 2D/3D/Columbus view toggle with keyboard shortcut (V key); auto-restore from URL

🤝 Contributing

Contributions are welcome! Please follow these guidelines:

Fork the repository
Create a feature branch (git checkout -b feature/amazing-feature)
Commit your changes (git commit -m 'Add amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

Code Style

Follow PEP 8 for Python code
Use ESLint for JavaScript
Add docstrings to functions and classes
Include tests for new features

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

DeepGIS‑XR builds on concepts and systems originally developed for the Oceanographic Decision Support System (ODSS, MBARI), the Agricultural Decision Support System (AgDSS, University of Pennsylvania), the OpenUAV Project (University of Pennsylvania, Arizona State University), and DeepGIS (Arizona State University). The DeepGIS project acknowledges support from the National Science Foundation, the United States Department of Agriculture, and the National Aeronautics and Space Administration.

Technology Acknowledgments

CesiumJS: 3D globe visualization
Meta AI: Segment Anything Model
IDEA Research: Grounding DINO open-vocabulary detection
Ultralytics: YOLOv8 real-time detection
NASA/GSFC/ASU: LROC QuickMap lunar imagery
COCO Dataset: Object detection categories

📧 Contact & Support

Repository: Earth-Innovation-Hub/deepgis-xr
Issues: GitHub Issues

🗺️ Roadmap

✅ Completed

🔄 Q1 2026 - Near Term

🔮 Q1-Q4 2026 - Long Term

⚠️ Disclaimer

This software is provided "as is" without warranty. Use at your own risk. Intended for research and educational purposes. AI analysis results should be validated independently for critical applications.

Powered by Earth Innovation Hub (Arizona STEAM non-profit corporation)

Name		Name	Last commit message	Last commit date
Latest commit History 218 Commits
deepgis_xr		deepgis_xr
scripts		scripts
staticfiles		staticfiles
-xr		-xr
.env.example		.env.example
.gitignore		.gitignore
Dockerfile		Dockerfile
Dockerfile.cpu-backup		Dockerfile.cpu-backup
LICENSE		LICENSE
README.md		README.md
copy_static.sh		copy_static.sh
deepgis-xr-manuscript.tex		deepgis-xr-manuscript.tex
deepgis_topology_server.py		deepgis_topology_server.py
docker-compose.yml		docker-compose.yml
docker-compose.yml.backup		docker-compose.yml.backup
grounding_dino_api_client.py		grounding_dino_api_client.py
integrate-nginx-3d-models-fixed.sh		integrate-nginx-3d-models-fixed.sh
integrate-nginx-3d-models.sh		integrate-nginx-3d-models.sh
manage.py		manage.py
nginx-3d-models.conf		nginx-3d-models.conf
nginx-integration-patch.conf		nginx-integration-patch.conf
optimize_large_glb.py		optimize_large_glb.py
package.json		package.json
prepare_data.py		prepare_data.py
requirements-topology.txt		requirements-topology.txt
requirements.txt		requirements.txt
vite.config.js		vite.config.js

License

Earth-Innovation-Hub/deepgis-xr

Folders and files

Latest commit

History

Repository files navigation