Riverst

Do you need/want to...

• build interactive user-avatar experiences (speech-based, with video/multimodal support)?
• collect high-quality conversational data for research or industry projects?
• automatically analyze conversations for behavioral, linguistic, or engagement metrics?

Session overview	Automated analysis summary

If so, Riverst is for you.

---

Video demos

Video Demo	Description
Riverst Demo	Demo of the Riverst platform in action. What can I do with Riverst?
KIVA Activity Demo	Demonstration of a vocabulary teaching activity workflow, exploiting pipecat-ai-flows under the hood.
Riverst + KIVA Activity Summary	Overview and summary of configuration pages in Riverst and automatically generated summary of the KIVA activity.

---

What is Riverst?

Riverst is a platform for building, running, and analyzing interactive user-avatar conversations. It enables you to:

• Create engaging, speech-driven (and optionally multimodal) avatar interactions.
• Use these interactions for real-time applications, data collection, or research studies.
• Automatically process and analyze collected conversations with built-in pipelines (leveraging senselab) for behavioral and speech analysis.

---

How it works

1. User interacts with an avatar (primarily via speech, with optional video/multimodal input).
2. Conversations are recorded and stored for later review or analysis.
3. Automated pipelines process the data, extracting features and generating insights (e.g., speech metrics, behavioral markers).
4. Results can be used for research, product feedback, or to power adaptive experiences.

---

Project Structure

src/
├── server/              # Bot server implementation (FastAPI)
│   ├── main.py          # Server entrypoint
│   └── requirements.txt
└── flow-builder/        # Complex conversational flow/tree builder
└── client/              # Client implementations
    └── react/           # React web client
        └── index.html   # Client main page

---

Requirements

• Supported OS:

  • ✅ Apple Silicon (M-chip) macOS
  • ✅ Ubuntu Linux
  • ❌ Not supported on Windows

• Python 3.11+

• Node.js 16+ (for React client)

• API keys for 3rd party services (see .env.example files in both client and server)

• Modern web browser with WebRTC support (e.g., Chrome 134+)

---

Getting Started

1. Clone the repository

git clone https://github.com/sensein/riverst.git
cd riverst

2. Set up environment variables

• In src/server/, rename .env.example to .env and fill in the required API keys and configuration
• In src/client/react/, rename .env.example to .env and fill in the required API keys and configuration

Note: Not all API KEYS are strictly required. Only if you want to use a remote service, you need to expose the corresponding API KEY

3. Run

3a. Run with Docker

docker compose up --build

3b. Run manually in two different tabs of your terminal

• Start the server:

  cd src/server
  conda create -n riverst python=3.11
  conda activate riverst
  conda install -c conda-forge "ffmpeg=7.*"
  pip install -r requirements.txt
  python main.py

• Start the client:

  cd src/client/react
  npm install
  npm run dev

⚠️ Note: The server must be running before starting the client.

ℹ️ Note 2: For AWS EC2 deployment instructions, see here.

---

🙏 Acknowledgments

Riverst builds on the work of these fantastic open-source projects:

• TalkingHead — WebGL/Three.js talking-head renderer for avatar animation.
• Contextless Phonemes (CUPE) — Efficient phoneme modeling utilities.
• Pipecat — Real-time, multimodal agent framework for low-latency streaming.
• senselab — Python package for speech processing, feature extraction, and behavioral analysis.

---

Project board

The project is in continuous progress. Follow the project plan here.