An autonomous AI-powered UI reasoning and quality assurance platform that understands, tests, repairs, and verifies modern web interfaces using multi-agent workflows, computer vision, and browser automation.
Moving beyond scripted testing toward autonomous interface reasoning.
Modern end-to-end testing frameworks rely on predefined selectors and brittle test scripts. Even minor UI changes can cause tests to fail despite the application functioning correctly.
SentinelUI approaches quality assurance differently.
Instead of simply executing scripts, SentinelUI acts as an autonomous software agent that:
- Understands the structure of a web interface
- Plans user interactions to accomplish a goal
- Recovers when the interface changes
- Verifies whether failures are reproducible
- Produces actionable reports for developers
The objective is not to replace traditional testing frameworks, but to augment them with reasoning, adaptability, and autonomous decision making.
Imagine an AI QA engineer that can say:
"The Login button has moved from the navigation bar into the sidebar. I located it, updated the execution plan, completed the workflow successfully, and no regression was detected."
instead of
"Locator not found."
That is the long-term vision behind SentinelUI.
Transforms high-level user goals into executable browser workflows.
Example:
Goal:
Purchase a product
↓
Open homepage
↓
Search product
↓
Add to cart
↓
Checkout
↓
Verify confirmation
Rather than relying only on CSS selectors, SentinelUI combines:
- DOM structure
- Accessibility Tree
- Visual screenshots
- Bounding regions
- Computed styles
- Interaction metadata
to build a richer understanding of the interface.
The platform fuses visual understanding with structural page information to reason about interfaces more reliably than screenshots or DOM inspection alone.
If a button moves or the layout changes, SentinelUI attempts to recover automatically instead of immediately failing.
Recovery strategies include:
- Alternative element search
- Semantic similarity
- Navigation replanning
- Multiple interaction candidates
Large web pages often contain thousands of irrelevant DOM nodes.
SentinelUI compiles pages into compact semantic representations by removing:
- Scripts
- Styles
- Hidden nodes
- Analytics elements
- Decorative containers
while preserving interactive components.
Every detected issue is validated before reporting.
Verification may include:
- Retry execution
- Fresh browser session
- Different viewport sizes
- Alternate navigation path
- Multiple reproduction attempts
Only reproducible failures are reported.
Instead of only identifying a failed interaction, SentinelUI aims to explain why it failed.
Examples:
- Missing DOM node
- Hidden element
- Rendering issue
- Disabled interaction
- Network failure
- Client-side state inconsistency
Generated reports include:
- Executive summary
- Reproduction steps
- Screenshots
- DOM snapshots
- Console logs
- Network logs
- Suggested root cause
- Reproducible Playwright script
User Goal
│
▼
Planning Agent
│
▼
Browser Interaction
│
┌──────────┴──────────┐
▼ ▼
Vision Understanding DOM Compiler
│ │
└──────────┬──────────┘
▼
Interface Reasoner
│
▼
Action Planner
│
▼
Playwright Execution
│
▼
Verification Engine
│
┌─────────┴─────────┐
▼ ▼
Success Failure Analysis
│
▼
Recovery Planner
│
▼
Developer Report
| Layer | Technology |
|---|---|
| Language | Python |
| Backend API | FastAPI |
| Browser Automation | Playwright |
| Agent Framework | LangGraph |
| Vision Models | OpenAI Vision Models |
| Containerization | Docker |
| UI Dashboard | React + TypeScript |
| Database | PostgreSQL (planned) |
| Observability | OpenTelemetry (planned) |
- Browser automation
- FastAPI backend
- Playwright execution engine
- Docker sandbox
- Initial reporting
- Semantic DOM compiler
- Vision + DOM fusion
- Task planner
- Multi-agent workflow
- Adaptive navigation
- Failure recovery
- Retry strategies
- Confidence scoring
- Root cause analysis
- Automatic severity classification
- Report generation
- Performance metrics
- Plugin architecture
- Additional browser support
- Custom reasoning modules
- Benchmark suite
- Documentation improvements
SentinelUI is being built as an open-source research and engineering project.
Contributions of all sizes are welcome.
Areas where contributors can help include:
- Browser automation
- Agent workflows
- Computer vision
- LLM orchestration
- FastAPI
- React frontend
- Documentation
- Testing
- Performance optimization
If you'd like to contribute:
- Fork the repository.
- Create a feature branch.
- Make your changes.
- Add tests where applicable.
- Open a pull request describing your work.
Constructive discussions, architecture suggestions, and issue reports are just as valuable as code contributions.
- Reliability over hype
- Measurable performance
- Transparent reasoning
- Modular architecture
- Security by default
- Open collaboration
🚧 Early Development
SentinelUI is currently in active development. The architecture and roadmap may evolve as the project matures.
This project will be released under the MIT License.
SentinelUI is inspired by ongoing research in browser agents, autonomous software engineering, multimodal reasoning, and modern software quality assurance.
The goal is not to replicate existing tools, but to explore a modular, open, and research-friendly architecture that the community can extend together.
If you find the idea interesting, consider giving the project a ⭐ and following its progress.
Contributions, feedback, discussions, and ideas are always welcome.