Integrate Oudelaa backend features, security, tests, and deployment updates

هذا الالتزام موجود في:
boutmoun123
2026-07-26 16:58:53 +03:00
الأصل 1b24ca4294
التزام 2fd5322ef7
178 ملفات معدلة مع 19068 إضافات و2592 حذوفات

71
docs/FEED_RANKING.md Normal file
عرض الملف

@@ -0,0 +1,71 @@
# Home feed ranking
The home feed contract remains `GET /api/v1/feed/me`; no query parameter or response field was removed or renamed.
## Candidate generation
Each request merges two bounded candidate pools:
- Recent eligible posts (65%).
- High-quality eligible posts ranked by saves, shares, comments, and likes (35%).
The merged pool is deduplicated and capped between 400 and 1,000 candidates depending on the requested page size. The default feed remains focused on followed accounts and falls back to public discovery only when the default following feed is empty.
## Hard safety filters
The ranking layer cannot override these rules:
- Deleted and archived posts are excluded.
- Posts with `moderationStatus=hidden` are excluded.
- Blocked or blocking authors are excluded.
- Disabled and missing authors are excluded after population.
- Posts reported by the current viewer are excluded unless the report was rejected.
- Public, followers-only, and own-post visibility rules remain enforced.
## Personalized signals
A short-lived ranking profile is computed from the viewer's last 90 days of:
- Likes (weight 1.5).
- Comments (weight 3).
- Saves (weight 4).
- Shares (weight 5).
Those events produce capped affinities for authors, post types, and hashtags. Existing profile preferences, follows, location, and requested post type remain signals. The profile is cached for 30 seconds by default.
## Scoring
Raw counters use `log1p` normalization so viral totals cannot grow without bound and erase personalization. Ranking combines:
- Log-normalized engagement quality.
- Exponential freshness decay with a small boost for posts younger than six hours.
- Follow, author affinity, post-type affinity, and hashtag affinity.
- Profile interests and a bounded geographic boost.
- Small verification and follower-count priors.
- A penalty for posts the viewer already interacted with, reducing repetition without hiding them.
## Diversity
After relevance scoring, a deterministic greedy reranker applies penalties for:
- Consecutive posts from the same author.
- Repeated appearances by the same author.
- Overrepresented post types.
- Repeated hashtags.
This preserves strong content while avoiding a page dominated by one creator or one format.
## Cursor stability
New cursors contain an opaque version, offset, and ranking timestamp. Every page in the cursor chain applies the same `createdAt <= rankedAt` window and computes freshness against the same timestamp, so newly created posts do not shift already paginated results. Legacy numeric offset cursors remain supported.
## Configuration
```dotenv
FEED_CACHE_ENABLED=true
FEED_CACHE_USER_TTL_SECONDS=15
FEED_CACHE_TRENDING_TTL_SECONDS=30
FEED_CACHE_RANKING_PROFILE_TTL_SECONDS=30
```
Viewer-level watch duration and explicit "Not interested" events do not exist in the current data model. They should be added as future ranking signals when the mobile client starts emitting those events; global `viewCount` and `playCount` are used only as weak quality priors.

عرض الملف

@@ -0,0 +1,80 @@
# Search architecture
The public API contract is unchanged:
- `GET /api/v1/search`
- `GET /api/v1/search/users`
- `GET /api/v1/search/posts`
- `GET /api/v1/search/hashtags`
- `GET /api/v1/search/suggestions`
The existing `q`, `type`, `page`, `limit`, and `sortOrder` query parameters and all response and pagination shapes remain compatible with existing clients.
## Search engines
`SEARCH_ENGINE` accepts:
- `auto` (default): use Atlas Search and automatically use compatibility search if Atlas Search or its indexes are unavailable.
- `atlas`: prefer Atlas Search. With `SEARCH_FALLBACK_ENABLED=true`, service remains available during an Atlas Search failure.
- `regex`: use the original MongoDB regex implementation only.
When Atlas Search fails, the service logs one warning and waits `SEARCH_ATLAS_RETRY_SECONDS` before retrying it. Requests continue through the compatibility implementation during that interval.
## Ranking
User ranking combines:
1. Exact username relevance.
2. Username, stage name, and display name autocomplete relevance.
3. One-character typo tolerance for queries of at least three characters.
4. Whether the viewer follows the result.
5. Whether the viewer recently liked or saved posts from the result.
6. Verification and a logarithmic follower-count boost.
Post ranking combines:
1. Content, hashtag, style, maqam, and rhythm relevance.
2. Typo tolerance for queries of at least four characters.
3. Follow and recent interaction affinity with the author.
4. A logarithmic engagement boost.
5. A seven-day recency decay.
Blocked users, disabled authors, deleted or archived posts, moderation state, and visibility are hard filters. They never become ranking signals and cannot be bypassed by a high search score.
## Atlas Search index deployment
The index definitions are stored in:
- `ops/atlas-search/users_search.json`
- `ops/atlas-search/posts_search.json`
Set `MONGODB_URI` to the target Atlas database and run:
```bash
npm run search:sync-indexes
```
The script creates missing indexes and updates existing indexes using `SEARCH_ATLAS_USER_INDEX` and `SEARCH_ATLAS_POST_INDEX`. Atlas builds indexes asynchronously; `SEARCH_ENGINE=auto` continues using compatibility search until they become queryable.
Recommended production configuration:
```dotenv
SEARCH_ENGINE=auto
SEARCH_ATLAS_USER_INDEX=users_search
SEARCH_ATLAS_POST_INDEX=posts_search
SEARCH_FALLBACK_ENABLED=true
SEARCH_ATLAS_RETRY_SECONDS=300
```
For local MongoDB Community development, keep `auto` for production parity or use `regex` to suppress the initial Atlas capability check.
## Verification
Run:
```bash
npm run build
npm test -- --runInBand src/modules/search/search.service.spec.ts
```
The tests verify response compatibility, Atlas result ordering, privacy clauses, and fallback retry throttling.

عرض الملف

@@ -0,0 +1,105 @@
# System strength report
Date: 2026-07-22
## Result
The backend is functionally stable and performs well in the tested local environment. It is not yet possible to claim production-scale capacity because the test used a single Node.js process, local MongoDB, a small dataset, and disabled Redis/S3/Atlas dependencies.
## Verification summary
| Area | Result |
| --- | --- |
| TypeScript build | Passed |
| ESLint | Passed |
| Unit/integration suites | 85/85 passed |
| Unit/integration tests | 947/947 passed |
| End-to-end scenarios | 14/14 passed |
| Dependency audit | 0 cached advisory findings; live audit runs in CI |
| Statement coverage | 93.08% |
| Branch coverage | 67.97% |
| Function coverage | 92.93% |
| Line coverage | 93.26% |
The executable service layer is also protected by an aggregate coverage gate: 92.81% statements,
93.02% lines, 93.67% functions, and 74.96% branches across 43 service files. `npm run
test:coverage` enforces both the project-wide gates and these service-layer gates, and the same
command is mandatory in CI.
## Performance results
Tests ran on one local process and are useful for regression comparison, not public capacity promises.
| Workload | Concurrency | Throughput | Average | p95 | p99 | Success |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| Liveness health | 20 | 2357 req/s | 8.47 ms | 12.64 ms | 15.92 ms | 100% |
| Cached personalized feed | 20 | 567 req/s | 35.20 ms | 40.78 ms | 45.25 ms | 100% |
| Liveness burst | 100 | 2448 req/s | 40.71 ms | 48.89 ms | 72.62 ms | 100% |
| Uncached personalized feed | 20 | 65 req/s | 304.34 ms | 392.81 ms | 445.92 ms | 100% |
| Uncached compatibility search | 10 | 109 req/s | 91.38 ms | 100.38 ms | 106.79 ms | 100% |
| Hardened health gate | 50 | 2041 req/s | 24.40 ms | 39.85 ms | 48.57 ms | 100% |
| Cold-cache/coalesced feed gate | 30 | 336 req/s | 88.79 ms | 116.14 ms | 168.69 ms | 100% |
Cold startup was approximately 4.8 seconds.
## Security checks
- Forged JWT was rejected with 401.
- NoSQL-shaped login input was rejected with 400.
- Unknown privileged fields were rejected with 400.
- Malformed JSON was rejected with 400.
- A request above the configured body limit was rejected with 413.
- Login throttling returned 429 after the allowed attempts were consumed.
- Production CORS returned the configured origin and did not return an allow-origin header for an untrusted origin.
- HSTS, X-Frame-Options, X-Content-Type-Options, and Referrer-Policy were present in production mode.
- The npm audit reported zero known vulnerabilities.
## Concurrency and realtime checks
- Twenty simultaneous duplicate like requests produced one like and a correct counter.
- Authenticated Socket.IO clients connected to both `chat` and `notifications` namespaces.
- An anonymous Socket.IO client was disconnected.
- Refresh-token rotation, notification delivery, feed exclusion, comments, uploads, and superadmin sessions passed end-to-end.
## Improvements made during testing
- Added `/api/v1/health/ready`, which checks MongoDB, Redis when enabled, and storage, and returns 503 when degraded.
- Added readiness unit and end-to-end coverage.
- Added real Socket.IO end-to-end tests.
- Added concurrent idempotency coverage for likes.
- Improved startup benchmark diagnostics to include the child exit code and recent logs.
- Corrected local storage configuration to use direct media access.
- Added local single-flight and ownership-safe Redis leases to prevent cache stampedes.
- Added configurable MongoDB connection pools and disabled automatic production index builds by default.
- Added response compression and bounded Node HTTP transport settings.
- Added request deadlines, graceful request draining, and drain-aware readiness.
- Added event-loop, in-flight request, timeout, and bounded-cardinality route metrics.
- Added direct, user-scoped S3 PUT uploads so large media can bypass Node memory.
- Added CI performance gates and made unit, E2E, audit, and Docker build checks mandatory before deployment.
- Added behavioral, security, failure-path, controller-delegation, repository-persistence, and
bootstrap tests, increasing the regression suite to 947 tests.
- Added enforced project-wide and service-layer coverage gates so the achieved baseline cannot
silently regress.
- Hardened the production container with a non-root user, `tini`, a healthcheck, and secret-safe Docker context exclusions.
## Remaining production validation
Before a high-traffic launch, run the same tests in staging with production-sized data and production topology:
1. Multiple application replicas behind the real load balancer.
2. Redis enabled for cache, throttling, queues, and Socket.IO fan-out.
3. S3-compatible storage, signed media URLs, and CDN behavior.
4. Atlas Search indexes and representative Arabic/English datasets.
5. Long soak tests (2-8 hours), failover tests, and MongoDB/Redis latency injection.
6. Continue raising branch coverage, prioritizing rare infrastructure and dependency-failure combinations.
7. Run an external DAST/SAST and penetration test before handling sensitive production data.
## Current assessment
- Functional correctness: strong.
- Local performance: strong with cache; acceptable but database-bound without cache.
- Security controls: strong baseline.
- Operational readiness: good after adding dependency readiness checks.
- Automated test maturity: strong baseline, with more than 90% statement, line, and function coverage
plus separate runtime-service gates.
- Production-scale confidence: pending staging tests with real infrastructure and data volume.