Sunday, April 13, 2025

Here’s a full walkthrough on how to set up a Private Cloud Storage system with

1.JWT Authentication (Register/Login)

2.File Upload Feature

3.Dockerized Deployment
Project Structure:

cloudbox/
├── backend/
│   ├── app/
│   │   ├── auth/
│   │   │   ├── __init__.py
│   │   │   ├── auth.py        # Registration, Login, JWT creation
│   │   ├── files/
│   │   │   ├── __init__.py
│   │   │   ├── files.py       # File upload, management
│   │   ├── db/
│   │   │   ├── __init__.py
│   │   │   ├── models.py      # Database models
│   │   ├── main.py            # FastAPI main file
│   ├── Dockerfile
│   ├── requirements.txt
├── frontend/
│   ├── public/
│   │   ├── index.html         # Frontend files
│   ├── src/
│   └── Dockerfile
├── docker-compose.yml
├── .env
└── README.md

Backend Implementation: FastAPI + JWT + File Upload
backend/requirements.txt (Install dependencies)

fastapi==0.95.0
uvicorn==0.22.0
pydantic==1.10.2
python-dotenv==0.21.1
sqlalchemy==2.0.6
psycopg2==2.9.3
passlib[bcrypt]==1.7.4
fastapi-jwt-auth==1.9.1

backend/app/db/models.py (Database Models)
We use PostgreSQL to store user data and file information:

from sqlalchemy import Column, Integer, String, Text, ForeignKey
from sqlalchemy.orm import relationship
from . import Base

class User(Base):
    __tablename__ = 'users'
    id = Column(Integer, primary_key=True, index=True)
    username = Column(String(100), unique=True, index=True)
    email = Column(String(100), unique=True, index=True)
    hashed_password = Column(String, nullable=False)
    files = relationship('File', back_populates='owner')

class File(Base):
    __tablename__ = 'files'
    id = Column(Integer, primary_key=True, index=True)
    filename = Column(String(100))
    path = Column(Text)
    user_id = Column(Integer, ForeignKey('users.id'))

    owner = relationship('User', back_populates='files')

backend/app/auth/auth.py (JWT Authentication & Register/Login)

from fastapi import APIRouter, Depends, HTTPException, status
from pydantic import BaseModel
from passlib.context import CryptContext
from fastapi_jwt_auth import AuthJWT
from sqlalchemy.orm import Session
from ..db import crud, models, database

router = APIRouter()
pwd_context = CryptContext(schemes=["bcrypt"], deprecated="auto")

# User Registration Model
class UserCreate(BaseModel):
    username: str
    email: str
    password: str

# Login Model
class UserLogin(BaseModel):
    username: str
    password: str

# Create user
@router.post("/register")
def register_user(user: UserCreate, db: Session = Depends(database.get_db)):
    db_user = crud.get_user_by_username(db, user.username)
    if db_user:
        raise HTTPException(status_code=400, detail="Username already registered")
    hashed_password = pwd_context.hash(user.password)
    new_user = crud.create_user(db=db, username=user.username, email=user.email, password=hashed_password)
    return {"message": "User created successfully!"}

# Login user, returns JWT token
@router.post("/login")
def login_user(user: UserLogin, db: Session = Depends(database.get_db), Authorize: AuthJWT = Depends()):
    db_user = crud.get_user_by_username(db, user.username)
    if not db_user or not pwd_context.verify(user.password, db_user.hashed_password):
        raise HTTPException(status_code=401, detail="Invalid credentials")
    access_token = Authorize.create_access_token(subject=db_user.username)
    return {"access_token": access_token, "token_type": "bearer"}

backend/app/files/files.py (File Upload API)

from fastapi import APIRouter, File, UploadFile, Depends
from fastapi.responses import FileResponse
from sqlalchemy.orm import Session
from ..db import crud, models, database
import os

router = APIRouter()

# Upload file
@router.post("/upload/")
async def upload_file(file: UploadFile = File(...), db: Session = Depends(database.get_db)):
    file_location = f"files/{file.filename}"
    with open(file_location, "wb") as f:
        f.write(await file.read())
    new_file = crud.create_file(db=db, filename=file.filename, path=file_location)
    return {"filename": file.filename}

# Download file
@router.get("/files/{filename}")
async def download_file(filename: str):
    file_location = f"files/{filename}"
    if os.path.exists(file_location):
        return FileResponse(file_location)
    return {"error": "File not found"}

backend/app/main.py (FastAPI Entry Point)

from fastapi import FastAPI
from .auth import auth
from .files import files

app = FastAPI()

app.include_router(auth.router, prefix="/auth", tags=["auth"])
app.include_router(files.router, prefix="/files", tags=["files"])

backend/Dockerfile (Backend Docker Configuration)

FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install -r requirements.txt

COPY ./app /app

CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "5000"]

2. Frontend: Static Page (Example)
Assuming you already have a basic index.html, below is how you can set up a static website in the frontend/ folder.

frontend/Dockerfile

FROM nginx:alpine

COPY ./public /usr/share/nginx/html
COPY ./nginx.conf /etc/nginx/nginx.conf

EXPOSE 80

3. Docker Compose Configuration
The following docker-compose.yml will bring up the backend, frontend, and PostgreSQL database.

version: '3.8'

services:
  backend:
    build: ./backend
    container_name: netdisk-backend
    volumes:
      - ./backend:/app
      - ./storage:/app/files
    ports:
      - "5000:5000"
    depends_on:
      - db
    environment:
      - DATABASE_URL=postgresql://user:password@db:5432/netdisk

  frontend:
    build: ./frontend
    container_name: netdisk-frontend
    ports:
      - "80:80"
    depends_on:
      - backend

  db:
    image: postgres:13
    container_name: netdisk-db
    environment:
      POSTGRES_USER: user
      POSTGRES_PASSWORD: password
      POSTGRES_DB: netdisk
    volumes:
      - postgres_data:/var/lib/postgresql/data

volumes:
  postgres_data:

4. Initialize Database
After spinning up the containers, run the following to initialize your database:

docker-compose up -d
docker exec -it netdisk-backend bash
python -c "from app.db import models, database; models.Base.metadata.create_all(bind=database.engine)"

5. Start Services
Run the following command in the root project directory:

docker-compose up -d

Once everything is up, visit localhost or your server IP to access the application. You can register and log in, and then upload files.
Finished! You now have a Private Cloud Storage system with:
JWT Authentication (Login/Registration)

File Upload API

PostgreSQL Storage

Dockerized Deployment

OVER!!!

Wednesday, April 9, 2025

Download HD audio and video files with yt-dlp

#!/bin/bash

# Create a temporary file to store URLs
TMP_URLS=$(mktemp)

# Prompt user to paste URLs, each on a new line, and press Enter after each one. Press Enter on an empty line to finish.
echo "📥 Please paste the video or playlist URLs (one per line), press Enter after each one. When done, press Enter on an empty line to finish:"

# Read URLs from user input and store them in a temporary file
while true; do
    read -p "> " url
    [[ -z "$url" ]] && break  # Stop if the input is empty (empty line)
    echo "$url" >> "$TMP_URLS"  # Append URL to the temporary file
done

echo "🚀 Starting the download..."

# Loop through each URL and perform video and audio download
while read -r URL; do
    echo "🎬 Processing: $URL"

    # Download the video (MP4/WebM)
    yt-dlp -f "bv*+ba/bestvideo+bestaudio/best" 
    --merge-output-format mp4 
    --write-auto-sub --sub-lang en --convert-subs srt 
    --write-thumbnail --write-description --write-info-json 
    --embed-metadata 
    -o "%(upload_date)s_%(title)s.%(ext)s" "$URL"

    # Extract audio in Opus format
    yt-dlp -x --audio-format opus 
    --write-thumbnail --embed-thumbnail 
    --write-description --write-info-json 
    --write-auto-sub --sub-lang en --convert-subs srt 
    --embed-metadata 
    -o "%(upload_date)s_%(title)s_audio.%(ext)s" "$URL"

    echo "✅ Completed: $URL"
    echo ""
done < "$TMP_URLS"

# Clean up temporary file
rm "$TMP_URLS"

echo "🎉 All done! Video & Opus audio downloads completed."

Tutorial: How to Use This Script
Here’s a step-by-step guide to help you use the script correctly:

1. Install Dependencies
Before running the script, make sure that you have the necessary software installed:

yt-dlp: This is the tool used to download videos and audio.

Install yt-dlp:

pip install -U yt-dlp  # Using pip

# or
sudo apt install yt-dlp # For Debian/Ubuntu systems
2. Save the Script
Copy the script above and paste it into a file, say yt-dlp-download.sh.

Make the script executable:

chmod +x yt-dlp-download.sh

3. Run the Script
Now, run the script using:

./yt-dlp-download.sh

The script will prompt you to paste URLs.

4. Input URLs
After running the script, you will be prompted to paste the URLs of the videos or playlists you want to download.

Paste each URL one by one, and press Enter after each one.

When you’re done, just press Enter on an empty line (i.e., an empty input) to finish entering the URLs.

Example:

> https://www.youtube.com/watch?v=abc123
> https://www.youtube.com/watch?v=xyz456
>

5. Download Starts Automatically
Once you press Enter on the empty line, the script will start downloading:

It will first download the video in the best quality (MP4 or WebM).

It will then extract the audio in Opus format.

Each video will be processed and saved to the current directory with the format:
YYYYMMDD_Title.extension (for videos)
YYYYMMDD_Title_audio.extension (for audio).

6. Script Completion
Once all URLs are processed, the script will clean up temporary files and print:

🎉 All done! Video & Opus audio downloads completed.

Advanced Customization (Optional)
You can customize the script to fit your needs:

1. Change the output format: By default, the video is downloaded in MP4. If you want it in WebM or other formats, change the -f option in the script.

2. Audio formats: The script extracts audio in Opus. To extract audio in another format (e.g., MP3), modify the –audio-format option:

To download audio in MP3:

yt-dlp -x --audio-format mp3 ...

3. Additional options: You can add more options (like subtitle downloads, metadata, etc.) based on your requirements. Check the yt-dlp documentation for more advanced features.
Troubleshooting
If the script doesn’t work as expected, double-check that you’ve properly installed yt-dlp and that you are running the script from the directory where it’s located.

If you encounter issues like download failures or format errors, ensure that your yt-dlp version is up to date:

yt-dlp --update

😎

Wednesday, March 26, 2025

Rsync User Guide

Rsync is a powerful file synchronization tool used to transfer files and directories between local and remote servers. It supports incremental transfers, compression, encryption, and file attribute preservation, making it ideal for backups and data migrations.
✅ 1. Basic Syntax

rsync [options] source destination

Source: A local or remote path.

Destination: Also a local or remote path.

Remote Path Format:

user@host:/path/to/directory

✅ 2. Common Options

Option Description (English)
-a Archive mode (preserve permissions, timestamps).
-v Verbose mode for detailed output.
-z Enable compression during transfer.
-e ssh Use SSH for secure transfers.
--progress Show progress during transfer.
--delete Remove extra files at the destination.
-P Enable progress display and resume interrupted transfers.
-r Recursively copy directories.
--bwlimit=KB Limit bandwidth (in KB/s).
--exclude='pattern' Exclude specific files or directories.

✅ 3. Use Cases
📍 Local Sync

rsync -av /source/path/ /destination/path/

Synchronize contents between two local directories.
📍 Local to Remote

rsync -avz /local/path/ user@remote:/remote/path/

Transfer local files to a remote server.
📍 Remote to Local

rsync -avz user@remote:/remote/path/ /local/path/

Download files from a remote server to the local machine.
📍 Remote to Remote

rsync -avz -e ssh user_A@host_A:/path/to/source/ user_B@host_B:/path/to/destination/

Transfer files between two remote servers using your local machine.
📍 Specify SSH Port

rsync -avz -e "ssh -p 2222" /local/path/ user@remote:/remote/path/

Connect using a custom SSH port.
📍 Resume Interrupted Transfers

rsync -avzP user@remote:/remote/path/ /local/path/

Resume interrupted transfers with progress display.
📍 Exclude Files

rsync -avz --exclude='*.log' --exclude='cache/' /local/path/ user@remote:/remote/path/

Exclude .log files and cache directory.
📍 Delete Extra Files on Destination

rsync -avz --delete /local/path/ user@remote:/remote/path/

Delete files on the destination that are no longer in the source.
📍 Limit Bandwidth

rsync -avz --bwlimit=1000 /local/path/ user@remote:/remote/path/

Limit transfer speed to 1000KB/s.

Sunday, March 16, 2025

How to Use nginx -vt Instead of /sbin/nginx -vt

1️⃣ Method 1: Add /sbin to $PATH (Temporary)
🔹 For the current session only

export PATH=$PATH:/sbin

✅ Now you can run:

nginx -vt

⚠️ This change will be lost after closing the terminal.
2️⃣ Method 2: Permanently Add /sbin to $PATH
🔹 For all future sessions

Modify ~/.bashrc or ~/.profile (Debian typically uses ~/.profile):

echo 'export PATH=$PATH:/sbin' >> ~/.profile

Then apply the changes:

source ~/.profile

✅ Now nginx -vt will work every time you open a terminal.

📌 This applies to the current user only. To apply it system-wide, modify /etc/profile.
3️⃣ Method 3: Create a Symbolic Link (Recommended)
🔹 The simplest and most effective method

sudo ln -s /sbin/nginx /usr/local/bin/nginx

✅ Now you can run nginx -vt from anywhere:

nginx -vt

📌 Summary
Nginx Path Summary

📌 Summary

Method Persistent? Use Case Notes
Method 1: export PATH=$PATH:/sbin ❌ No (Only for current session) Temporary fix
Method 2: Edit ~/.profile ✅ Yes (For current user) User-specific solution
Method 3: ln -s /sbin/nginx /usr/local/bin/nginx ✅ Yes (For all users) Best and simplest solution

If the reboot command is missing, try these solutions:

1. Use systemctl reboot
On Debian 12 or other systemd-based distributions, reboot is essentially an alias for systemctl reboot. If reboot is not found, try:

systemctl reboot

2. Run /sbin/reboot Directly
Sometimes, reboot might not be in your PATH. Try running it directly:

/sbin/reboot

If this works, your PATH might be misconfigured. Check it with:

echo $PATH

If /sbin is missing, temporarily add it:

export PATH=$PATH:/sbin:/usr/sbin

3. Check If reboot Is Missing
If the command is truly missing, reinstall it with:

apt update
apt install systemd-sysv -y

The systemd-sysv package includes reboot, shutdown, and other essential commands.

4. Use shutdown Instead
If reboot is unavailable, try:
shutdown -r now
-r means reboot
now means execute immediately
If none of these solutions work, check if /sbin/reboot exists:

ls -l /sbin/reboot

If it’s missing, verify whether it was removed or replaced using:

which reboot
type reboot