Sunday, April 13, 2025

Here’s a full walkthrough on how to set up a Private Cloud Storage system with

1.JWT Authentication (Register/Login)

2.File Upload Feature

3.Dockerized Deployment
Project Structure:

cloudbox/
├── backend/
│   ├── app/
│   │   ├── auth/
│   │   │   ├── __init__.py
│   │   │   ├── auth.py        # Registration, Login, JWT creation
│   │   ├── files/
│   │   │   ├── __init__.py
│   │   │   ├── files.py       # File upload, management
│   │   ├── db/
│   │   │   ├── __init__.py
│   │   │   ├── models.py      # Database models
│   │   ├── main.py            # FastAPI main file
│   ├── Dockerfile
│   ├── requirements.txt
├── frontend/
│   ├── public/
│   │   ├── index.html         # Frontend files
│   ├── src/
│   └── Dockerfile
├── docker-compose.yml
├── .env
└── README.md

Backend Implementation: FastAPI + JWT + File Upload
backend/requirements.txt (Install dependencies)

fastapi==0.95.0
uvicorn==0.22.0
pydantic==1.10.2
python-dotenv==0.21.1
sqlalchemy==2.0.6
psycopg2==2.9.3
passlib[bcrypt]==1.7.4
fastapi-jwt-auth==1.9.1

backend/app/db/models.py (Database Models)
We use PostgreSQL to store user data and file information:

from sqlalchemy import Column, Integer, String, Text, ForeignKey
from sqlalchemy.orm import relationship
from . import Base

class User(Base):
    __tablename__ = 'users'
    id = Column(Integer, primary_key=True, index=True)
    username = Column(String(100), unique=True, index=True)
    email = Column(String(100), unique=True, index=True)
    hashed_password = Column(String, nullable=False)
    files = relationship('File', back_populates='owner')

class File(Base):
    __tablename__ = 'files'
    id = Column(Integer, primary_key=True, index=True)
    filename = Column(String(100))
    path = Column(Text)
    user_id = Column(Integer, ForeignKey('users.id'))

    owner = relationship('User', back_populates='files')

backend/app/auth/auth.py (JWT Authentication & Register/Login)

from fastapi import APIRouter, Depends, HTTPException, status
from pydantic import BaseModel
from passlib.context import CryptContext
from fastapi_jwt_auth import AuthJWT
from sqlalchemy.orm import Session
from ..db import crud, models, database

router = APIRouter()
pwd_context = CryptContext(schemes=["bcrypt"], deprecated="auto")

# User Registration Model
class UserCreate(BaseModel):
    username: str
    email: str
    password: str

# Login Model
class UserLogin(BaseModel):
    username: str
    password: str

# Create user
@router.post("/register")
def register_user(user: UserCreate, db: Session = Depends(database.get_db)):
    db_user = crud.get_user_by_username(db, user.username)
    if db_user:
        raise HTTPException(status_code=400, detail="Username already registered")
    hashed_password = pwd_context.hash(user.password)
    new_user = crud.create_user(db=db, username=user.username, email=user.email, password=hashed_password)
    return {"message": "User created successfully!"}

# Login user, returns JWT token
@router.post("/login")
def login_user(user: UserLogin, db: Session = Depends(database.get_db), Authorize: AuthJWT = Depends()):
    db_user = crud.get_user_by_username(db, user.username)
    if not db_user or not pwd_context.verify(user.password, db_user.hashed_password):
        raise HTTPException(status_code=401, detail="Invalid credentials")
    access_token = Authorize.create_access_token(subject=db_user.username)
    return {"access_token": access_token, "token_type": "bearer"}

backend/app/files/files.py (File Upload API)

from fastapi import APIRouter, File, UploadFile, Depends
from fastapi.responses import FileResponse
from sqlalchemy.orm import Session
from ..db import crud, models, database
import os

router = APIRouter()

# Upload file
@router.post("/upload/")
async def upload_file(file: UploadFile = File(...), db: Session = Depends(database.get_db)):
    file_location = f"files/{file.filename}"
    with open(file_location, "wb") as f:
        f.write(await file.read())
    new_file = crud.create_file(db=db, filename=file.filename, path=file_location)
    return {"filename": file.filename}

# Download file
@router.get("/files/{filename}")
async def download_file(filename: str):
    file_location = f"files/{filename}"
    if os.path.exists(file_location):
        return FileResponse(file_location)
    return {"error": "File not found"}

backend/app/main.py (FastAPI Entry Point)

from fastapi import FastAPI
from .auth import auth
from .files import files

app = FastAPI()

app.include_router(auth.router, prefix="/auth", tags=["auth"])
app.include_router(files.router, prefix="/files", tags=["files"])

backend/Dockerfile (Backend Docker Configuration)

FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install -r requirements.txt

COPY ./app /app

CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "5000"]

2. Frontend: Static Page (Example)
Assuming you already have a basic index.html, below is how you can set up a static website in the frontend/ folder.

frontend/Dockerfile

FROM nginx:alpine

COPY ./public /usr/share/nginx/html
COPY ./nginx.conf /etc/nginx/nginx.conf

EXPOSE 80

3. Docker Compose Configuration
The following docker-compose.yml will bring up the backend, frontend, and PostgreSQL database.

version: '3.8'

services:
  backend:
    build: ./backend
    container_name: netdisk-backend
    volumes:
      - ./backend:/app
      - ./storage:/app/files
    ports:
      - "5000:5000"
    depends_on:
      - db
    environment:
      - DATABASE_URL=postgresql://user:password@db:5432/netdisk

  frontend:
    build: ./frontend
    container_name: netdisk-frontend
    ports:
      - "80:80"
    depends_on:
      - backend

  db:
    image: postgres:13
    container_name: netdisk-db
    environment:
      POSTGRES_USER: user
      POSTGRES_PASSWORD: password
      POSTGRES_DB: netdisk
    volumes:
      - postgres_data:/var/lib/postgresql/data

volumes:
  postgres_data:

4. Initialize Database
After spinning up the containers, run the following to initialize your database:

docker-compose up -d
docker exec -it netdisk-backend bash
python -c "from app.db import models, database; models.Base.metadata.create_all(bind=database.engine)"

5. Start Services
Run the following command in the root project directory:

docker-compose up -d

Once everything is up, visit localhost or your server IP to access the application. You can register and log in, and then upload files.
Finished! You now have a Private Cloud Storage system with:
JWT Authentication (Login/Registration)

File Upload API

PostgreSQL Storage

Dockerized Deployment

OVER!!!

Wednesday, April 9, 2025

Download HD audio and video files with yt-dlp

#!/bin/bash

# Create a temporary file to store URLs
TMP_URLS=$(mktemp)

# Prompt user to paste URLs, each on a new line, and press Enter after each one. Press Enter on an empty line to finish.
echo "📥 Please paste the video or playlist URLs (one per line), press Enter after each one. When done, press Enter on an empty line to finish:"

# Read URLs from user input and store them in a temporary file
while true; do
    read -p "> " url
    [[ -z "$url" ]] && break  # Stop if the input is empty (empty line)
    echo "$url" >> "$TMP_URLS"  # Append URL to the temporary file
done

echo "🚀 Starting the download..."

# Loop through each URL and perform video and audio download
while read -r URL; do
    echo "🎬 Processing: $URL"

    # Download the video (MP4/WebM)
    yt-dlp -f "bv*+ba/bestvideo+bestaudio/best" 
    --merge-output-format mp4 
    --write-auto-sub --sub-lang en --convert-subs srt 
    --write-thumbnail --write-description --write-info-json 
    --embed-metadata 
    -o "%(upload_date)s_%(title)s.%(ext)s" "$URL"

    # Extract audio in Opus format
    yt-dlp -x --audio-format opus 
    --write-thumbnail --embed-thumbnail 
    --write-description --write-info-json 
    --write-auto-sub --sub-lang en --convert-subs srt 
    --embed-metadata 
    -o "%(upload_date)s_%(title)s_audio.%(ext)s" "$URL"

    echo "✅ Completed: $URL"
    echo ""
done < "$TMP_URLS"

# Clean up temporary file
rm "$TMP_URLS"

echo "🎉 All done! Video & Opus audio downloads completed."

Tutorial: How to Use This Script
Here’s a step-by-step guide to help you use the script correctly:

1. Install Dependencies
Before running the script, make sure that you have the necessary software installed:

yt-dlp: This is the tool used to download videos and audio.

Install yt-dlp:

pip install -U yt-dlp  # Using pip

# or
sudo apt install yt-dlp # For Debian/Ubuntu systems
2. Save the Script
Copy the script above and paste it into a file, say yt-dlp-download.sh.

Make the script executable:

chmod +x yt-dlp-download.sh

3. Run the Script
Now, run the script using:

./yt-dlp-download.sh

The script will prompt you to paste URLs.

4. Input URLs
After running the script, you will be prompted to paste the URLs of the videos or playlists you want to download.

Paste each URL one by one, and press Enter after each one.

When you’re done, just press Enter on an empty line (i.e., an empty input) to finish entering the URLs.

Example:

> https://www.youtube.com/watch?v=abc123
> https://www.youtube.com/watch?v=xyz456
>

5. Download Starts Automatically
Once you press Enter on the empty line, the script will start downloading:

It will first download the video in the best quality (MP4 or WebM).

It will then extract the audio in Opus format.

Each video will be processed and saved to the current directory with the format:
YYYYMMDD_Title.extension (for videos)
YYYYMMDD_Title_audio.extension (for audio).

6. Script Completion
Once all URLs are processed, the script will clean up temporary files and print:

🎉 All done! Video & Opus audio downloads completed.

Advanced Customization (Optional)
You can customize the script to fit your needs:

1. Change the output format: By default, the video is downloaded in MP4. If you want it in WebM or other formats, change the -f option in the script.

2. Audio formats: The script extracts audio in Opus. To extract audio in another format (e.g., MP3), modify the –audio-format option:

To download audio in MP3:

yt-dlp -x --audio-format mp3 ...

3. Additional options: You can add more options (like subtitle downloads, metadata, etc.) based on your requirements. Check the yt-dlp documentation for more advanced features.
Troubleshooting
If the script doesn’t work as expected, double-check that you’ve properly installed yt-dlp and that you are running the script from the directory where it’s located.

If you encounter issues like download failures or format errors, ensure that your yt-dlp version is up to date:

yt-dlp --update

😎