From 79d240f5b7b27ff5f55f73f4296ad4d127993c52 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Sebasti=C3=A1n=20Ram=C3=ADrez?= Date: Thu, 19 Sep 2024 20:11:33 +0200 Subject: [PATCH] =?UTF-8?q?=E2=99=BB=EF=B8=8F=20Simplify=20domains=20with?= =?UTF-8?q?=20`api.example.com`=20for=20API=20and=20`dashboard.example.com?= =?UTF-8?q?`=20for=20frontend,=20improve=20local=20development=20with=20`l?= =?UTF-8?q?ocalhost`=20(#1344)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .env | 8 +++ .github/workflows/playwright.yml | 2 +- backend/README.md | 52 +++----------- backend/app/core/config.py | 10 +-- backend/app/utils.py | 4 +- deployment.md | 14 ++-- development.md | 120 +++++++++++++++++++------------ docker-compose.override.yml | 10 ++- docker-compose.yml | 12 ++-- frontend/.env | 2 +- frontend/README.md | 6 +- 11 files changed, 122 insertions(+), 118 deletions(-) diff --git a/.env b/.env index 98c8196..1d44286 100644 --- a/.env +++ b/.env @@ -1,6 +1,14 @@ # Domain # This would be set to the production domain with an env var on deployment +# used by Traefik to transmit traffic and aqcuire TLS certificates DOMAIN=localhost +# To test the local Traefik config +# DOMAIN=localhost.tiangolo.com + +# Used by the backend to generate links in emails to the frontend +FRONTEND_HOST=http://localhost:5173 +# In staging and production, set this env var to the frontend host, e.g. +# FRONTEND_HOST=https://dashboard.example.com # Environment: local, staging, production ENVIRONMENT=local diff --git a/.github/workflows/playwright.yml b/.github/workflows/playwright.yml index 60f4cea..a884800 100644 --- a/.github/workflows/playwright.yml +++ b/.github/workflows/playwright.yml @@ -41,7 +41,7 @@ jobs: working-directory: frontend - run: docker compose build - run: docker compose down -v --remove-orphans - - run: docker compose up -d --wait + - run: docker compose up -d --wait backend mailcatcher - name: Run Playwright tests run: npx playwright test --fail-on-flaky-tests --trace=retain-on-failure working-directory: frontend diff --git a/backend/README.md b/backend/README.md index 7e78296..e7782d4 100644 --- a/backend/README.md +++ b/backend/README.md @@ -5,45 +5,11 @@ * [Docker](https://www.docker.com/). * [Poetry](https://python-poetry.org/) for Python package and environment management. -## Local Development +## Docker Compose -* Start the stack with Docker Compose: +Start the local development environment with Docker Compose following the guide in [../development.md](../development.md). -```bash -docker compose up -d -``` - -* Now you can open your browser and interact with these URLs: - -Frontend, built with Docker, with routes handled based on the path: http://localhost - -Backend, JSON based web API based on OpenAPI: http://localhost/api/ - -Automatic interactive documentation with Swagger UI (from the OpenAPI backend): http://localhost/docs - -Adminer, database web administration: http://localhost:8080 - -Traefik UI, to see how the routes are being handled by the proxy: http://localhost:8090 - -**Note**: The first time you start your stack, it might take a minute for it to be ready. While the backend waits for the database to be ready and configures everything. You can check the logs to monitor it. - -To check the logs, run: - -```bash -docker compose logs -``` - -To check the logs of a specific service, add the name of the service, e.g.: - -```bash -docker compose logs backend -``` - -If your Docker is not running in `localhost` (the URLs above wouldn't work) you would need to use the IP or domain where your Docker is running. - -## Backend local development, additional details - -### General workflow +## General Workflow By default, the dependencies are managed with [Poetry](https://python-poetry.org/), go there and install it. @@ -63,13 +29,13 @@ Make sure your editor is using the correct Python virtual environment. Modify or add SQLModel models for data and SQL tables in `./backend/app/models.py`, API endpoints in `./backend/app/api/`, CRUD (Create, Read, Update, Delete) utils in `./backend/app/crud.py`. -### VS Code +## VS Code There are already configurations in place to run the backend through the VS Code debugger, so that you can use breakpoints, pause and explore variables, etc. The setup is also already configured so you can run the tests through the VS Code Python tests tab. -### Docker Compose Override +## Docker Compose Override During development, you can change Docker Compose settings that will only affect the local development environment in the file `docker-compose.override.yml`. @@ -123,7 +89,7 @@ Nevertheless, if it doesn't detect a change but a syntax error, it will just sto ...this previous detail is what makes it useful to have the container alive doing nothing and then, in a Bash session, make it run the live reload server. -### Backend tests +## Backend tests To test the backend run: @@ -135,7 +101,7 @@ The tests run with Pytest, modify and add tests to `./backend/app/tests/`. If you use GitHub Actions the tests will run automatically. -#### Test running stack +### Test running stack If your stack is already up and you just want to run the tests, you can use: @@ -151,11 +117,11 @@ For example, to stop on first error: docker compose exec backend bash /app/tests-start.sh -x ``` -#### Test Coverage +### Test Coverage When the tests are run, a file `htmlcov/index.html` is generated, you can open it in your browser to see the coverage of the tests. -### Migrations +## Migrations As during local development your app directory is mounted as a volume inside the container, you can also run the migrations with `alembic` commands inside the container and the migration code will be in your app directory (instead of being only inside the container). So you can add it to your git repository. diff --git a/backend/app/core/config.py b/backend/app/core/config.py index 1e3a440..464aa9d 100644 --- a/backend/app/core/config.py +++ b/backend/app/core/config.py @@ -31,17 +31,9 @@ class Settings(BaseSettings): SECRET_KEY: str = secrets.token_urlsafe(32) # 60 minutes * 24 hours * 8 days = 8 days ACCESS_TOKEN_EXPIRE_MINUTES: int = 60 * 24 * 8 - DOMAIN: str = "localhost" + FRONTEND_HOST: str = "http://localhost:5173" ENVIRONMENT: Literal["local", "staging", "production"] = "local" - @computed_field # type: ignore[prop-decorator] - @property - def server_host(self) -> str: - # Use HTTPS for anything other than local development - if self.ENVIRONMENT == "local": - return f"http://{self.DOMAIN}" - return f"https://{self.DOMAIN}" - BACKEND_CORS_ORIGINS: Annotated[ list[AnyUrl] | str, BeforeValidator(parse_cors) ] = [] diff --git a/backend/app/utils.py b/backend/app/utils.py index d5ccf31..2679937 100644 --- a/backend/app/utils.py +++ b/backend/app/utils.py @@ -64,7 +64,7 @@ def generate_test_email(email_to: str) -> EmailData: def generate_reset_password_email(email_to: str, email: str, token: str) -> EmailData: project_name = settings.PROJECT_NAME subject = f"{project_name} - Password recovery for user {email}" - link = f"{settings.server_host}/reset-password?token={token}" + link = f"{settings.FRONTEND_HOST}/reset-password?token={token}" html_content = render_email_template( template_name="reset_password.html", context={ @@ -90,7 +90,7 @@ def generate_new_account_email( "username": username, "password": password, "email": email_to, - "link": settings.server_host, + "link": settings.FRONTEND_HOST, }, ) return EmailData(html_content=html_content, subject=subject) diff --git a/deployment.md b/deployment.md index 6bcbe40..9301f80 100644 --- a/deployment.md +++ b/deployment.md @@ -12,7 +12,7 @@ But you have to configure a couple things first. 🤓 * Have a remote server ready and available. * Configure the DNS records of your domain to point to the IP of the server you just created. -* Configure a wildcard subdomain for your domain, so that you can have multiple subdomains for different services, e.g. `*.fastapi-project.example.com`. This will be useful for accessing different components, like `traefik.fastapi-project.example.com`, `adminer.fastapi-project.example.com`, etc. And also for `staging`, like `staging.fastapi-project.example.com`, `staging.adminer.fastapi-project.example.com`, etc. +* Configure a wildcard subdomain for your domain, so that you can have multiple subdomains for different services, e.g. `*.fastapi-project.example.com`. This will be useful for accessing different components, like `dashboard.fastapi-project.example.com`, `api.fastapi-project.example.com`, `traefik.fastapi-project.example.com`, `adminer.fastapi-project.example.com`, etc. And also for `staging`, like `dashboard.staging.fastapi-project.example.com`, `adminer.staging..fastapi-project.example.com`, etc. * Install and configure [Docker](https://docs.docker.com/engine/install/) on the remote server (Docker Engine, not Docker Desktop). ## Public Traefik @@ -284,20 +284,20 @@ Traefik UI: `https://traefik.fastapi-project.example.com` ### Production -Frontend: `https://fastapi-project.example.com` +Frontend: `https://dashboard.fastapi-project.example.com` -Backend API docs: `https://fastapi-project.example.com/docs` +Backend API docs: `https://api.fastapi-project.example.com/docs` -Backend API base URL: `https://fastapi-project.example.com/api/` +Backend API base URL: `https://api.fastapi-project.example.com` Adminer: `https://adminer.fastapi-project.example.com` ### Staging -Frontend: `https://staging.fastapi-project.example.com` +Frontend: `https://dashboard.staging.fastapi-project.example.com` -Backend API docs: `https://staging.fastapi-project.example.com/docs` +Backend API docs: `https://api.staging.fastapi-project.example.com/docs` -Backend API base URL: `https://staging.fastapi-project.example.com/api/` +Backend API base URL: `https://api.staging.fastapi-project.example.com` Adminer: `https://adminer.staging.fastapi-project.example.com` diff --git a/development.md b/development.md index 857a4e0..38d17eb 100644 --- a/development.md +++ b/development.md @@ -1,78 +1,100 @@ # FastAPI Project - Development -## Development in `localhost` with a custom domain +## Docker Compose -You might want to use something different than `localhost` as the domain. For example, if you are having problems with cookies that need a subdomain, and Chrome is not allowing you to use `localhost`. +* Start the local stack with Docker Compose: -In that case, you have two options: you could use the instructions to modify your system `hosts` file with the instructions below in **Development with a custom IP** or you can just use `localhost.tiangolo.com`, it is set up to point to `localhost` (to the IP `127.0.0.1`) and all its subdomains too. And as it is an actual domain, the browsers will store the cookies you set during development, etc. - -If you used the default CORS enabled domains while generating the project, `localhost.tiangolo.com` was configured to be allowed. If you didn't, you will need to add it to the list in the variable `BACKEND_CORS_ORIGINS` in the `.env` file. - -To configure it in your stack, follow the section **Change the development "domain"** below, using the domain `localhost.tiangolo.com`. +```bash +docker compose up -d +``` -After performing those steps you should be able to open: http://localhost.tiangolo.com and it will be served by your stack in `localhost`. +* Now you can open your browser and interact with these URLs: -Check all the corresponding available URLs in the section at the end. +Frontend, built with Docker, with routes handled based on the path: http://localhost:5173 -## Development with a custom IP +Backend, JSON based web API based on OpenAPI: http://localhost:8000 -If you are running Docker in an IP address different than `127.0.0.1` (`localhost`), you will need to perform some additional steps. That will be the case if you are running a custom Virtual Machine or your Docker is located in a different machine in your network. +Automatic interactive documentation with Swagger UI (from the OpenAPI backend): http://localhost:8000/docs -In that case, you will need to use a fake local domain (`dev.example.com`) and make your computer think that the domain is served by the custom IP (e.g. `192.168.99.150`). +Adminer, database web administration: http://localhost:8080 -If you have a custom domain like that, you need to add it to the list in the variable `BACKEND_CORS_ORIGINS` in the `.env` file. +Traefik UI, to see how the routes are being handled by the proxy: http://localhost:8090 -* Open your `hosts` file with administrative privileges using a text editor: +**Note**: The first time you start your stack, it might take a minute for it to be ready. While the backend waits for the database to be ready and configures everything. You can check the logs to monitor it. - * **Note for Windows**: If you are in Windows, open the main Windows menu, search for "notepad", right click on it, and select the option "open as Administrator" or similar. Then click the "File" menu, "Open file", go to the directory `c:\Windows\System32\Drivers\etc\`, select the option to show "All files" instead of only "Text (.txt) files", and open the `hosts` file. - * **Note for Mac and Linux**: Your `hosts` file is probably located at `/etc/hosts`, you can edit it in a terminal running `sudo nano /etc/hosts`. +To check the logs, run: -* Additional to the contents it might have, add a new line with the custom IP (e.g. `192.168.99.150`) a space character, and your fake local domain: `dev.example.com`. +```bash +docker compose logs +``` -The new line might look like: +To check the logs of a specific service, add the name of the service, e.g.: -``` -192.168.99.150 dev.example.com +```bash +docker compose logs backend ``` -* Save the file. - * **Note for Windows**: Make sure you save the file as "All files", without an extension of `.txt`. By default, Windows tries to add the extension. Make sure the file is saved as is, without extension. +## Local Development -...that will make your computer think that the fake local domain is served by that custom IP, and when you open that URL in your browser, it will talk directly to your locally running server when it is asked to go to `dev.example.com` and think that it is a remote server while it is actually running in your computer. +The Docker Compose files are configured so that each of the services is available in a different port in `localhost`. -To configure it in your stack, follow the section **Change the development "domain"** below, using the domain `dev.example.com`. +For the backend and frontend, they use the same port that would be used by their local development server, so, the backend is at `http://localhost:8000` and the frontend at `http://localhost:5173`. -After performing those steps you should be able to open: http://dev.example.com and it will be server by your stack in `192.168.99.150`. +This way, you could turn off a Docker Compose service and start its local development service, and everything would keep working, because it all uses the same ports. -Check all the corresponding available URLs in the section at the end. +For example, you can stop that `frontend` service in the Docker Compose: -## Change the development "domain" +```bash +docker compose stop frontend +``` -If you need to use your local stack with a different domain than `localhost`, you need to make sure the domain you use points to the IP where your stack is set up. +And then start the local frontend development server: -To simplify your Docker Compose setup, for example, so that the API docs (Swagger UI) knows where is your API, you should let it know you are using that domain for development. +```bash +cd frontend +npm run dev +``` -* Open the file located at `./.env`. It would have a line like: +Or you could stop the `backend` Docker Compose service: -``` -DOMAIN=localhost +```bash +docker compose stop backend ``` -* Change it to the domain you are going to use, e.g.: +And then you can run the local development server for the backend: +```bash +cd backend +fastapi dev app/main.py ``` + +## Docker Compose in `localhost.tiangolo.com` + +When you start the Docker Compose stack, it uses `localhost` by default, with different ports for each service (backend, frontend, adminer, etc). + +When you deploy it to production (or staging), it will deploy each service in a different subdomain, like `api.example.com` for the backend and `dashboard.example.com` for the frontend. + +In the guide about [deployment](deployment.md) you can read about Traefik, the configured proxy. That's the component in charge of transmitting traffic to each service based on the subdomain. + +If you want to test that it's all working locally, you can edit the local `.env` file, and change: + +```dotenv DOMAIN=localhost.tiangolo.com ``` -That variable will be used by the Docker Compose files. +That will be used by the Docker Compose files to configure the base domain for the services. -After that, you can restart your stack with: +Traefik will use this to transmit traffic at `api.localhost.tiangolo.com` to the backend, and traffic at `dashboard.localhost.tiangolo.com` to the frontend. + +The domain `localhost.tiangolo.com` is a special domain that is configured (with all its subdomains) to point to `127.0.0.1`. This way you can use that for your local development. + +After you update it, run again: ```bash docker compose up -d ``` -and check all the corresponding available URLs in the section at the end. +When deploying, for example in production, the main Traefik is configured outside of the Docker Compose files. For local development, there's an included Traefik in `docker-compose.override.yml`, just to let you test that the domains work as expected, for example with `api.localhost.tiangolo.com` and `dashboard.localhost.tiangolo.com`. ## Docker Compose files and env vars @@ -84,6 +106,12 @@ These Docker Compose files use the `.env` file containing configurations to be i They also use some additional configurations taken from environment variables set in the scripts before calling the `docker compose` command. +After changing variables, make sure you restart the stack: + +```bash +docker compose up -d +``` + ## The .env file The `.env` file is the one that contains all your configurations, generated keys and passwords, etc. @@ -92,7 +120,7 @@ Depending on your workflow, you could want to exclude it from Git, for example i One way to do it could be to add each environment variable to your CI/CD system, and updating the `docker-compose.yml` file to read that specific env var instead of reading the `.env` file. -### Pre-commits and code linting +## Pre-commits and code linting we are using a tool called [pre-commit](https://pre-commit.com/) for code linting and formatting. @@ -146,29 +174,29 @@ The production or staging URLs would use these same paths, but with your own dom Development URLs, for local development. -Frontend: http://localhost +Frontend: http://localhost:5173 -Backend: http://localhost/api/ +Backend: http://localhost:8000 -Automatic Interactive Docs (Swagger UI): http://localhost/docs +Automatic Interactive Docs (Swagger UI): http://localhost:8000/docs -Automatic Alternative Docs (ReDoc): http://localhost/redoc +Automatic Alternative Docs (ReDoc): http://localhost:8000/redoc Adminer: http://localhost:8080 Traefik UI: http://localhost:8090 -### Development in localhost with a custom domain URLs +### Development URLs with `localhost.tiangolo.com` Configured Development URLs, for local development. -Frontend: http://localhost.tiangolo.com +Frontend: http://dashboard.localhost.tiangolo.com -Backend: http://localhost.tiangolo.com/api/ +Backend: http://api.localhost.tiangolo.com -Automatic Interactive Docs (Swagger UI): http://localhost.tiangolo.com/docs +Automatic Interactive Docs (Swagger UI): http://api.localhost.tiangolo.comdocs -Automatic Alternative Docs (ReDoc): http://localhost.tiangolo.com/redoc +Automatic Alternative Docs (ReDoc): http://api.localhost.tiangolo.comredoc Adminer: http://localhost.tiangolo.com:8080 diff --git a/docker-compose.override.yml b/docker-compose.override.yml index 418b535..55965e4 100644 --- a/docker-compose.override.yml +++ b/docker-compose.override.yml @@ -1,5 +1,10 @@ services: + # Local services are available on their ports, but also available on: + # http://api.localhost.tiangolo.com: backend + # http://dashboard.localhost.tiangolo.com: frontend + # etc. To enable it, update .env, set: + # DOMAIN=localhost.tiangolo.com proxy: image: traefik:3.0 volumes: @@ -54,6 +59,7 @@ services: restart: "no" ports: - "8888:8888" + - "8000:80" volumes: - ./backend/:/app build: @@ -76,10 +82,12 @@ services: frontend: restart: "no" + ports: + - "5173:80" build: context: ./frontend args: - - VITE_API_URL=http://${DOMAIN?Variable not set} + - VITE_API_URL=http://localhost:8000 - NODE_ENV=development networks: diff --git a/docker-compose.yml b/docker-compose.yml index 93bfadb..d655191 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -1,4 +1,5 @@ services: + db: image: postgres:12 restart: always @@ -55,6 +56,7 @@ services: - .env environment: - DOMAIN=${DOMAIN} + - FRONTEND_HOST=${FRONTEND_HOST?Variable not set} - ENVIRONMENT=${ENVIRONMENT} - BACKEND_CORS_ORIGINS=${BACKEND_CORS_ORIGINS} - SECRET_KEY=${SECRET_KEY?Variable not set} @@ -88,10 +90,10 @@ services: - traefik.http.services.${STACK_NAME?Variable not set}-backend.loadbalancer.server.port=80 - - traefik.http.routers.${STACK_NAME?Variable not set}-backend-http.rule=Host(`${DOMAIN?Variable not set}`) && (PathPrefix(`/api`) || PathPrefix(`/docs`) || PathPrefix(`/redoc`)) + - traefik.http.routers.${STACK_NAME?Variable not set}-backend-http.rule=Host(`api.${DOMAIN?Variable not set}`) - traefik.http.routers.${STACK_NAME?Variable not set}-backend-http.entrypoints=http - - traefik.http.routers.${STACK_NAME?Variable not set}-backend-https.rule=Host(`${DOMAIN?Variable not set}`) && (PathPrefix(`/api`) || PathPrefix(`/docs`) || PathPrefix(`/redoc`)) + - traefik.http.routers.${STACK_NAME?Variable not set}-backend-https.rule=Host(`api.${DOMAIN?Variable not set}`) - traefik.http.routers.${STACK_NAME?Variable not set}-backend-https.entrypoints=https - traefik.http.routers.${STACK_NAME?Variable not set}-backend-https.tls=true - traefik.http.routers.${STACK_NAME?Variable not set}-backend-https.tls.certresolver=le @@ -108,7 +110,7 @@ services: build: context: ./frontend args: - - VITE_API_URL=https://${DOMAIN?Variable not set} + - VITE_API_URL=https://api.${DOMAIN?Variable not set} - NODE_ENV=production labels: - traefik.enable=true @@ -117,10 +119,10 @@ services: - traefik.http.services.${STACK_NAME?Variable not set}-frontend.loadbalancer.server.port=80 - - traefik.http.routers.${STACK_NAME?Variable not set}-frontend-http.rule=Host(`${DOMAIN?Variable not set}`) + - traefik.http.routers.${STACK_NAME?Variable not set}-frontend-http.rule=Host(`dashboard.${DOMAIN?Variable not set}`) - traefik.http.routers.${STACK_NAME?Variable not set}-frontend-http.entrypoints=http - - traefik.http.routers.${STACK_NAME?Variable not set}-frontend-https.rule=Host(`${DOMAIN?Variable not set}`) + - traefik.http.routers.${STACK_NAME?Variable not set}-frontend-https.rule=Host(`dashboard.${DOMAIN?Variable not set}`) - traefik.http.routers.${STACK_NAME?Variable not set}-frontend-https.entrypoints=https - traefik.http.routers.${STACK_NAME?Variable not set}-frontend-https.tls=true - traefik.http.routers.${STACK_NAME?Variable not set}-frontend-https.tls.certresolver=le diff --git a/frontend/.env b/frontend/.env index f829bd1..5934e2e 100644 --- a/frontend/.env +++ b/frontend/.env @@ -1 +1 @@ -VITE_API_URL=http://localhost +VITE_API_URL=http://localhost:8000 diff --git a/frontend/README.md b/frontend/README.md index 9a01970..fde8267 100644 --- a/frontend/README.md +++ b/frontend/README.md @@ -112,7 +112,7 @@ Notice that everytime the backend changes (changing the OpenAPI schema), you sho If you want to use a remote API, you can set the environment variable `VITE_API_URL` to the URL of the remote API. For example, you can set it in the `frontend/.env` file: ```env -VITE_API_URL=https://my-remote-api.example.com +VITE_API_URL=https://api.my-domain.example.com ``` Then, when you run the frontend, it will use that URL as the base URL for the API. @@ -134,7 +134,7 @@ The frontend code is structured as follows: The frontend includes initial end-to-end tests using Playwright. To run the tests, you need to have the Docker Compose stack running. Start the stack with the following command: ```bash -docker compose up -d +docker compose up -d --wait backend ``` Then, you can run the tests with the following command: @@ -157,4 +157,4 @@ docker compose down -v To update the tests, navigate to the tests directory and modify the existing test files or add new ones as needed. -For more information on writing and running Playwright tests, refer to the official [Playwright documentation](https://playwright.dev/docs/intro). \ No newline at end of file +For more information on writing and running Playwright tests, refer to the official [Playwright documentation](https://playwright.dev/docs/intro).