Messy lifeapi dev notes

10.1.1 step by step deployment

1. ssh to the GCP server
   

   First - ssh-into-gcp-server

   Second(optional) - login-to-gcp-server-with-username-and-password

2. winscp to GCP server
   

   add private key - connect

   private key ends in .pem? store that private key in such location, in wsl for example:

   \\wsl.localhost\Ubuntu\home\arvy\arvydas_privatekey_gcp.pem

   the file must remain there, otherwise the connection will not work

   https://winscp.net/eng/docs/guide_google_compute_engine

3. setup virtual env/server modifications
   

   sudo su - apt update apt upgrade apt install git mkdir /opt/app cd /opt/app git clone https://github.com/arvydasg/lifeapi.git apt install python3-venv python3-pip python3-dev libpq-dev postgresql postgresql-contrib nginx curl

   inspiration for deployment - - https://www.youtube.com/watch?v=bD75adrlkes&ab_channel=TheCodrammers

   python3 -m venv venv source /venv/bin/activate

   pradedu ties situ commit - 7904e80430b7b0f54c869c3e4366b0eb02d2e2c9 (Merge pull request #4 from arvydasg/DEV …) - https://github.com/arvydasg/lifeapi/commit/7904e80430b7b0f54c869c3e4366b0eb02d2e2c9

   python manage.py migrate –settings=settings.production python manage.py createsuperuser –settings=settings.production collectstatic

   python manage.py collectstatic –settings=settings.production then git ignore collectstatic dir which is "staticfiles-cdn/"

   git pull origin master # on the server open 8000 port in server

   apt install ufw ufw allow 8000 allow 8000 on gcp

   3 dots on server → view network details → default(under network) → firewalls → add firewall rule

   how it looks before:

   pictures how some firewall configuration was done in gcp

   cd /opt/app/lifeapi touch .env_prod vim .env_prod

   ALLOWED_HOSTS=34.88.9.23 python manage.py runserver –settings=settings.production 0.0.0.0:8000

4. Okay why I am spending so much time on gunicorn and nginx
   

   I was like dude if I can just run django in the terminal.. why bother.

   same like with the db. Why bother with postgresql if you can use sqlite and why bother with nginx/gunicorn if you can use simle process in systemd or just run in the terminal.

   Seems like there are some benefits… but still thinking about the simple solution. Its MVP after all dude!!!!

   Using Gunicorn and Nginx in combination with Django offers several benefits over running Django in the terminal. Here are some advantages:

   Improved performance and scalability: Gunicorn is a production-ready web server that can handle multiple concurrent requests efficiently, improving the overall performance of your Django application. It utilizes a pre-fork worker model and supports multiple worker processes, allowing it to handle high traffic loads. Nginx acts as a reverse proxy server, offloading some of the processing tasks from Gunicorn and efficiently serving static files, resulting in better scalability.

   Load balancing and high availability: By using Gunicorn and Nginx together, you can set up multiple Gunicorn worker processes and distribute incoming requests among them. Nginx can act as a load balancer, evenly distributing the requests across the available Gunicorn workers. This setup improves your application's availability and can handle more traffic without overwhelming a single server instance.

   Handling concurrent connections: Gunicorn's ability to handle multiple concurrent connections makes it more suitable for production environments compared to running Django in the terminal. It allows your application to serve multiple requests simultaneously, providing a better experience for your users.

   Separation of concerns: Nginx and Gunicorn serve different purposes in a typical Django deployment setup. Nginx specializes in serving static files, handling SSL termination, and acting as a reverse proxy server. On the other hand, Gunicorn focuses on running the Django application and managing the dynamic content. This separation allows each component to perform its task efficiently and enables easier troubleshooting and maintenance.

   Regarding the issue with displaying the CSS of the admin panel when using Nginx and Gunicorn, this problem is typically related to the configuration of Nginx. You need to ensure that Nginx is configured to correctly serve static files and that the appropriate static file directories are specified. By properly configuring Nginx, you should be able to resolve this issue and have the admin panel CSS displayed correctly.

   Gunicorn Translates the client requests from http to our python calls that our application can process

   webserver gateway interface module - wsgi

   pip install gunicorn (ADD THIS TO REQUIREMENTS.TXT)

   gunicorn –bind 0.0.0.0:8000 project_name.wsgi

   gunicorn –env DJANGO_SETTINGS_MODULE=settings.production –bind 0.0.0.0:8000 lifeapi_project.wsgi

   gunicorn.socket

   vim /etc/systemd/system/gunicorn.socket

   [Unit] Description=gunicorn.socket

   [Socket] ListenStream=/run/gunicorn.sock

   [Install] WantedBy=sockets.target gunicorn.service #file name should math. Can see that both are gunicorn.

   vim /etc/systemd/system/gunicorn.service

   [Unit] Description=gunicorn daemon Requires=gunicorn.socket After=network.target

   [Service] User=root Group=www-data WorkingDirectory=/opt/app/lifeapi ExecStart=/opt/app/lifeapi/venv/bin/gunicorn \ –access-logfile - \ –workers 3 \ –bind unix:/run/gunicorn.sock \ lifeapi_project.wsgi:application \ –env DJANGO_SETTINGS_MODULE=settings.production

   [Install] WantedBy=multi-user.target systemctl start gunicorn.socket

   systemctl enable gunicorn.socket

   When a connection is made to gunicorn.socket systemd will automatically will start systemd.service to handle it.

   systemctl status gunicorn.socket - should be active

   systemctl status gunicorn.service- should be inactive (dead), since it did nto receive any connections

   test activation mechanism:

   curl –unix-socket /run/gunicorn.sock localhost

   Should see some html outputted. For the dude in the video it was full html, for me some html and 400 errors.

   systemctl status gunicorn.service - now should be active

   nginx vim /etc/nginx/sites-available/lifeapi

   server { listen 80; server_name 34.88.9.236; location = favicon.ico { access_log off; log_not_found off; } location /static { autoindex on; alias opt/app/lifeapi/staticfiles-cdn; } location / { include proxy_params; proxy_pass http://unix/run/gunicorn.sock; }

   } sudo ln -s etc/nginx/sites-available/lifeapi /etc/nginx/sites-enabled

   check any syntax errors:

   nginx -t

   systemctl restart nginx

   open firewall for normal traffic on the port 80

   ufw allow ‘Nginx Full’

   We no longer need access to the development server (8000), so we can remove this firewall rule:

   ufw delete allow 8000

   go to the 34.88.9.236 without any ports. Should work.

5. Configuring static files
   

   this video inspiration - 74 - Static Files in Development - Python & Django 3.2 Tutorial Series

   global styles are better if wanting to override all the app styles in one place. better for the future

   can override admin styles also and other django stylesheets

   remember to turn off caching in network panel browser

   debug on - takes from static folder debug off - collecstatic need

   in prod then do:

   python manage.py collectstatic --settings=settings.production
   # 126 static files copied to '/home/arvy/src/lifeapi/staticfiles-cdn'.
   

   Ok we have stored mine and django’s static files in one location, but they are not used by anything now. When debug is true - django takes the static files from usual places, where we set them.

   But if we are in production and we ant them to be server, we need to do that with whitenoise.

   pip install whitenoise
   

   add middleware like in WhiteNoise 6.5.0 documentation . After the security one. At the top.

   MIDDLEWARE = [
       # ...
       "django.middleware.security.SecurityMiddleware",
       "whitenoise.middleware.WhiteNoiseMiddleware",
       # ...
   ]
   

   pip freeze > requirements.txt
   

   Run the server again. Even with debug = false static files for admin panel should be displayed, as well as your global styles.

6. Running the server
   

   python manage.py runserver --settings=settings.production 0.0.0.0:8000
   python manage.py runserver --settings=settings.production
   python manage.py collectstatic --settings=settings.production
   

   in dev:

   • change branch to dev
   • create .env_dev file next to manage.py

   • add this:

     ALLOWED_HOSTS=127.0.0.1,localhost
     DEBUG=1
     

   • python manage.py makemigrations –settings=settings.dev
   • python manage.py migrate –settings=settings.dev
   • python manage.py createsuperuser –settings=settings.dev (root pass)
   • python manage.py collectstatic –settings=settings.dev
   • python manage.py runserver –settings=settings.dev

   or simply reuse the prod DB

7. Restarting the services
   

   Link to the instructions - Restarting the services

8. Source venv on each login to root
   

   sudo su -
   vim ~/.bashrc
   # add the following line
   source /opt/app/lifeapi/venv/bin/activate
   :wq
   
   source ~/.bashrc
   

9. Cron job to fetch data automatically
   

   set the correct server time

   cd /opt/app/lifeapi
   chmod +x weather_job_prod.py
   crontab -e
   0 17 * * * /opt/app/lifeapi/venv/bin/python /opt/app/lifeapi/weather_job_prod.py >> /opt/app/lifeapi/weather_job_prod.log 2>&1
   

   This cron job now should run every day at 17:00.

10.1.2 deployment to gcp

10.2.1 The plan

10.2.2 Creating a new GCP instance

1. Code for creating instance
   

   Can look at this code and know what settings I have chosen.

   gcloud compute instances create instance-2 \
       --project=lifeapi-392202 \
       --zone=us-central1-a \
       --machine-type=e2-micro \
       --network-interface=network-tier=PREMIUM,stack-type=IPV4_ONLY,subnet=default \
       --maintenance-policy=MIGRATE \
       --provisioning-model=STANDARD \
       --service-account=46844210845-compute@developer.gserviceaccount.com \
       --scopes=https://www.googleapis.com/auth/devstorage.read_only,https://www.googleapis.com/auth/logging.write,https://www.googleapis.com/auth/monitoring.write,https://www.googleapis.com/auth/servicecontrol,https://www.googleapis.com/auth/service.management.readonly,https://www.googleapis.com/auth/trace.append \
       --tags=http-server,https-server,lb-health-check \
       --create-disk=auto-delete=yes,boot=yes,device-name=instance-2,image=projects/debian-cloud/global/images/debian-11-bullseye-v20231212,mode=rw,size=10,type=projects/lifeapi-392202/zones/us-central1-a/diskTypes/pd-standard \
       --no-shielded-secure-boot \
       --shielded-vtpm \
       --shielded-integrity-monitoring \
       --labels=goog-ec-src=vm_add-gcloud \
       --reservation-affinity=any
   

2. Pricing
   

   Basically it's a e2-micro instance

   Monthly estimate

   US$6.51 That's about US$0.01 hourly

   Pay for what you use: No upfront costs and per-second billing

   Item Monthly estimate 2 vCPU + 1 GB memory US$6.11 10 GB standard persistent disk US$0.40 Total US$6.51

10.2.3 Connect to the machine over gcp console

10.2.4 Connect over ssh

1. ssh key info for instace2 (everything is no longer relevant here)
   

   fingerprint

   ssh-rsa 2048 SHA256:9gdAeVoX567uowDrkLCSOFVGlziSlKnSbpZDVLLdHdU

   key comment(username)

   arvydas

   public_key

   ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCufZnGwCFbJyYGJtRULgEBf1qDzb8jsCFoArSd8AkCUr9mAsEbHwvuxPKQ84w3BiIeizS/Y7w168ZHcaUheWAG/pSPYxhgGU6QknW5PAVVCHPIxiXb6p6LhUbrluahcE5Cq5+HYhFZXQQ+pftI7fG/OGvBAG6f9FoxuxFaUJMhvKrQ1DG1j65I6U1UMfWVdOtyqZTw0qMpXTMjgd3rKy5bTyxEqIXH+9PL6pmSCckQZOPXXeouAqQMBRQuoxKpsG28ctTgKbjDOMoyytW9YmAAHFzxoP9tLViQMHkScWBtPKuud95IyTEdgNbwjnjUD8SCfNbjBGFITFOdURgVQwgx arvydas

   private_key

   PuTTY-User-Key-File-3: ssh-rsa Encryption: none Comment: arvydas Public-Lines: 6 AAAAB3NzaC1yc2EAAAADAQABAAABAQCufZnGwCFbJyYGJtRULgEBf1qDzb8jsCFo ArSd8AkCUr9mAsEbHwvuxPKQ84w3BiIeizS/Y7w168ZHcaUheWAG/pSPYxhgGU6Q knW5PAVVCHPIxiXb6p6LhUbrluahcE5Cq5+HYhFZXQQ+pftI7fG/OGvBAG6f9Fox uxFaUJMhvKrQ1DG1j65I6U1UMfWVdOtyqZTw0qMpXTMjgd3rKy5bTyxEqIXH+9PL 6pmSCckQZOPXXeouAqQMBRQuoxKpsG28ctTgKbjDOMoyytW9YmAAHFzxoP9tLViQ MHkScWBtPKuud95IyTEdgNbwjnjUD8SCfNbjBGFITFOdURgVQwgx Private-Lines: 14 AAABAFUBPUP034sflEeU7QWhb74CA9+IASDqsiuQfdsfT9RA6ZtRpi+HPXHxolX5 QAqiQ0br/CNs/AistuihNZgMIDroFQmRdhOC4KJPp2g5FEPrnTRnS5RKRTilEfq9 hdeJ9aZHI615mggV53Z5t+Q8fvPwEZZxlnL4QGRPxNFhxXu+NQWItmZ2Omkg3st7 8oJhTgi1WuYvVnZcIzeNJhITf1pYdrOat4tFAL7J4QjMVIFa+KpKuK3Fa4DKY0XD r+KAZuigU3vFdQjdMRvi27oJqb1qnyL1G1Z2iR3cjYWFM3KFT7mewQ7X+FERDhBp APjQ9ac4NieojYoQNtrvL00uHX0AAACBAPIz8KqgrUW1OEbr1XH9PrTdmxPCgZEp mAUfETiqYfDvhBKGzJNxekS9nHSUxjVQXWWDiXJ6KePCPu0Y1H/oKcX4ntxin5Wk 8CljbfbPHR+dvFC3jV0iGv8Gdctq8bp+eEp4ZCTU3YDWgO5tjeup3TarpooPFv2Y LwMZprbNZoyvAAAAgQC4bjVa1+BPCSpNZAiYOyUbXCv5mGWTbstyB+7dTqEOjCHy QEHdrtK8YoxmYY3ItJdjuoqKB0Souo9pSn7Vi/xN75DfapnSkF/JA1qIV5BHyuQG P6OyPLAY08PGqapxKqI0LEyFQ9WxcJBAoxUyCdQDvysO979Snr2oLVcdvV2xHwAA AIEA1tfFjfPQCHPzW40hhB/SA6l96lVRKykAtNQoDwNz7pv6Arg662yY6A8+RHRJ FuWeBwlmg/BHgAW40RSOYIkw95CFXqOF7YxH7XgnR1RDcMwyJKoJ71lJLKFOOJze RVLnm1zeO/BSSS2lKXEGEo6weH5ur4sNLYLi1aBKPTYAJMw= Private-MAC: 89ee5eb4d7113171bf86a39f453364f288e910bac34081acf1b4c117bf01a8f3

   cronjobs

10.8.1 Check services status

systemctl status gunicorn.socket
systemctl status gunicorn.service
systemctl status nginx

10.8.2 Fetch changes

10.11.1 First created the models:

from django.db import models
from django.utils import timezone

# Create your models here.

class Question(models.Model):
    description = models.TextField()

class Answer(models.Model):
    question = models.ForeignKey(Question, on_delete=models.CASCADE)
    answer = models.CharField(max_length=255, blank=False)
    date_added = models.DateTimeField(default=timezone.now)

10.11.2 Then the views:

from django.shortcuts import render, redirect
from .models import Question, Answer
from django.http import HttpResponse
from django.utils import timezone
from datetime import date
from django.contrib import messages
from .forms import QuestionForm


def quiz_app_home(request):
    return render(request, 'quiz_app_home.html')


def quiz_questions(request):
    '''View to display all the questions'''
    questions = Question.objects.all()
    context = {'questions': questions}
    return render(request, 'quiz_app_questions.html', context)


def quiz_add_question(request):
    '''View to add a posibility to add new questions'''
    if request.method == 'POST':
form = QuestionForm(request.POST)
if form.is_valid():
    form.save()
    return redirect('quiz_questions')
    else:
form = QuestionForm()

    context = {'form': form}
    return render(request, 'quiz_app_add_question.html', context)


def quiz_start(request):
    '''View to add a posibility to add new questions'''
    if request.method == 'POST':
# Check if the user clicked the "Start Quiz" button
if 'start_quiz' in request.POST:
    # Check if entries for today already exist
    today = date.today()
    if Answer.objects.filter(date_added__date=today).exists():
messages.warning(request, "You have already answered the quiz for today.")
    else:
# Retrieve the first question from the database
first_question = Question.objects.first()
context = {'question': first_question}

return render(request, 'quiz_app_question.html', context)

# Check if the user clicked the "Delete Answers" button
elif 'delete_answers' in request.POST:
    today = date.today()
    Answer.objects.filter(date_added__date=today).delete()
    messages.success(request, "Your answers for today have been deleted.")

    # Retrieve any flash messages and pass them to the template context
    messages_to_display = messages.get_messages(request)
    context = {'messages': messages_to_display}

    return render(request, 'quiz_app_ready.html', context)


def quiz_question(request, question_id):
    if request.method == 'POST':
question_id = int(request.POST.get('question_id'))
answer_text = request.POST.get('answer')

# Save the answer to the database
answer = Answer(question_id=question_id, answer=answer_text)
answer.save()

# Redirect to the next question or finish the quiz if all questions are answered
next_question_id = question_id + 1
if next_question_id > Question.objects.count():
    return redirect('quiz_summary')  # Redirect to the quiz summary page
else:
    question = Question.objects.get(id=next_question_id)
    context = {'question': question}
    return render(request, 'quiz_app_question.html', context)

    # Retrieve the question based on the question_id
    question = Question.objects.get(id=question_id)
    context = {'question': question}
    return render(request, 'quiz_app_question.html', context)


def quiz_summary(request):
    '''View to display all entries of answers table'''
    answers = Answer.objects.all()
    context = {'answers': answers}
    return render(request, 'quiz_app_summary.html', context)

10.11.3 Then the urls:

from django.urls import path
from . import views

urlpatterns = [
    path("", views.quiz_app_home, name="quiz_app_home"),
    path('start_quiz/', views.quiz_start, name='quiz_start'),
    path("quiz/questions/", views.quiz_questions, name="quiz_questions"),
    path('quiz/<int:question_id>/', views.quiz_question, name='quiz_question'),
    path('quiz/summary/', views.quiz_summary, name='quiz_summary'),
    path('add-question/', views.quiz_add_question, name='quiz_add_question'),
]

10.11.4 And forms:

from django import forms
from .models import Question

class QuestionForm(forms.ModelForm):
    class Meta:
model = Question
fields = ['description']

10.12.1 quiz_app_add_question.html

{% extends 'base.html' %}

{% block content %}
    <h2>Add Question</h2>
    <form method="POST">
{% csrf_token %}
{{ form.as_p }}
<button type="submit">Add</button>
    </form>
{% endblock %}

10.12.2 quiz_app_home.html

{% extends 'base.html' %}

{% block content %}
    <h1>Welcome to My QUIZ APP</h1>
    <p>Here you can answer some questions bla.</p>
    <p>Quiz is here - <a href="{% url 'quiz_start' %}">Here</a></p>
    <p>all questions can see - <a href="{% url 'quiz_questions' %}">Here</a></p>
{% endblock %}

10.12.3 quiz_app_question.html

{% extends 'base.html' %}

{% block content %}
    <h2>Question:</h2>
    <p>{{ question.description }}</p>
    <form action="{% url 'quiz_question' question_id=question.id %}" method="POST">
{% csrf_token %}
<input type="hidden" name="question_id" value="{{ question.id }}">
<input type="text" name="answer" placeholder="Enter your answer" required>
<button type="submit">Submit</button>
    </form>
{% endblock %}

10.12.4 quiz_app_questions.html

{% extends 'base.html' %}

{% block content %}
    <h1>Questions</h1>
    <table class="table">
<thead>
    <tr>
<th>ID</th>
<th>Description</th>
    </tr>
</thead>
<tbody>
    {% for question in questions %}
<tr>
    <td>{{ question.id }}</td>
    <td>{{ question.description }}</td>
</tr>
    {% endfor %}
</tbody>
    </table>

    <a href="{% url 'quiz_add_question' %}" class="btn btn-primary">Add Question</a>
{% endblock %}

10.12.5 quiz_app_ready.html

{% extends 'base.html' %}

{% block content %}
    <h2>Ready to start the quiz?</h2>
    <form action="{% url 'quiz_start' %}" method="POST">
{% csrf_token %}
<button type="submit" name="start_quiz">Start Quiz</button>
    </form>

    <form action="{% url 'quiz_start' %}" method="POST">
{% csrf_token %}
<button type="submit" name="delete_answers">Delete My Answers for Today</button>
    </form>

    {% for message in messages %}
<div class="flash-message">{{ message }}</div>
    {% endfor %}
{% endblock %}

10.12.6 quiz_app_summary.html

{% extends 'base.html' %}

{% block content %}
    <h2>Quiz Summary</h2>
    <table>
<thead>
    <tr>
<th>Question</th>
<th>Answer</th>
    </tr>
</thead>
<tbody>
    {% for answer in answers %}
    <tr>
<td>{{ answer.question.description }}</td>
<td>{{ answer.answer }}</td>
    </tr>
    {% endfor %}
</tbody>
    </table>
{% endblock %}

10.12.7 Quiz looks like this: