Merge pull request '# Geometry conflicts on API' (#544) from 534_Return_geometry_conflicts_on_API into master

Reviewed-on: #544

Dockerfile

@@ -1,36 +0,0 @@
-# Nutze ein schlankes Python-Image
-FROM python:3.13-slim-bullseye
-
-ENV PYTHONUNBUFFERED 1
-
-WORKDIR /konova
-
-# Installiere System-Abhängigkeiten
-RUN apt-get update && apt-get install -y --no-install-recommends \
-    gdal-bin redis-server nginx \
-    && rm -rf /var/lib/apt/lists/*  # Platz sparen
-
-# Erstelle benötigte Verzeichnisse & setze Berechtigungen
-RUN mkdir -p /var/log/nginx /var/log/gunicorn /var/lib/nginx /tmp/nginx_client_body \
-    && touch /var/log/nginx/access.log /var/log/nginx/error.log \
-    && chown -R root:root /var/log/nginx /var/lib/nginx /tmp/nginx_client_body
-
-# Kopiere und installiere Python-Abhängigkeiten
-COPY ./requirements.txt /konova/
-RUN pip install --upgrade pip && pip install --no-cache-dir -r requirements.txt
-
-# Entferne Standard-Nginx-Site und ersetze sie durch eigene Config
-RUN rm -rf /etc/nginx/sites-enabled/default
-COPY ./nginx.conf /etc/nginx/conf.d
-
-# Kopiere restliche Projektdateien
-COPY . /konova/
-
-# Sammle statische Dateien
-RUN python manage.py collectstatic --noinput
-
-# Exponiere Ports
-#EXPOSE 80 6379 8000
-
-# Setze Entrypoint
-ENTRYPOINT ["/konova/docker-entrypoint.sh"]

README.md

@@ -4,7 +4,6 @@ the database postgresql and the css library bootstrap as well as the icon packag
 fontawesome for a modern look, following best practices from the industry.
 
 ## Background processes
-### !!! For non-docker run
 Konova uses celery for background processing. To start the worker you need to run
 ```shell
 $ celery -A konova worker -l INFO
@@ -19,58 +18,3 @@ Technical documention is provided in the projects git wiki.
 A user documentation is not available (and not needed, yet). 
 
 
-# Docker 
-To run the docker-compose as expected, you need to take the following steps:
-
-1. Create a database containing docker, using an appropriate Dockerfile, e.g. the following
-```
-version: '3.3'
-services:
-  postgis:
-    image: postgis/postgis
-    restart: always
-    container_name: postgis-docker
-    ports: 
-      - 5433:5432
-    volumes:
-      - db-volume:/var/lib/postgresql/data
-    environment:
-      - POSTGRES_PASSWORD=postgres
-      - POSTGRES_USER=postgres
-    networks:
-      - db-network-bridge
-
-networks:
-  db-network-bridge:
-    driver: "bridge"
-
-volumes:
-  db-volume:
-```
-This Dockerfile creates a Docker container running postgresql and postgis, creates the default superuser postgres, 
-creates a named volume for persisting the database and creates a new network bridge, which **must be used by any other
-container, which wants to write/read on this database**. 
-
-2. Make sure the name of the network bridge above matches the network in the konova docker-compose.yml  
-3. Get into the running postgis container (`docker exec -it postgis-docker bash`) and create new databases, users and so on. Make sure the database `konova` exists now!
-4. Replace all `CHANGE_ME_xy` values inside of konova/docker-compose.yml for your installation. Make sure the `SSO_HOST` holds the proper SSO host, e.g. for the arnova project `arnova.example.org` (Arnova must be installed and the webserver configured as well, of course)
-5. Take a look on konova/settings.py and konova/sub_settings/django_settings.py. Again: Replace all occurences of `CHANGE_ME` with proper values for your installation.
-   1. Make sure you have the proper host strings added to `ALLOWED_HOSTS` inside of django_settings.py.  
-6. Build and run the docker setup using `docker-compose build` and `docker-compose start` from the main directory of this project (where the docker-compose.yml lives)
-7. Run migrations! To do so, get into the konova service container (`docker exec -it konova-docker bash`) and run the needed commands (`python manage.py makemigrations LIST_OF_ALL_MIGRATABLE_APPS`, then `python manage.py migrate`)
-8. Run the setup command `python manage.py setup` and follow the instructions on the CLI
-9. To enable **SMTP** mail support, make sure your host machine (the one where the docker container run) has the postfix service configured properly. Make sure the `mynetworks` variable is xtended using the docker network bridge ip, created in the postgis container and used by the konova services.
-   1. **Hint**: You can find out this easily by trying to perform a test mail in the running konova web application (which will fail, of course). Then take a look to the latest entries in `/var/log/mail.log` on your host machine. The failed IP will be displayed there.
-   2. **Please note**: This installation guide is based on SMTP using postfix! 
-   3. Restart the postfix service on your host machine to reload the new configuration (`service postfix restart`)
-10. Finally, make sure your host machine webserver passes incoming requests properly to the docker nginx webserver of konova. A proper nginx config for the host machine may look like this:
-```
-server {
-    server_name konova.domain.org;
-
-    location / {
-        proxy_pass http://localhost:KONOVA_NGINX_DOCKER_PORT/;
-        proxy_set_header Host $host;
-    }
-}
-```

api/utils/serializer/serializer.py

@@ -202,3 +202,29 @@ class AbstractModelAPISerializer:
             obj (Intervention)
         """
         raise NotImplementedError("Must be implemented in subclasses")
+
+
+    def _geometry_conflicts_to_list(self, geometry) -> list:
+        """ Serializes geometry conflict ids into dict
+
+        Args:
+            geometry (Geometry): The geometry to fetch geometry conflicts from
+
+        Returns:
+            ids (list): Serialized geometry conflicts as dict objects inside a list
+        """
+        ids = []
+        conflict_geometries = geometry.get_conflict_geometries()
+        for geom in conflict_geometries:
+            try:
+                data = geom.get_data_objects(["identifier", "id"])[0]
+            except KeyError:
+                raise AssertionError(f"Geometry {geom.id} is not attached to any entries. Contact an admin!")
+            ids.append(
+                {
+                    "identifier": data["identifier"],
+                    "id": data["id"],
+                }
+            )
+
+        return ids

api/utils/serializer/v1/serializer.py

@@ -54,6 +54,7 @@ class AbstractModelAPISerializerV1(AbstractModelAPISerializer):
             "created_on": self._created_on_to_json(entry),
             "modified_on": self._modified_on_to_json(entry),
             "external_identifiers": ext_ids,
+            "geometry_conflicts": self._geometry_conflicts_to_list(entry.geometry)
         }
         self._extend_properties_data(entry)
         geo_json["properties"] = self.properties_data

docker-compose.yml

@@ -1,21 +0,0 @@
-services:
-  konova:
-    external_links:
-      - postgis:db
-      - arnova-nginx-server:arnova
-    build: .
-    image: "ksp/konova:x.y"
-    container_name: "konova-docker"
-    command: ./docker-entrypoint.sh
-    restart: always
-    volumes:
-      - /data/apps/konova/uploaded_files:/konova_uploaded_files
-    ports:
-      - "1337:80"
-
-# Instead of an own, new network, we need to connect to the existing one, which is provided by the postgis container
-# NOTE: THIS NETWORK MUST EXIST
-networks:
-  default:
-    name: postgis_nat_it_backend
-    external: true

docker-entrypoint.sh

@@ -1,27 +0,0 @@
-#!/bin/bash
-
-set -e  # Beende Skript bei Fehlern
-set -o pipefail  # Fehler in Pipelines nicht ignorieren
-
-# Starte Redis
-redis-server --daemonize yes
-
-# Starte Celery Worker im Hintergrund
-celery -A konova worker --loglevel=info &
-
-# Starte Nginx als Hintergrundprozess
-nginx -g "daemon off;" &
-
-# Setze Gunicorn Worker-Anzahl (Standard: (2*CPUs)+1)
-WORKERS=${GUNICORN_WORKERS:-$((2 * $(nproc) + 1))}
-
-# Stelle sicher, dass Logs existieren
-mkdir -p /var/log/gunicorn
-touch /var/log/gunicorn/access.log /var/log/gunicorn/error.log
-
-# Starte Gunicorn als Hauptprozess
-exec gunicorn --workers="$WORKERS" konova.wsgi:application \
-    --bind=0.0.0.0:8000 \
-    --access-logfile /var/log/gunicorn/access.log \
-    --error-logfile /var/log/gunicorn/error.log \
-    --access-logformat '%({x-real-ip}i)s via %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"'

intervention/tests/test_views.py

@@ -36,6 +36,7 @@ class InterventionViewTestCase(BaseViewTestCase):
         self.run_check_url = reverse("intervention:check", args=(self.intervention.id,))
         self.record_url = reverse("intervention:record", args=(self.intervention.id,))
         self.report_url = reverse("intervention:report", args=(self.intervention.id,))
+        self.compensation_remove_url = reverse("intervention:remove-compensation", args=(self.compensation.intervention.id, self.compensation.id))
 
         self.deduction.intervention = self.intervention
         self.deduction.save()
@@ -83,6 +84,7 @@ class InterventionViewTestCase(BaseViewTestCase):
             self.revocation_new_url: f"{login_redirect_base}{self.revocation_new_url}",
             self.revocation_edit_url: f"{login_redirect_base}{self.revocation_edit_url}",
             self.revocation_remove_url: f"{login_redirect_base}{self.revocation_remove_url}",
+            self.compensation_remove_url: f"{login_redirect_base}{self.compensation_remove_url}",
         }
 
         self.assert_url_success(client, success_urls)
@@ -124,6 +126,7 @@ class InterventionViewTestCase(BaseViewTestCase):
             self.deduction_new_url,
             self.deduction_edit_url,
             self.deduction_remove_url,
+            self.compensation_remove_url,
         ]
 
         self.assert_url_success(client, success_urls)
@@ -162,6 +165,7 @@ class InterventionViewTestCase(BaseViewTestCase):
             self.deduction_new_url,
             self.deduction_edit_url,
             self.deduction_remove_url,
+            self.compensation_remove_url,
         ]
         fail_urls = [
             self.run_check_url,
@@ -212,6 +216,7 @@ class InterventionViewTestCase(BaseViewTestCase):
             self.deduction_new_url,
             self.deduction_edit_url,
             self.deduction_remove_url,
+            self.compensation_remove_url,
         ]
         success_urls_redirect = {
             self.share_url: self.detail_url
@@ -258,6 +263,7 @@ class InterventionViewTestCase(BaseViewTestCase):
             self.deduction_new_url,
             self.deduction_edit_url,
             self.deduction_remove_url,
+            self.compensation_remove_url,
         ]
         success_urls_redirect = {
             self.share_url: self.detail_url
@@ -304,6 +310,7 @@ class InterventionViewTestCase(BaseViewTestCase):
             self.deduction_new_url,
             self.deduction_edit_url,
             self.deduction_remove_url,
+            self.compensation_remove_url,
         ]
         success_urls_redirect = {
             self.share_url: self.detail_url
@@ -350,6 +357,7 @@ class InterventionViewTestCase(BaseViewTestCase):
             self.deduction_new_url,
             self.deduction_edit_url,
             self.deduction_remove_url,
+            self.compensation_remove_url,
         ]
         success_urls_redirect = {
             self.share_url: self.detail_url
@@ -396,6 +404,7 @@ class InterventionViewTestCase(BaseViewTestCase):
             self.deduction_new_url,
             self.deduction_edit_url,
             self.deduction_remove_url,
+            self.compensation_remove_url,
         ]
         # Define urls where a redirect to a specific location is the proper response
         success_urls_redirect = {

intervention/views/compensation.py

@@ -14,7 +14,7 @@ from django.utils.decorators import method_decorator
 from django.views import View
 
 from intervention.models import Intervention
-from konova.decorators import shared_access_required
+from konova.decorators import shared_access_required, default_group_required
 from konova.forms.modals import RemoveModalForm
 from konova.utils.message_templates import COMPENSATION_REMOVED_TEMPLATE
 
@@ -45,10 +45,12 @@ class RemoveCompensationFromInterventionView(LoginRequiredMixin, View):
             redirect_url=reverse("intervention:detail", args=(id,)) + "#related_data",
         )
 
+    @method_decorator(default_group_required)
     @method_decorator(shared_access_required(Intervention, "id"))
     def get(self, request, id: str, comp_id: str, *args, **kwargs) -> HttpResponse:
         return self.__process_request(request, id, comp_id, *args, **kwargs)
 
+    @method_decorator(default_group_required)
     @method_decorator(shared_access_required(Intervention, "id"))
     def post(self, request, id: str, comp_id: str, *args, **kwargs) -> HttpResponse:
         return self.__process_request(request, id, comp_id, *args, **kwargs)

konova/models/geometry.py

@@ -125,7 +125,7 @@ class Geometry(BaseResource):
                 deleted=None
             )
             if limit_to_attrs:
-                objs += set_objs.values_list(*limit_to_attrs, flat=True)
+                objs += set_objs.values(*limit_to_attrs)
             else:
                 objs += set_objs
 
@@ -135,16 +135,16 @@ class Geometry(BaseResource):
             Q(deleted=None) & Q(intervention__deleted=None)
         )
         if limit_to_attrs:
-            objs += comp_objs.values_list(*limit_to_attrs, flat=True)
+            objs += comp_objs.values(*limit_to_attrs)
         else:
             objs += comp_objs
         return objs
 
-    def get_data_object(self):
+    def get_data_object(self, limit_to_attrs: list = None):
         """
         Getter for the specific data object which is related to this geometry
         """
-        objs = self.get_data_objects()
+        objs = self.get_data_objects(limit_to_attrs)
         assert (len(objs) <= 1)
         result = objs.pop()
         return result
@@ -436,6 +436,16 @@ class Geometry(BaseResource):
             output_geom.transform(DEFAULT_SRID_RLP)
         return output_geom
 
+    def get_conflict_geometries(self):
+        """ Getter for geometry ids which conflict with this geometry or are conflicted by this one
+
+        Returns:
+            geom_ids (list): List of geometry ids
+        """
+        conflict_geoms_id = GeometryConflict.get_conflict_geometries(self)
+        conflict_geoms = Geometry.objects.filter(id__in=conflict_geoms_id)
+        return conflict_geoms
+
 
 class GeometryConflict(UuidModel):
     """
@@ -459,3 +469,23 @@ class GeometryConflict(UuidModel):
 
     def __str__(self):
         return f"{self.conflicting_geometry.id} conflicts with {self.affected_geometry.id}"
+
+    @staticmethod
+    def get_conflict_geometries(geometry: Geometry):
+        """ Getter for geometries which conflict in one or another way with the given one
+
+        Args:
+            geometry (Geometry): The geometry which shall be checked
+
+        Returns:
+            conflict_geometries (QuerySet): QuerySet of geometries which have conflicts with the given geometry
+        """
+        conflict_geometries = GeometryConflict.objects.filter(
+            affected_geometry=geometry.id,
+        ).values_list("conflicting_geometry__id", flat=True)
+        conflict_geometries = conflict_geometries.union(
+            GeometryConflict.objects.filter(
+                conflicting_geometry=geometry.id,
+            ).values_list("affected_geometry__id", flat=True)
+        )
+        return conflict_geometries

konova/models/object.py

@@ -676,24 +676,15 @@ class GeoReferencedMixin(models.Model):
         if self.geometry is None:
             return request
 
-        instance_objs = []
-        needed_data_object_attrs = [
-            "identifier"
-        ]
-        conflicts = self.geometry.conflicts_geometries.iterator()
+        conflicting_geometries = self.geometry.get_conflict_geometries()
+        data_object_identifiers = []
+        for conflicting_geom in conflicting_geometries:
+            data_object_identifiers.append(conflicting_geom.get_data_object(["identifier"]))
 
-        for conflict in conflicts:
-            # Only check the affected geometry of this conflict, since we know the conflicting geometry is self.geometry
-            instance_objs += conflict.affected_geometry.get_data_objects(needed_data_object_attrs)
-
-        conflicts = self.geometry.conflicted_by_geometries.iterator()
-        for conflict in conflicts:
-            # Only check the conflicting geometry of this conflict, since we know the affected geometry is self.geometry
-            instance_objs += conflict.conflicting_geometry.get_data_objects(needed_data_object_attrs)
-
-        add_message = len(instance_objs) > 0
+        add_message = len(data_object_identifiers) > 0
         if add_message:
-            instance_identifiers = ", ".join(instance_objs)
+            data_object_identifiers = [x["identifier"] for x in data_object_identifiers]
+            instance_identifiers = ", ".join(data_object_identifiers)
             message_str = GEOMETRY_CONFLICT_WITH_TEMPLATE.format(instance_identifiers)
             messages.info(request, message_str)
         return request

konova/sub_settings/django_settings.py

@@ -191,10 +191,11 @@ STATICFILES_DIRS = [
 ]
 
 # EMAIL (see https://docs.djangoproject.com/en/dev/topics/email/)
+
+# CHANGE_ME !!! ONLY FOR DEVELOPMENT !!!
 if DEBUG:
-    # ONLY FOR DEVELOPMENT NEEDED
     EMAIL_BACKEND = 'django.core.mail.backends.filebased.EmailBackend'
-    EMAIL_FILE_PATH = '/tmp/app-messages'
+    EMAIL_FILE_PATH = '/tmp/app-messages' # change this to a proper location
 
 DEFAULT_FROM_EMAIL = env("DEFAULT_FROM_EMAIL")  # The default email address for the 'from' element
 SERVER_EMAIL = DEFAULT_FROM_EMAIL  # The default email sender address, which is used by Django to send errors via mail

konova/utils/generators.py

@@ -5,22 +5,18 @@ Contact: michel.peltriaux@sgdnord.rlp.de
 Created on: 09.11.20
 
 """
-import random
+import secrets
 import string
-import qrcode
-import qrcode.image.svg
-
-from io import BytesIO
 
 
-def generate_token() -> str:
+def generate_token(length: int = 64) -> str:
     """ Shortcut for default generating of e.g. API token
 
     Returns:
         token (str)
     """
     return generate_random_string(
-        length=64,
+        length=length,
         use_numbers=True,
         use_letters_lc=True
     )
@@ -39,7 +35,7 @@ def generate_random_string(length: int, use_numbers: bool = False, use_letters_l
         elements.append(string.ascii_uppercase)
 
     elements = "".join(elements)
-    ret_val = "".join(random.choice(elements) for i in range(length))
+    ret_val = "".join(secrets.choice(elements) for i in range(length))
     return ret_val
 
 class IdentifierGenerator:

nginx.conf

@@ -1,25 +0,0 @@
-server {
-    listen 80;
-    client_max_body_size 25M;
-
-    location / {
-        proxy_pass http://127.0.0.1:8000;
-        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-        proxy_set_header X-Real-IP $remote_addr;
-        proxy_set_header Host $host;
-        proxy_redirect off;
-        proxy_cache_bypass $http_upgrade;
-    }
-
-    location /static/ {
-        alias /konova/static/;
-        access_log /var/log/nginx/access.log;
-        autoindex off;
-        types {
-            text/css css;
-            application/javascript js;
-        }
-    }
-
-    error_log /var/log/nginx/error.log;
-}

user/views/propagate.py

@@ -7,9 +7,10 @@ Created on: 10.05.24
 """
 import base64
 import hashlib
+import http
 import json
 
-from cryptography.fernet import Fernet
+from cryptography.fernet import Fernet, InvalidToken
 from django.core.exceptions import ObjectDoesNotExist
 from django.http import HttpRequest, JsonResponse
 from django.utils.decorators import method_decorator
@@ -27,34 +28,77 @@ class PropagateUserView(View):
     proper rights management)
 
     """
+    class PropagateStatus:
+        UNPROCESSED = "unprocessed"
+        UPDATED = "updated"
+        CREATED = "created"
 
     @method_decorator(csrf_exempt)
     def dispatch(self, request, *args, **kwargs):
         return super().dispatch(request, *args, **kwargs)
 
     def post(self, request: HttpRequest, *args, **kwargs):
+        response_data = {
+            "success": None,
+            "status": None
+        }
+
         # Decrypt
         encrypted_body = request.body
         _hash = hashlib.md5()
         _hash.update(PROPAGATION_SECRET.encode("utf-8"))
         key = base64.urlsafe_b64encode(_hash.hexdigest().encode("utf-8"))
         fernet = Fernet(key)
-        body = fernet.decrypt(encrypted_body).decode("utf-8")
-        body = json.loads(body)
-
         try:
-            status = "updated"
-            user = User.resolve_user_using_propagation_data(body)
-            user = user.update_user_using_propagation_data(body)
-        except ObjectDoesNotExist:
-            user = User(**body)
-            status = "created"
+            body = fernet.decrypt(encrypted_body).decode("utf-8")
+            body = json.loads(body)
+        except InvalidToken:
+            response_data["error"] = "Invalid Token"
+            response_data["success"] = False
+            response_data["status"] = self.PropagateStatus.UNPROCESSED
+            return JsonResponse(
+                status=http.HTTPStatus.UNPROCESSABLE_CONTENT,
+                data=response_data
+            )
+        except (json.JSONDecodeError) as e:
+            response_data["error"] = str(e)
+            response_data["success"] = False
+            response_data["status"] = self.PropagateStatus.UNPROCESSED
+            return JsonResponse(
+                status=http.HTTPStatus.UNPROCESSABLE_CONTENT,
+                data=response_data
+            )
+
+        # Process decrypted user data
+        processing_ret_vals = self.__process_user_data(body)
+        response_data["success"] = processing_ret_vals[0]
+        response_data["status"] = processing_ret_vals[1]
+        user = processing_ret_vals[2]
+
         user.set_unusable_password()
         user.save()
 
-        data = {
-            "success": True,
-            "status": status
-        }
+        return JsonResponse(
+            status=http.HTTPStatus.OK,
+            data=response_data
+        )
 
-        return JsonResponse(data)
+    def __process_user_data(self, body: dict) -> (bool, str, User):
+        """ Process decrypted user data
+
+        Args:
+            body:
+
+        Returns:
+            success (bool): Whether the processing was successful
+            status (str): In which way the data was used ('created' | 'updated')
+            user (User): Processed user object
+        """
+        try:
+            user = User.resolve_user_using_propagation_data(body)
+            user = user.update_user_using_propagation_data(body)
+            status = self.PropagateStatus.UPDATED
+        except ObjectDoesNotExist:
+            user = User(**body)
+            status = self.PropagateStatus.CREATED
+        return True, status, user