up_snapshots: simplify and clarify errors

specially, do not halt in case of errors, just print them in logs

README.md

@@ -1,20 +1,123 @@
-# INSTALACIÓN:
+# Device Hub
 
-La instalación es muy estándar
+DeviceHub is an IT Asset Management System focused on reusing devices, created under the [eReuse.org](https://www.ereuse.org) project.
 
+## Overview
+
+DeviceHub aims to:
+
+- Provide a common IT Asset Management platform for donors, receivers, and IT professionals.
+- Automatically collect, analyze, and share device metadata while ensuring privacy and traceability.
+- Integrate with existing IT Asset Management Systems.
+- Operate in a decentralized manner.
+
+DeviceHub primarily works with three types of objects:
+
+1. **Devices**: Including computers, smartphones, and their components.
+2. **Events**: Actions performed on devices (e.g., Repair, Allocate).
+3. **Accounts**: Users who perform events on devices.
+
+## Installation
+
+Assuming a host with debian stable
+
+### Quickstart
+
+For a quick start with dummy data in localhost, DeviceHub can be run directly with docker. To do so, from the root of the project run:
+
+```bash
+./docker-reset.sh
 ```
+
+Note that everytime you perform the `docker-reset.sh` script, all data is lost.
+
+Also there is a demo running in http://demo.ereuse.org/. The token for accessing the instance will be always: `token=5018dd65-9abd-4a62-8896-80f34ac66150`, but the instance will be reset every day at 4 am.
+
+For production needs, review and change .env file properly
+
+## Running from baremetal
+
+### Prerequisites
+
+- Python 3.10
+- pip
+- virtualenv
+
+Specially when developing, is quite convenient to run DeviceHub from a virtual environment. To start with this deployment, create a virtual environment to isolate our project dependencies:
+
+```bash
 python -m venv env
-source env/bin/actevate
+source env/bin/activate
-python install -r requirements.txt
+pip install -r requirements.txt
 ```
 
-## IMPORTANT EXTERNAL DEPENDENCIES
+### System Dependencies
 
-Para arrancarlo es necesario tener el paquete `xapian-bindings` en tu ordenador. No se instala mediante `pip`, así que depende de cada [sistema operativo](https://xapian.org/download).
+#### Xapian
 
-Luego solo necesitas:
+Now, install the xapian dependencies (xapian library and python bindings)
 
+```bash
+sudo apt-get install python3-xapian libxapian-dev
 ```
-./manage.py migrate
+
-./manage.py runserver
+Allow the virtual environment to use system-installed packages:
+
+```bash
+export PYTHONPATH="${PYTHONPATH}:/usr/lib/python3/dist-packages"
 ```
+
+#### Environment Variables
+
+Now, configure the environment variables. For this, we will expand a `.env` file. You can use the following content as an example:
+
+```source
+STATIC_ROOT=/tmp/static/
+MEDIA_ROOT=/tmp/media/
+ALLOWED_HOSTS=localhost,localhost:8000,127.0.0.1,
+DOMAIN=localhost
+DEBUG=True
+```
+
+Now, expand the environment variables:
+
+```bash
+source .env
+```
+
+### Migrations
+
+Now, apply migrations
+
+```bash
+python manage.py makemigrations
+python manage.py migrate
+```
+
+Also, we can add some dummy data into the database to play along:
+
+```bash
+python manage.py add_institution Pangea
+python manage.py add_user Pangea user@example.org 1234
+python manage.py up_snapshots example/snapshots/ user@example.org
+```
+
+### Run DeviceHub
+
+Finally, we can run the DeviceHub service by running:
+
+```bash
+python manage.py runserver
+```
+
+### Clean up
+
+To clean up the deployment and start fresh, just delete Django's database:
+
+```bash
+rm db/*
+```
+
+## License
+
+DeviceHub is released under the [GNU Affero General Public License v3.0](LICENSE).

device/templates/details.html

@@ -58,7 +58,7 @@
         <div class="col-lg-9 col-md-8">{{ object.type }}</div>
       </div>
 
-      {% if object.is_websnapshot %}
+      {% if object.is_websnapshot and object.last_user_evidence %}
         {% for k, v in object.last_user_evidence %}
           <div class="row mb-3">
             <div class="col-lg-3 col-md-4 label">{{ k }}</div>

evidence/forms.py

@@ -15,7 +15,7 @@ from utils.save_snapshots import move_json, save_in_disk
 class UploadForm(forms.Form):
     evidence_file = MultipleFileField(label=_("File"))
 
-    def clean(self):
+    def clean_evidence_file(self):
         self.evidences = []
         data = self.cleaned_data.get('evidence_file')
         if not data:
@@ -33,13 +33,20 @@ class UploadForm(forms.Form):
                 exist_annotation = Annotation.objects.filter(
                     uuid=file_json['uuid']
                 ).first()
-
+    
                 if exist_annotation:
-                    raise ValidationError("error: {} exist".format(file_name))
+                    raise ValidationError(              
-
+                        _("The snapshot already exists"),
-            except Exception:
+                        code="duplicate_snapshot",
-                raise ValidationError("error in: {}".format(file_name))
+                    )
-
+                
+            #Catch any error and display it as Validation Error so the Form handles it
+            except Exception as e:
+                raise ValidationError(
+                    _("Error on '%(file_name)s': %(error)s"),
+                    code="error",
+                    params={"file_name": file_name, "error": getattr(e, 'message', str(e))},
+                )
             self.evidences.append((file_name, file_json))
 
         return True
@@ -123,7 +130,15 @@ class ImportForm(forms.Form):
         data = self.cleaned_data["file_import"]
 
         self.file_name = data.name
-        df = pd.read_excel(data)
+
+        try:
+            df = pd.read_excel(data)
+        except Exception as e:
+            raise ValidationError(
+                _("Error on '%(file_name)s': Invalid File"),
+                params={"file_name": self.file_name}
+            )
+
         df.fillna('', inplace=True)
 
         data_pd = df.to_dict(orient='index')

evidence/management/commands/up_snapshots.py

@@ -47,17 +47,22 @@ class Command(BaseCommand):
                 self.open(filepath)
 
     def open(self, filepath):
-        with open(filepath, 'r') as file:
+        try:
-            content = json.loads(file.read())
+            with open(filepath, 'r') as file:
-            path_name = save_in_disk(content, self.user.institution.name)
+                content = json.loads(file.read())
-            self.snapshots.append((content, path_name))
+                path_name = save_in_disk(content, self.user.institution.name)
+
+                self.snapshots.append((content, path_name))
+
+        except Exception as e:
+            logger.error("Could not open file %s: %s", filepath, e)
 
     def parsing(self):
         for s, p in self.snapshots:
             try:
                 self.devices.append(Build(s, self.user))
                 move_json(p, self.user.institution.name)
-            except Exception as err:
+            except Exception as e:
                 snapshot_id = s.get("uuid", "")
-                txt = "Could not parse snapshot: %s"
+                txt = "Could not parse snapshot %s: %s"
-                logger.error(txt, snapshot_id)
+                logger.error(txt, snapshot_id, e)

evidence/parse.py

@@ -4,6 +4,7 @@ import logging
 
 from dmidecode import DMIParse
 from json_repair import repair_json
+from evidence.parse_details import get_lshw_child
 
 from evidence.models import Annotation
 from evidence.xapian import index
@@ -12,16 +13,7 @@ from utils.constants import CHASSIS_DH
 
 logger = logging.getLogger('django')
 
-
-def get_network_cards(child, nets):
-    if child['id'] == 'network' and "PCI:" in child.get("businfo"):
-        nets.append(child)
-    if child.get('children'):
-        [get_network_cards(x, nets) for x in child['children']]
-
-
 def get_mac(lshw):
-    nets = []
     try:
         if type(lshw) is dict:
             hw = lshw
@@ -30,18 +22,16 @@ def get_mac(lshw):
     except json.decoder.JSONDecodeError:
         hw = json.loads(repair_json(lshw))
 
-    try:
+    nets = []
-        get_network_cards(hw, nets)
+    get_lshw_child(hw, nets, 'network')
-    except Exception as ss:
-        logger.warning("%s", ss)
-        return
 
     nets_sorted = sorted(nets, key=lambda x: x['businfo'])
-    # This funcion get the network card integrated in motherboard
-    # integrate = [x for x in nets if "pci@0000:00:" in x.get('businfo', '')]
 
     if nets_sorted:
-        return nets_sorted[0]['serial']
+        mac = nets_sorted[0]['serial']
+        logger.debug("The snapshot has the following MAC: %s" , mac)
+        return mac
+
 
 
 class Build:

evidence/templates/upload.html

@@ -8,23 +8,21 @@
   </div>
 </div>
 
+<!-- override invalid-feedback class -->
+<style>
+  .invalid-feedback {
+    color: #670000; 
+    font-size: 1rem; 
+  }
+  </style>
+
 {% load django_bootstrap5 %}
 <form role="form" method="post" enctype="multipart/form-data">
 {% csrf_token %}
-{% if form.errors %}
+
-<div class="alert alert-danger alert-icon alert-icon-border alert-dismissible" role="alert">
+{% bootstrap_form form alert_error_type="none" error_css_class="alert alert-danger alert-icon alert-icon-border" %}
-  <div class="icon"><span class="mdi mdi-close-circle-o"></span></div>
+      <div class="form-actions-no-box">
-  <div class="message">
+  <a class="btn btn-grey"       href="{% url 'dashboard:unassigned_devices' %}">{% translate "Cancel" %}</a>
-    {% for field, error in form.errors.items %}
-      {{ error }}<br />
-    {% endfor %}
-    <button class="btn-close" type="button" data-dismiss="alert" aria-label="Close"></button>
-  </div>
-</div>
-{% endif %}
-{% bootstrap_form form %}
-<div class="form-actions-no-box">
-  <a class="btn btn-grey" href="{% url 'dashboard:unassigned_devices' %}">{% translate "Cancel" %}</a>
   <input class="btn btn-green-admin" type="submit" name="submit" value="{% translate 'Save' %}" />
 </div>
 

evidence/views.py

@@ -1,5 +1,6 @@
 import json
 
+from django.contrib import messages
 from urllib.parse import urlparse
 from django.http import HttpResponse
 from django.utils.translation import gettext_lazy as _
@@ -47,6 +48,7 @@ class UploadView(DashboardView, FormView):
 
     def form_valid(self, form):
         form.save(self.request.user)
+        messages.success(self.request, _("Evidence uploaded successfully."))
         response = super().form_valid(form)
         return response
 
@@ -70,6 +72,7 @@ class ImportView(DashboardView, FormView):
 
     def form_valid(self, form):
         form.save()
+        messages.success(self.request, _("Evidence imported successfully."))
         response = super().form_valid(form)
         return response
 

utils/logger.py

@@ -13,9 +13,9 @@ class CustomFormatter(logging.Formatter):
         if record.levelname == "ERROR":
             color = RED
         elif record.levelname == "WARNING":
-            color = PURPLE
-        elif record.levelname in ["INFO", "DEBUG"]:
             color = YELLOW
+        elif record.levelname in ["INFO", "DEBUG"]:
+            color = PURPLE
         else:
             color = RESET