ereuse
/
devicehub-django
8
0
Fork 0
Code Issues 4 Pull Requests 3 Packages Projects Releases Wiki Activity

devicehub fails when adding problematic (clonic) snapshots #13

New Issue
Closed
opened 2024-10-16 19:03:37 +00:00 by pedro · 6 comments
pedro commented 2024-10-16 19:03:37 +00:00
Owner
Copy Link

I added additional snapshots to test the clonic issues

cd example/snapshots
wget https://cas.cat/ereuse-informes-septiembre-2024/ekoa_ids-clonicos/colision_ids-computer1_snapshot.json
wget https://cas.cat/ereuse-informes-septiembre-2024/ekoa_ids-clonicos/colision_ids-computer2_snapshot.json
wget https://cas.cat/ereuse-informes-septiembre-2024/ekoa_ids-clonicos/2024-10-16/20240825-17_35_56_91187f6f-0c29-43c6-a29e-348b8141bee0.json
wget https://cas.cat/ereuse-informes-septiembre-2024/ekoa_ids-clonicos/2024-10-16/20240825-17_35_56_91187f6f-0c29-43c6-a29e-348b8141bee0.json

the result is a general failure in the application, what should happen instead is an error handling that this snapshot could not annotate on id

devicehub-django-1  | + ./manage.py up_snapshots example/snapshots/ user@example.org
devicehub-django-1  | Traceback (most recent call last):
devicehub-django-1  |   File "/opt/devicehub-django/./manage.py", line 22, in <module>
devicehub-django-1  | DOMAIN: localhost
devicehub-django-1  | WARNING: Could not retrieve MAC address in snapshot c9649582-e1f3-4e7c-8b1a-a2df923a2066
devicehub-django-1  |     main()
devicehub-django-1  |   File "/opt/devicehub-django/./manage.py", line 18, in main
devicehub-django-1  |     execute_from_command_line(sys.argv)
devicehub-django-1  |   File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
devicehub-django-1  |     utility.execute()
devicehub-django-1  |   File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
devicehub-django-1  |     self.fetch_command(subcommand).run_from_argv(self.argv)
devicehub-django-1  |   File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 413, in run_from_argv
devicehub-django-1  |     self.execute(*args, **cmd_options)
devicehub-django-1  |   File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 459, in execute
devicehub-django-1  |     output = self.handle(*args, **options)
devicehub-django-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
devicehub-django-1  |   File "/opt/devicehub-django/evidence/management/commands/up_snapshots.py", line 35, in handle
devicehub-django-1  |     self.parsing()
devicehub-django-1  |   File "/opt/devicehub-django/evidence/management/commands/up_snapshots.py", line 50, in parsing
devicehub-django-1  |     self.devices.append(Build(s, self.user))
devicehub-django-1  |                         ^^^^^^^^^^^^^^^^^^^
devicehub-django-1  |   File "/opt/devicehub-django/evidence/parse.py", line 42, in __init__
devicehub-django-1  |     self.generate_chids()
devicehub-django-1  |   File "/opt/devicehub-django/evidence/parse.py", line 56, in generate_chids
devicehub-django-1  |     'hidalgo1': self.get_hid_14(),
devicehub-django-1  |                 ^^^^^^^^^^^^^^^^^
devicehub-django-1  |   File "/opt/devicehub-django/evidence/parse.py", line 63, in get_hid_14
devicehub-django-1  |     device = self.json['device']
devicehub-django-1  |              ~~~~~~~~~^^^^^^^^^^
devicehub-django-1  | KeyError: 'device'
devicehub-django-1 exited with code 1

the problematic snapshots are particularly AND/OR:

I added additional snapshots to test the clonic issues ```sh cd example/snapshots wget https://cas.cat/ereuse-informes-septiembre-2024/ekoa_ids-clonicos/colision_ids-computer1_snapshot.json wget https://cas.cat/ereuse-informes-septiembre-2024/ekoa_ids-clonicos/colision_ids-computer2_snapshot.json wget https://cas.cat/ereuse-informes-septiembre-2024/ekoa_ids-clonicos/2024-10-16/20240825-17_35_56_91187f6f-0c29-43c6-a29e-348b8141bee0.json wget https://cas.cat/ereuse-informes-septiembre-2024/ekoa_ids-clonicos/2024-10-16/20240825-17_35_56_91187f6f-0c29-43c6-a29e-348b8141bee0.json ``` the result is a general failure in the application, what should happen instead is an error handling that this snapshot could not annotate on id ``` devicehub-django-1 | + ./manage.py up_snapshots example/snapshots/ user@example.org devicehub-django-1 | Traceback (most recent call last): devicehub-django-1 | File "/opt/devicehub-django/./manage.py", line 22, in <module> devicehub-django-1 | DOMAIN: localhost devicehub-django-1 | WARNING: Could not retrieve MAC address in snapshot c9649582-e1f3-4e7c-8b1a-a2df923a2066 devicehub-django-1 | main() devicehub-django-1 | File "/opt/devicehub-django/./manage.py", line 18, in main devicehub-django-1 | execute_from_command_line(sys.argv) devicehub-django-1 | File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line devicehub-django-1 | utility.execute() devicehub-django-1 | File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute devicehub-django-1 | self.fetch_command(subcommand).run_from_argv(self.argv) devicehub-django-1 | File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 413, in run_from_argv devicehub-django-1 | self.execute(*args, **cmd_options) devicehub-django-1 | File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 459, in execute devicehub-django-1 | output = self.handle(*args, **options) devicehub-django-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ devicehub-django-1 | File "/opt/devicehub-django/evidence/management/commands/up_snapshots.py", line 35, in handle devicehub-django-1 | self.parsing() devicehub-django-1 | File "/opt/devicehub-django/evidence/management/commands/up_snapshots.py", line 50, in parsing devicehub-django-1 | self.devices.append(Build(s, self.user)) devicehub-django-1 | ^^^^^^^^^^^^^^^^^^^ devicehub-django-1 | File "/opt/devicehub-django/evidence/parse.py", line 42, in __init__ devicehub-django-1 | self.generate_chids() devicehub-django-1 | File "/opt/devicehub-django/evidence/parse.py", line 56, in generate_chids devicehub-django-1 | 'hidalgo1': self.get_hid_14(), devicehub-django-1 | ^^^^^^^^^^^^^^^^^ devicehub-django-1 | File "/opt/devicehub-django/evidence/parse.py", line 63, in get_hid_14 devicehub-django-1 | device = self.json['device'] devicehub-django-1 | ~~~~~~~~~^^^^^^^^^^ devicehub-django-1 | KeyError: 'device' devicehub-django-1 exited with code 1 ``` the problematic snapshots are particularly AND/OR: - https://cas.cat/ereuse-informes-septiembre-2024/ekoa_ids-clonicos/colision_ids-computer1_snapshot.json - https://cas.cat/ereuse-informes-septiembre-2024/ekoa_ids-clonicos/colision_ids-computer2_snapshot.json
cayop commented 2024-10-17 17:09:49 +00:00
Owner
Copy Link

El problema en ambos casos es que el campo software pone "Workbench" y en el pareso para los nuevos workbench debe poner:
"EreuseWorkbench"
Tanto por el formulario como por la api guarda el snapshot correctamente en la carpeta errors del disco duro.
He mejorado la respuesta de la api. Ahora es algo mejor pero no te dice porque no entra.

El problema en ambos casos es que el campo software pone "Workbench" y en el pareso para los nuevos workbench debe poner: "EreuseWorkbench" Tanto por el formulario como por la api guarda el snapshot correctamente en la carpeta errors del disco duro. He mejorado la respuesta de la api. Ahora es algo mejor pero no te dice porque no entra.
pedro commented 2024-10-18 15:40:25 +00:00
Author
Owner
Copy Link

hemos propuesto cambiar la validación del campo software de EreuseWorkbench a workbench-script, porque es exactamente el programa que se usa y es más claro

hemos propuesto cambiar la validación del campo software de EreuseWorkbench a workbench-script, porque es exactamente el programa que se usa y es más claro
pedro commented 2024-10-18 15:54:45 +00:00
Author
Owner
Copy Link

el error en devicehub se ve

devicehub-django-1  | Internal Server Error: /api/snapshot/
devicehub-django-1  | Internal Server Error: /api/snapshot/
devicehub-django-1  | [18/Oct/2024 15:42:25] "POST /api/snapshot/ HTTP/1.1" 500 28

y en workbench-script sale así:

[2024-10-18 17:42:25,731] workbench: ERROR: Snapshot not remotely sent to URL 'http://localhost:8000/api/snapshot/'. Do you have internet? Is your server up & running? Is the url token authorized?
    HTTP Error 500: Internal Server Error

queremos mejorar el mensaje de error por un lado en workbench y por otro en devicehub, seguimos en otro momento, un trabajo parcial fue forzar fallo en campo software de workbench y este patch

-        try:
             response = json.loads(response_text)
             if response.get('url'):
                 # apt install qrencode
@@ -218,8 +217,8 @@ def send_snapshot_to_devicehub(snapshot, token, url):
                 print("url: {}".format(response['url']))
             if response.get("dhid"):
                 print("dhid: {}".format(response['dhid']))
-        except Exception:
-            logger.error(response_text)
+        else:
+            raise Exception(response_text)
el error en devicehub se ve ``` devicehub-django-1 | Internal Server Error: /api/snapshot/ devicehub-django-1 | Internal Server Error: /api/snapshot/ devicehub-django-1 | [18/Oct/2024 15:42:25] "POST /api/snapshot/ HTTP/1.1" 500 28 ``` y en workbench-script sale así: ``` [2024-10-18 17:42:25,731] workbench: ERROR: Snapshot not remotely sent to URL 'http://localhost:8000/api/snapshot/'. Do you have internet? Is your server up & running? Is the url token authorized? HTTP Error 500: Internal Server Error ``` queremos mejorar el mensaje de error por un lado en workbench y por otro en devicehub, seguimos en otro momento, un trabajo parcial fue forzar fallo en campo software de workbench y este patch ```diff - try: response = json.loads(response_text) if response.get('url'): # apt install qrencode @@ -218,8 +217,8 @@ def send_snapshot_to_devicehub(snapshot, token, url): print("url: {}".format(response['url'])) if response.get("dhid"): print("dhid: {}".format(response['dhid'])) - except Exception: - logger.error(response_text) + else: + raise Exception(response_text) ```
pedro commented 2024-10-31 13:33:46 +00:00
Author
Owner
Copy Link

in the meantime the parsing improved, but right now, with latest commit right now d4f50961bc, and one of the target snapshots, I still have problems.

With only this snapshot wget https://cas.cat/ereuse-informes-septiembre-2024/ekoa_ids-clonicos/2024-10-16/20240825-17_35_56_91187f6f-0c29-43c6-a29e-348b8141bee0.json

The trace is duplicated two times, the first error messages is confusing, the second one is the right one

I tried to fix it myself, but I don't know, I document this here:

devicehub-django-1  | + ./manage.py up_snapshots example/snapshots/ user@example.org
devicehub-django-1  | 'device'
devicehub-django-1  | Traceback (most recent call last):
devicehub-django-1  |   File "/opt/devicehub-django/evidence/management/commands/up_snapshots.py", line 58, in parsing
devicehub-django-1  |     self.devices.append(Build(s, self.user))
devicehub-django-1  |                         ^^^^^^^^^^^^^^^^^^^
devicehub-django-1  |   File "/opt/devicehub-django/evidence/parse.py", line 53, in __init__
devicehub-django-1  |     self.generate_chids()
devicehub-django-1  |   File "/opt/devicehub-django/evidence/parse.py", line 67, in generate_chids
devicehub-django-1  |     'hidalgo1': self.get_hid_14(),
devicehub-django-1  |                 ^^^^^^^^^^^^^^^^^
devicehub-django-1  |   File "/opt/devicehub-django/evidence/parse.py", line 74, in get_hid_14
devicehub-django-1  |     device = self.json['device']
devicehub-django-1  |              ~~~~~~~~~^^^^^^^^^^
devicehub-django-1  | KeyError: 'device'
devicehub-django-1  | ERROR 2024-10-31 13:32:03,374 'device'
devicehub-django-1  | Traceback (most recent call last):
devicehub-django-1  |   File "/opt/devicehub-django/evidence/management/commands/up_snapshots.py", line 58, in parsing
devicehub-django-1  |     self.devices.append(Build(s, self.user))
devicehub-django-1  |                         ^^^^^^^^^^^^^^^^^^^
devicehub-django-1  |   File "/opt/devicehub-django/evidence/parse.py", line 53, in __init__
devicehub-django-1  |     self.generate_chids()
devicehub-django-1  |   File "/opt/devicehub-django/evidence/parse.py", line 67, in generate_chids
devicehub-django-1  |     'hidalgo1': self.get_hid_14(),
devicehub-django-1  |                 ^^^^^^^^^^^^^^^^^
devicehub-django-1  |   File "/opt/devicehub-django/evidence/parse.py", line 74, in get_hid_14
devicehub-django-1  |     device = self.json['device']
devicehub-django-1  |              ~~~~~~~~~^^^^^^^^^^
devicehub-django-1  | KeyError: 'device'
devicehub-django-1  | It is not possible to parse snapshot: 91187f6f-0c29-43c6-a29e-348b8141bee0
devicehub-django-1  | ERROR 2024-10-31 13:32:03,375 It is not possible to parse snapshot: 91187f6f-0c29-43c6-a29e-348b8141bee0

the expected log would be

devicehub-django-1  | + ./manage.py up_snapshots example/snapshots/ user@example.org
devicehub-django-1  | Traceback (most recent call last):
devicehub-django-1  |   File "/opt/devicehub-django/evidence/management/commands/up_snapshots.py", line 58, in parsing
devicehub-django-1  |     self.devices.append(Build(s, self.user))
devicehub-django-1  |                         ^^^^^^^^^^^^^^^^^^^
devicehub-django-1  |   File "/opt/devicehub-django/evidence/parse.py", line 53, in __init__
devicehub-django-1  |     self.generate_chids()
devicehub-django-1  |   File "/opt/devicehub-django/evidence/parse.py", line 67, in generate_chids
devicehub-django-1  |     'hidalgo1': self.get_hid_14(),
devicehub-django-1  |                 ^^^^^^^^^^^^^^^^^
devicehub-django-1  |   File "/opt/devicehub-django/evidence/parse.py", line 74, in get_hid_14
devicehub-django-1  |     device = self.json['device']
devicehub-django-1  |              ~~~~~~~~~^^^^^^^^^^
devicehub-django-1  | KeyError: 'device'
devicehub-django-1  | It is not possible to parse snapshot: 91187f6f-0c29-43c6-a29e-348b8141bee0
devicehub-django-1  | ERROR 2024-10-31 13:32:03,375 It is not possible to parse snapshot: 91187f6f-0c29-43c6-a29e-348b8141bee0
in the meantime the parsing improved, but right now, with latest commit right now d4f50961bc6e5137fc513628fd59c7ebd48d3d0b, and one of the target snapshots, I still have problems. With only this snapshot `wget https://cas.cat/ereuse-informes-septiembre-2024/ekoa_ids-clonicos/2024-10-16/20240825-17_35_56_91187f6f-0c29-43c6-a29e-348b8141bee0.json` The trace is duplicated two times, the first error messages is confusing, the second one is the right one I tried to fix it myself, but I don't know, I document this here: ``` devicehub-django-1 | + ./manage.py up_snapshots example/snapshots/ user@example.org devicehub-django-1 | 'device' devicehub-django-1 | Traceback (most recent call last): devicehub-django-1 | File "/opt/devicehub-django/evidence/management/commands/up_snapshots.py", line 58, in parsing devicehub-django-1 | self.devices.append(Build(s, self.user)) devicehub-django-1 | ^^^^^^^^^^^^^^^^^^^ devicehub-django-1 | File "/opt/devicehub-django/evidence/parse.py", line 53, in __init__ devicehub-django-1 | self.generate_chids() devicehub-django-1 | File "/opt/devicehub-django/evidence/parse.py", line 67, in generate_chids devicehub-django-1 | 'hidalgo1': self.get_hid_14(), devicehub-django-1 | ^^^^^^^^^^^^^^^^^ devicehub-django-1 | File "/opt/devicehub-django/evidence/parse.py", line 74, in get_hid_14 devicehub-django-1 | device = self.json['device'] devicehub-django-1 | ~~~~~~~~~^^^^^^^^^^ devicehub-django-1 | KeyError: 'device' devicehub-django-1 | ERROR 2024-10-31 13:32:03,374 'device' devicehub-django-1 | Traceback (most recent call last): devicehub-django-1 | File "/opt/devicehub-django/evidence/management/commands/up_snapshots.py", line 58, in parsing devicehub-django-1 | self.devices.append(Build(s, self.user)) devicehub-django-1 | ^^^^^^^^^^^^^^^^^^^ devicehub-django-1 | File "/opt/devicehub-django/evidence/parse.py", line 53, in __init__ devicehub-django-1 | self.generate_chids() devicehub-django-1 | File "/opt/devicehub-django/evidence/parse.py", line 67, in generate_chids devicehub-django-1 | 'hidalgo1': self.get_hid_14(), devicehub-django-1 | ^^^^^^^^^^^^^^^^^ devicehub-django-1 | File "/opt/devicehub-django/evidence/parse.py", line 74, in get_hid_14 devicehub-django-1 | device = self.json['device'] devicehub-django-1 | ~~~~~~~~~^^^^^^^^^^ devicehub-django-1 | KeyError: 'device' devicehub-django-1 | It is not possible to parse snapshot: 91187f6f-0c29-43c6-a29e-348b8141bee0 devicehub-django-1 | ERROR 2024-10-31 13:32:03,375 It is not possible to parse snapshot: 91187f6f-0c29-43c6-a29e-348b8141bee0 ``` the expected log would be ``` devicehub-django-1 | + ./manage.py up_snapshots example/snapshots/ user@example.org devicehub-django-1 | Traceback (most recent call last): devicehub-django-1 | File "/opt/devicehub-django/evidence/management/commands/up_snapshots.py", line 58, in parsing devicehub-django-1 | self.devices.append(Build(s, self.user)) devicehub-django-1 | ^^^^^^^^^^^^^^^^^^^ devicehub-django-1 | File "/opt/devicehub-django/evidence/parse.py", line 53, in __init__ devicehub-django-1 | self.generate_chids() devicehub-django-1 | File "/opt/devicehub-django/evidence/parse.py", line 67, in generate_chids devicehub-django-1 | 'hidalgo1': self.get_hid_14(), devicehub-django-1 | ^^^^^^^^^^^^^^^^^ devicehub-django-1 | File "/opt/devicehub-django/evidence/parse.py", line 74, in get_hid_14 devicehub-django-1 | device = self.json['device'] devicehub-django-1 | ~~~~~~~~~^^^^^^^^^^ devicehub-django-1 | KeyError: 'device' devicehub-django-1 | It is not possible to parse snapshot: 91187f6f-0c29-43c6-a29e-348b8141bee0 devicehub-django-1 | ERROR 2024-10-31 13:32:03,375 It is not possible to parse snapshot: 91187f6f-0c29-43c6-a29e-348b8141bee0 ```
cayop commented 2024-10-31 14:26:14 +00:00
Owner
Copy Link

solucionado

solucionado
cayop closed this issue 2024-10-31 14:26:14 +00:00
pedro commented 2024-11-05 02:46:10 +00:00
Author
Owner
Copy Link

@cayop se refería a este commit e4124fb20b

el resultado son dos log ERROR en rojo por fallo de snapshot (ahora sí se ha eliminado la trace)

devicehub-django-1  | ERROR 2024-11-05 01:37:28,472 'device'
devicehub-django-1  | Traceback (most recent call last):
devicehub-django-1  |   File "/opt/devicehub-django/evidence/management/commands/up_snapshots.py", line 58, in parsing
devicehub-django-1  |     self.devices.append(Build(s, self.user))
devicehub-django-1  |                         ^^^^^^^^^^^^^^^^^^^
devicehub-django-1  |   File "/opt/devicehub-django/evidence/parse.py", line 53, in __init__
devicehub-django-1  |     self.generate_chids()
devicehub-django-1  |   File "/opt/devicehub-django/evidence/parse.py", line 67, in generate_chids
devicehub-django-1  |     'hidalgo1': self.get_hid_14(),
devicehub-django-1  |                 ^^^^^^^^^^^^^^^^^
devicehub-django-1  |   File "/opt/devicehub-django/evidence/parse.py", line 74, in get_hid_14
devicehub-django-1  |     device = self.json['device']
devicehub-django-1  |              ~~~~~~~~~^^^^^^^^^^
devicehub-django-1  | KeyError: 'device'
devicehub-django-1  | ERROR 2024-11-05 01:37:28,473 It is not possible to parse snapshot: c9649582-e1f3-4e7c-8b1a-a2df923a2066

en 79a34c9b55 he cambiado que ahora que si hay variable DEBUG definida en cualquier parte, logger intenta imprimir traceback, entonces, simplifica un poco el error handling

@cayop se refería a este commit https://gitea.pangea.org/ereuse/devicehub-django/commit/e4124fb20b11d2b63bde9e997c9a16a4fc36168a el resultado son dos log ERROR en rojo por fallo de snapshot (ahora sí se ha eliminado la trace) ``` devicehub-django-1 | ERROR 2024-11-05 01:37:28,472 'device' devicehub-django-1 | Traceback (most recent call last): devicehub-django-1 | File "/opt/devicehub-django/evidence/management/commands/up_snapshots.py", line 58, in parsing devicehub-django-1 | self.devices.append(Build(s, self.user)) devicehub-django-1 | ^^^^^^^^^^^^^^^^^^^ devicehub-django-1 | File "/opt/devicehub-django/evidence/parse.py", line 53, in __init__ devicehub-django-1 | self.generate_chids() devicehub-django-1 | File "/opt/devicehub-django/evidence/parse.py", line 67, in generate_chids devicehub-django-1 | 'hidalgo1': self.get_hid_14(), devicehub-django-1 | ^^^^^^^^^^^^^^^^^ devicehub-django-1 | File "/opt/devicehub-django/evidence/parse.py", line 74, in get_hid_14 devicehub-django-1 | device = self.json['device'] devicehub-django-1 | ~~~~~~~~~^^^^^^^^^^ devicehub-django-1 | KeyError: 'device' devicehub-django-1 | ERROR 2024-11-05 01:37:28,473 It is not possible to parse snapshot: c9649582-e1f3-4e7c-8b1a-a2df923a2066 ``` en https://gitea.pangea.org/ereuse/devicehub-django/commit/79a34c9b55b761d0cf7866dfeb052097e2b2ba54 he cambiado que ahora que si hay variable DEBUG definida en cualquier parte, logger intenta imprimir traceback, entonces, simplifica un poco el error handling
pedro reopened this issue 2024-11-05 02:46:10 +00:00
pedro closed this issue 2024-11-05 02:46:13 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
main
96-add-serial-number-in-public-web
bugfix/parsing_error
feature/actions
improve-readme
api_bugfix_public-url
Labels
Clear labels
No items
No Label
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: ereuse/devicehub-django#13
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?