pangea/django-orchestra
7
0
Fork 0
Code Issues 1 Pull requests 2 Packages Projects Releases Wiki Activity

Added docs readme file

Browse source
This commit is contained in:
Marc Aymerich 2016-02-16 10:37:57 +00:00
parent 6c665295be
commit 20fc755b5c
2 changed files with 23 additions and 30 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

11
docs/README.md
View file

@ -1,3 +1,4 @@
# Documentation
### Quick start
0. Install django-orchestra following any of these methods:
@ -9,7 +10,7 @@
```bash
orchestra@panel:~ ssh-copy-id root@server.address
```
Then add the servers to orchestra using the web interface `/admin/orchestration/servers`, check that the SSH connection is working and Orchestra can report the uptime of the servers.
Then add the servers using the web interface `/admin/orchestration/servers`, check that the SSH connection is working and Orchestra can report the uptime of the servers.
2. Now configure service by service (domains, databases, webapps, websites, ...):
1. Add the route via `/admin/orchestration/route/`
@ -22,12 +23,12 @@
### Architecture
Orchestration
Orders
Resources
* [Orchestration](../orchestra/contrib/orchestration)
* [Orders](../orchestra/contrib/orders)
* [Resources](../orchestra/contrib/resources)
### Creating New Services
### Creating new services
1. Think about if the service can fit into one of the existing service models like: SaaS or WebApps, refere to the related documentation if that is the case.
2. Create a new django app using `startapp` management command. For ilustrational purposes we will create a crontab services that will allow orchestra to manage user-based crontabs.
```bash

42
orchestra/contrib/orchestration/README.md
View file

@ -1,6 +1,6 @@
# Orchestration
This module handles the management of the services controlled by Orchestra.
This module handles the management of the services controlled by Orchestra. This app provides the means for detecting changes on the data model and execute scripts on the servers to reflect those changes.
Orchestration module has the following pieces:
@ -16,7 +16,7 @@ Orchestration module has the following pieces:
Routes
======
This application provides support for mapping services to server machines accross the network.
This application provides support for mapping Orchestra service instances to server machines accross the network.
It supports _routing_ based on Python expression, which means that you can efectively
control services that are distributed accross several machines. For example, different
@ -25,12 +25,12 @@ environment.
### OperationsMiddleware
When enabled, `middlewares.OperationsMiddleware` automatically executes the service backends when a change on the data model occurs. The main steps that performs are:
`middlewares.OperationsMiddleware` automatically executes the service backends when a change on the data model occurs. The main steps that performs are:
1. Collect all `save` and `delete` model signals triggered on each HTTP request
2. Find related backends using the routing backend
3. Generate a single script per server (_unit of work_)
4. Execute the scripts on the servers
4. Execute the generated scripts on the servers via SSH
### Service Management Properties
@ -39,25 +39,22 @@ We can identify three different characteristics regarding service management:
* **Authority**: Whether or not Orchestra is the only source of the service configuration. When Orchestra is the authority then service configuration is _completely generated_ from the Orchestra database (or services are configured to read their configuration directly from Orchestra database). Otherwise Orchestra will execute small tasks translating model changes into configuration changes, allowing manual configurations to be preserved.
* **Flow**: _push_, when Orchestra drives the execution or _pull_, when external services connects to Orchestra.
* **Execution**: _synchronous_, when the execution blocks the HTTP request, or _asynchronous_ when it doesn't. Asynchronous execution means concurrency, and concurrency scalability.
_Sorry for the bad terminology, I was not able to find more appropriate terms on the literature._
* **Execution**: _synchronous_, when the execution blocks the HTTP request, or _asynchronous_ when it doesn't. Asynchronous execution means concurrency, and concurrency scalability and complexity (i.e. reporting user feedback of success/failed backend executions).
### Registry vs Synchronization vs Task
From the above management properties we can extract three main service management strategies: (a) _registry based management_, (b) _synchronization based management_ and (c) _task based management_. Orchestra provides support for all of them, it is left to you to decide which one suits your requirements better.
From the above management properties we can extract three main service management strategies: (a) _task based management_, (b) _synchronization based management_ and (c) _registry based management_. Orchestra provides support for all of them, it is left to you to decide which one suits your requirements better.
Following a brief description and evaluation of the tradeoffs to help on your decision making.
#### a. Task Based Management (prefered)
This model refers when Orchestra is _not the only source of configuration_. Therefore, Orchestra translates isolated data model changes directly into localized changes on the service configuration, and executing them using a **push** strategy. For example `save()` or `delete()` object-level operations may have sibling configuration management operations. In contrast to _synchronization_, tasks are able to preserve configuration not performed by Orchestra.
#### a. Registry Based Management
When Orchestra acts as a pure **configuration registry (authority)**, doing nothing more than store service's configuration on the database. The configuration is **pulled** from Orchestra by the servers themselves, so it is **asynchronous** by nature.
This model is intuitive, efficient and also very consistent when tasks are execute **synchronously** with the request/response cycle. However, **asynchronous** task execution can have _consistency issues_; tasks have state, and this state can be lost when:
- A failure occur while applying some changes, e.g. network error or worker crash while deleting a service
- Scripts are executed out of order, e.g. create and delete a service is applied in inverse order
This strategy considers two different implementations:
- The service is configured to read the configuration directly from Orchestra database (or REST API). This approach simplifies configuration management but also can make Orchestra a single point of failure on your architecture.
- A client-side application periodically fetches the service configuration from the Orchestra database and regenerates the service configuration files. This approach is very tolerant to failures, since the services will keep on working, and the new configuration will be applied after recovering. A delay may occur until the changes are applied to the services (_eventual consistency_), but it can be mitigated by notifying the application when a relevant change occur.
In general, _synchornous execution of tasks is preferred_ over asynchornous, unless response delays are not tolerable.
#### b. Synchronization Based Management
@ -65,24 +62,19 @@ When Orchestra is the configuration **authority** and also _the responsible of a
In contrast to registry based management, synchronization management is _fully centralized_, all the management operations are driven by Orchestra so you don't need to install nor configure anything on your servers.
#### c. Registry Based Management
When Orchestra acts as a pure **configuration registry (authority)**, doing nothing more than store service's configuration on the database. The configuration is **pulled** from Orchestra by the servers themselves, so it is **asynchronous** by nature.
This strategy considers two different implementations:
#### c. Task Based Management
This model refers when Orchestra is _not the only source of configuration_. Therefore, Orchestra translates isolated data model changes directly into localized changes on the service configuration, and executing them using a **push** strategy. For example `save()` or `delete()` object-level operations may have sibling configuration management operations. In contrast to synchronization, tasks are able to preserve configuration not performed by Orchestra.
- The service is configured to read the configuration directly from Orchestra database (or REST API). This approach simplifies configuration management but also can make Orchestra a single point of failure on your architecture.
- An application (_agent_) periodically fetches the service configuration from the Orchestra database and regenerates the service configuration files. This approach is very tolerant to failures, since the services will keep working independenlty from orchestra, and the new configuration will be applied after recovering. A delay may occur until the changes are applied to the services (_eventual consistency_), but it can be mitigated by notifying the application when a relevant change occur. User feedback about the success or failure of appling the configuration needs to be implemented by the agent.
This model is intuitive, efficient and also very consistent when tasks are execute **synchronously** with the request/response cycle. However, **asynchronous** task execution can have _consistency issues_; tasks have state, and this state can be lost when:
- A failure occur while applying some changes, e.g. network error or worker crash while deleting a service
- Scripts are executed out of order, e.g. create and delete a service is applied in inverse order
In general, _synchornous execution of tasks is preferred_ over asynchornous, unless response delays are not tolerable.
##### What state does actually mean?
Lets assume you have deleted a mailbox, and Orchestra has created an script that deletes that mailbox on the mail server. However a failure has occurred and the mailbox deletion task has been lost. Since the state has also been lost it is not easy to tell what to do now in order to maintain consistency.
### Additional Notes
* The script that manage the service needs to be idempotent, i.e. the outcome of running the script is always the same, no matter how many times it is executed.
* Renaming of attributes may lead to undesirable effects, e.g. changing a database name will create a new database rather than just changing its name.
* The system does not magically perform data migrations between servers when its _route_ has changed