diff --git a/ROADMAP.md b/ROADMAP.md
index c4b3f6f8..fd8b2a25 100644
--- a/ROADMAP.md
+++ b/ROADMAP.md
@@ -13,8 +13,8 @@ Note `*` _for sustancial progress_
2. [x] [Orchestra-orm](https://github.com/glic3rinu/orchestra-orm) a Python library for easily interacting with Orchestra REST API
3. [x] Service orchestration framework
4. [ ] Data model, crazy input validation, admin and REST interfaces, permissions, unit and functional tests, service management, migration scripts and documentation of:
- 1. [x] PHP/static Web applications
- 1. [x] Websites with Apache
+ 1. [ ] *PHP/static Web applications
+ 1. [ ] *Websites with Apache
2. [x] FTP/rsync/scp/shell system accounts
2. [ ] *Databases and database users with MySQL
1. [x] Mail accounts, aliases, forwards with Postfix and Dovecot
diff --git a/TODO.md b/TODO.md
index 7df42033..216a5e82 100644
--- a/TODO.md
+++ b/TODO.md
@@ -16,9 +16,7 @@ TODO
* move invoice contact to invoices app?
* PHPbBckendMiixin with get_php_ini
* Apache: `IncludeOptional /etc/apache2/extra-vhos[t]/account-site-custom.con[f]`
-* rename account.user to main_user
* webmail identities and addresses
-* cached -> cached_property
* user.roles.mailbox its awful when combined with addresses:
* address.mailboxes filter by account is crap in admin and api
* address.mailboxes api needs a mailbox object endpoint (not nested user)
@@ -50,16 +48,13 @@ Remember that, as always with QuerySets, any subsequent chained methods which im
* EMAIL backend operations which contain stderr messages (because under certain failures status code is still 0)
-
* Settings dictionary like DRF2 in order to better override large settings like WEBSITES_APPLICATIONS.etc
-
* DOCUMENT: orchestration.middleware: we need to know when an operation starts and ends in order to perform bulk server updates and also to wait for related objects to be saved (base object is saved first and then related)
orders.signales: we perform changes right away because data model state can change under monitoring and other periodik task, and we should keep orders consistency under any situation.
dependency collector with max_recursion that matches the number of dots on service.match and service.metric
-
* backend logs with hal logo
* Use logs for storing monitored values
* set_password orchestration method?
@@ -94,9 +89,8 @@ Remember that, as always with QuerySets, any subsequent chained methods which im
return order.register_at.date()
* mail backend related_models = ('resources__content_type') ??
-* ignore orders
-* Dropdown menu for Account services/management object-tools
+* ignore orders (mark orders as ignored)
* Domain backend PowerDNS Bind validation support?
@@ -108,37 +102,23 @@ Remember that, as always with QuerySets, any subsequent chained methods which im
*jabber with mailbox accounts (dovecto mail notification)
-* rename accounts register to manager register
-
-* make accounts django auth users
- - when an account is created a mirrored system user is created
- - system users are independent users, so they can have different passwords and all.
+* rename accounts register to manager register or accounttools, accountutils
* take a look icons from ajenti ;)
-
* Disable services is_active should be computed on the fly in order to distinguish account.is_active from service.is_active when reactivation.
* Perhaps it is time to create a ServiceModel ?
-
-* COpy account.main_user.username to account.name for performance
-
-* service backend execution dependency? first create user on NIS master then create directories on service server
-
* prevent deletion of main user by the user itself
-
-
* AccountAdminMixin auto adds 'account__name' on searchfields and handle account_link on fieldsets
* Separate panel from server passwords? Store passwords on panel? set_password special backend operation?
* be more explicit about which backends are resources and which are service handling
-
* What fields we really need on contacts? name email phone and what more?
-
* Redirect junk emails and delete every 30 days?
* DOC: Complitely decouples scripts execution, billing, service definition
@@ -149,24 +129,12 @@ Remember that, as always with QuerySets, any subsequent chained methods which im
* Unify all users
-* backend admin message with link
-
* delete main user -> delete account or prevent delete main user
-
-APPS app?
-
* https://blog.flameeyes.eu/2011/01/mostly-unknown-openssh-tricks
-
* Ansible orchestration *method* (methods.py)
-* interdependency user <-> account with the old usermodel
-
-
* pip upgrade or install
-
-
-* disable account triggers save on cascade to execute backends save(update_field=[])
-
-
-* validate database user names
* multiple domains creation; line separated domains
+* Move MU webapps to SaaS?
+
+* DN: Transaction atomicity and backend failure
diff --git a/orchestra/apps/accounts/models.py b/orchestra/apps/accounts/models.py
index 2864619a..c9606b02 100644
--- a/orchestra/apps/accounts/models.py
+++ b/orchestra/apps/accounts/models.py
@@ -49,6 +49,10 @@ class Account(auth.AbstractBaseUser):
def is_staff(self):
return self.is_superuser
+ @property
+ def main_systemuser(self):
+ return self.systemusers.get(is_main=True)
+
@classmethod
def get_main(cls):
return cls.objects.get(pk=settings.ACCOUNTS_MAIN_PK)
@@ -63,7 +67,8 @@ class Account(auth.AbstractBaseUser):
created = not self.pk
super(Account, self).save(*args, **kwargs)
if created and hasattr(self, 'systemusers'):
- self.systemusers.create_user(self.username, account=self, password=self.password, is_main=True)
+ self.systemusers.create(username=self.username, account=self,
+ password=self.password, is_main=True)
def disable(self):
self.is_active = False
diff --git a/orchestra/apps/domains/backends.py b/orchestra/apps/domains/backends.py
index 29be0800..fbad6d0a 100644
--- a/orchestra/apps/domains/backends.py
+++ b/orchestra/apps/domains/backends.py
@@ -15,6 +15,7 @@ class Bind9MasterDomainBackend(ServiceController):
('domains.Record', 'domain__origin'),
('domains.Domain', 'origin'),
)
+ ignore_fields = ['serial']
@classmethod
def is_main(cls, obj):
@@ -25,6 +26,7 @@ class Bind9MasterDomainBackend(ServiceController):
def save(self, domain):
context = self.get_context(domain)
domain.refresh_serial()
+ print domain.render_zone()
context['zone'] = ';; %(banner)s\n' % context
context['zone'] += domain.render_zone()
self.append("{ echo -e '%(zone)s' | diff -N -I'^;;' %(zone_path)s - ; } ||"
diff --git a/orchestra/apps/domains/models.py b/orchestra/apps/domains/models.py
index 356f4c16..71dce022 100644
--- a/orchestra/apps/domains/models.py
+++ b/orchestra/apps/domains/models.py
@@ -1,5 +1,4 @@
from django.db import models
-from django.utils.functional import cached_property
from django.utils.translation import ugettext_lazy as _
from orchestra.core import services
diff --git a/orchestra/apps/domains/serializers.py b/orchestra/apps/domains/serializers.py
index 46aee9de..dffc2dab 100644
--- a/orchestra/apps/domains/serializers.py
+++ b/orchestra/apps/domains/serializers.py
@@ -34,6 +34,8 @@ class DomainSerializer(AccountSerializerMixin, serializers.HyperlinkedModelSeria
try:
validators.validate_zone(domain.render_zone())
except ValidationError as err:
- self._errors = { 'all': err.message }
+ self._errors = {
+ 'all': err.message
+ }
return None
return instance
diff --git a/orchestra/apps/domains/tests/functional_tests/tests.py b/orchestra/apps/domains/tests/functional_tests/tests.py
index 4060f41b..1755df0b 100644
--- a/orchestra/apps/domains/tests/functional_tests/tests.py
+++ b/orchestra/apps/domains/tests/functional_tests/tests.py
@@ -8,7 +8,7 @@ from django.core.urlresolvers import reverse
from selenium.webdriver.support.select import Select
from orchestra.apps.orchestration.models import Server, Route
-from orchestra.utils.tests import BaseLiveServerTestCase, random_ascii, snapshot_on_error
+from orchestra.utils.tests import BaseLiveServerTestCase, random_ascii, snapshot_on_error, save_response_on_error
from orchestra.utils.system import run
from ... import settings, utils, backends
@@ -129,7 +129,7 @@ class DomainTestMixin(object):
'domain_name': domain_name,
'server_addr': server_addr
}
- dig_soa = 'dig @%(server_addr)s %(domain_name)s SOA|grep "\sSOA\s"'
+ dig_soa = 'dig @%(server_addr)s %(domain_name)s SOA | grep "\sSOA\s"'
soa = run(dig_soa % context).stdout.split()
# testdomain.org. 3600 IN SOA ns.example.com. hostmaster.example.com. 2014021100 86400 7200 2419200 3600
self.assertEqual('%(domain_name)s.' % context, soa[0])
@@ -140,7 +140,7 @@ class DomainTestMixin(object):
hostmaster = utils.format_hostmaster(settings.DOMAINS_DEFAULT_HOSTMASTER)
self.assertEqual(hostmaster, soa[5])
- dig_ns = 'dig @%(server_addr)s %(domain_name)s NS|grep "\sNS\s"'
+ dig_ns = 'dig @%(server_addr)s %(domain_name)s NS |grep "\sNS\s"'
name_servers = run(dig_ns % context).stdout
ns_records = ['ns1.%s.' % self.domain_name, 'ns2.%s.' % self.domain_name]
self.assertEqual(2, len(name_servers.splitlines()))
@@ -153,7 +153,7 @@ class DomainTestMixin(object):
self.assertEqual('NS', ns[3])
self.assertIn(ns[4], ns_records)
- dig_mx = 'dig @%(server_addr)s %(domain_name)s MX|grep "\sMX\s"'
+ dig_mx = 'dig @%(server_addr)s %(domain_name)s MX | grep "\sMX\s"'
mx = run(dig_mx % context).stdout.split()
# testdomain.org. 3600 IN MX 10 orchestra.lan.
self.assertEqual('%(domain_name)s.' % context, mx[0])
@@ -163,7 +163,7 @@ class DomainTestMixin(object):
self.assertIn(mx[4], ['30', '40'])
self.assertIn(mx[5], ['mail3.orchestra.lan.', 'mail4.orchestra.lan.'])
- dig_cname = 'dig @%(server_addr)s www.%(domain_name)s CNAME|grep "\sCNAME\s"'
+ dig_cname = 'dig @%(server_addr)s www.%(domain_name)s CNAME | grep "\sCNAME\s"'
cname = run(dig_cname % context).stdout.split()
# testdomain.org. 3600 IN MX 10 orchestra.lan.
self.assertEqual('www.%(domain_name)s.' % context, cname[0])
@@ -194,13 +194,13 @@ class DomainTestMixin(object):
self.add(self.ns1_name, self.ns1_records)
self.add(self.ns2_name, self.ns2_records)
self.add(self.domain_name, self.domain_records)
- self.addCleanup(partial(self.delete, self.domain_name))
+# self.addCleanup(partial(self.delete, self.domain_name))
self.update(self.domain_name, self.domain_update_records)
- self.add(self.www_name, self.www_records)
+# self.add(self.www_name, self.www_records)
time.sleep(0.5)
self.validate_update(self.MASTER_SERVER_ADDR, self.domain_name)
- time.sleep(5)
- self.validate_update(self.SLAVE_SERVER_ADDR, self.domain_name)
+# time.sleep(5)
+# self.validate_update(self.SLAVE_SERVER_ADDR, self.domain_name)
def test_add_add_delete_delete(self):
self.add(self.ns1_name, self.ns1_records)
@@ -276,15 +276,18 @@ class RESTDomainMixin(DomainTestMixin):
self.rest_login()
self.add_route()
+ @save_response_on_error
def add(self, domain_name, records):
records = [ dict(type=type, value=value) for type,value in records ]
self.rest.domains.create(name=domain_name, records=records)
+ @save_response_on_error
def delete(self, domain_name):
domain = Domain.objects.get(name=domain_name)
domain = self.rest.domains.retrieve(id=domain.pk)
domain.delete()
+ @save_response_on_error
def update(self, domain_name, records):
records = [ dict(type=type, value=value) for type,value in records ]
domains = self.rest.domains.retrieve(name=domain_name)
diff --git a/orchestra/apps/lists/tests/functional_tests/tests.py b/orchestra/apps/lists/tests/functional_tests/tests.py
index ce08b33f..496f0262 100644
--- a/orchestra/apps/lists/tests/functional_tests/tests.py
+++ b/orchestra/apps/lists/tests/functional_tests/tests.py
@@ -64,7 +64,7 @@ class ListMixin(object):
backend = backends.MailmanBackend.get_name()
Route.objects.create(backend=backend, match=True, host=server)
- def atest_add(self):
+ def test_add(self):
name = '%s_list' % random_ascii(10)
password = '@!?%spppP001' % random_ascii(5)
admin_email = 'root@test3.orchestra.lan'
@@ -100,8 +100,8 @@ class RESTListMixin(ListMixin):
self.rest.lists.create(name=name, password=password, admin_email=admin_email, **extra)
@save_response_on_error
- def delete(self, username):
- list = self.rest.lists.retrieve(name=username).get()
+ def delete(self, name):
+ list = self.rest.lists.retrieve(name=name).get()
list.delete()
diff --git a/orchestra/apps/orchestration/manager.py b/orchestra/apps/orchestration/manager.py
index 6af93bd0..6c1e85d3 100644
--- a/orchestra/apps/orchestration/manager.py
+++ b/orchestra/apps/orchestration/manager.py
@@ -14,6 +14,7 @@ logger = logging.getLogger(__name__)
def as_task(execute):
def wrapper(*args, **kwargs):
+ """ failures on the backend execution doesn't fuck the request transaction atomicity """
db.transaction.set_autocommit(False)
try:
log = execute(*args, **kwargs)
diff --git a/orchestra/apps/orchestration/middlewares.py b/orchestra/apps/orchestration/middlewares.py
index a368bc19..410c235c 100644
--- a/orchestra/apps/orchestration/middlewares.py
+++ b/orchestra/apps/orchestration/middlewares.py
@@ -49,6 +49,7 @@ class OperationsMiddleware(object):
request = getattr(cls.thread_locals, 'request', None)
if request is None:
return
+ good_action = action
pending_operations = cls.get_pending_operations()
for backend in ServiceBackend.get_backends():
instance = None
@@ -75,7 +76,7 @@ class OperationsMiddleware(object):
if update_fields:
# "update_fileds=[]" is a convention for explicitly executing backend
# i.e. account.disable()
- if not update_fields == []:
+ if update_fields != []:
execute = False
for field in update_fields:
if field not in backend.ignore_fields:
@@ -84,13 +85,18 @@ class OperationsMiddleware(object):
if not execute:
continue
instance = copy.copy(instance)
+ good = instance
operation = Operation.create(backend, instance, action)
if action != Operation.DELETE:
# usually we expect to be using last object state,
# except when we are deleting it
pending_operations.discard(operation)
pending_operations.add(operation)
-
+ try:
+ print kwargs['instance'], good_action
+ except:
+ pass
+
def process_request(self, request):
""" Store request on a thread local variable """
type(self).thread_locals.request = request
diff --git a/orchestra/apps/systemusers/tests/functional_tests/tests.py b/orchestra/apps/systemusers/tests/functional_tests/tests.py
index 5055fdf5..369d1481 100644
--- a/orchestra/apps/systemusers/tests/functional_tests/tests.py
+++ b/orchestra/apps/systemusers/tests/functional_tests/tests.py
@@ -82,9 +82,9 @@ class SystemUserMixin(object):
# Home will be deleted on account delete, see test_delete_account
def validate_ftp(self, username, password):
- connection = ftplib.FTP(self.MASTER_SERVER)
- connection.login(user=username, passwd=password)
- connection.close()
+ ftp = ftplib.FTP(self.MASTER_SERVER)
+ ftp.login(user=username, passwd=password)
+ ftp.close()
def validate_sftp(self, username, password):
transport = paramiko.Transport((self.MASTER_SERVER, 22))
diff --git a/orchestra/apps/webapps/backends/__init__.py b/orchestra/apps/webapps/backends/__init__.py
index 07763320..e1077e22 100644
--- a/orchestra/apps/webapps/backends/__init__.py
+++ b/orchestra/apps/webapps/backends/__init__.py
@@ -1,23 +1,31 @@
import pkgutil
-
+import textwrap
class WebAppServiceMixin(object):
model = 'webapps.WebApp'
def create_webapp_dir(self, context):
- self.append("mkdir -p '%(app_path)s'" % context)
- self.append("chown %(user)s.%(group)s '%(app_path)s'" % context)
+ self.append(textwrap.dedent("""
+ path=""
+ for dir in $(echo %(app_path)s | tr "/" "\n"); do
+ path="${path}/${dir}"
+ [ -d $path ] || {
+ mkdir "${path}"
+ chown %(user)s.%(group)s "${path}"
+ }
+ done
+ """ % context))
def delete_webapp_dir(self, context):
self.append("rm -fr %(app_path)s" % context)
def get_context(self, webapp):
return {
- 'user': webapp.account.user.username,
- 'group': webapp.account.user.username,
+ 'user': webapp.account.username,
+ 'group': webapp.account.username,
'app_name': webapp.name,
'type': webapp.type,
- 'app_path': webapp.get_path(),
+ 'app_path': webapp.get_path().rstrip('/'),
'banner': self.get_banner(),
}
diff --git a/orchestra/apps/webapps/backends/awstats.py b/orchestra/apps/webapps/backends/awstats.py
deleted file mode 100644
index f5c60b27..00000000
--- a/orchestra/apps/webapps/backends/awstats.py
+++ /dev/null
@@ -1,9 +0,0 @@
-from django.utils.translation import ugettext_lazy as _
-
-from orchestra.apps.orchestration import ServiceController
-
-from . import WebAppServiceMixin
-
-
-class AwstatsBackend(WebAppServiceMixin, ServiceController):
- verbose_name = _("Awstats")
diff --git a/orchestra/apps/webapps/backends/phpfcgid.py b/orchestra/apps/webapps/backends/phpfcgid.py
index ce6baecf..27d45a90 100644
--- a/orchestra/apps/webapps/backends/phpfcgid.py
+++ b/orchestra/apps/webapps/backends/phpfcgid.py
@@ -1,4 +1,5 @@
import os
+import textwrap
from django.utils.translation import ugettext_lazy as _
@@ -15,9 +16,12 @@ class PHPFcgidBackend(WebAppServiceMixin, ServiceController):
context = self.get_context(webapp)
self.create_webapp_dir(context)
self.append("mkdir -p %(wrapper_dir)s" % context)
- self.append(
- "{ echo -e '%(wrapper_content)s' | diff -N -I'^\s*#' %(wrapper_path)s - ; } ||"
- " { echo -e '%(wrapper_content)s' > %(wrapper_path)s; UPDATED=1; }" % context)
+ self.append(textwrap.dedent("""\
+ {
+ echo -e '%(wrapper_content)s' | diff -N -I'^\s*#' %(wrapper_path)s -
+ } || {
+ echo -e '%(wrapper_content)s' > %(wrapper_path)s; UPDATED_APACHE=1
+ }""" % context))
self.append("chmod +x %(wrapper_path)s" % context)
self.append("chown -R %(user)s.%(group)s %(wrapper_dir)s" % context)
@@ -25,6 +29,10 @@ class PHPFcgidBackend(WebAppServiceMixin, ServiceController):
context = self.get_context(webapp)
self.delete_webapp_dir(context)
+ def commit(self):
+ super(PHPFcgidBackend, self).commit()
+ self.append("[[ $UPDATED_APACHE == 1 ]] && { /etc/init.d/apache reload; }")
+
def get_context(self, webapp):
context = super(PHPFcgidBackend, self).get_context(webapp)
init_vars = webapp.get_php_init_vars()
@@ -36,12 +44,12 @@ class PHPFcgidBackend(WebAppServiceMixin, ServiceController):
context['init_vars'] = ''
wrapper_path = settings.WEBAPPS_FCGID_PATH % context
context.update({
- 'wrapper_content': (
- "#!/bin/sh\n"
- "# %(banner)s\n"
- "export PHPRC=/etc/%(type)s/cgi/\n"
- "exec /usr/bin/%(type)s-cgi %(init_vars)s\n"
- ) % context,
+ 'wrapper_content': textwrap.dedent("""\
+ #!/bin/sh
+ # %(banner)s
+ export PHPRC=/etc/%(type)s/cgi/
+ exec /usr/bin/%(type)s-cgi %(init_vars)s
+ """ % context),
'wrapper_path': wrapper_path,
'wrapper_dir': os.path.dirname(wrapper_path),
})
diff --git a/orchestra/apps/webapps/backends/phpfpm.py b/orchestra/apps/webapps/backends/phpfpm.py
index 2e46851e..ee1cebd6 100644
--- a/orchestra/apps/webapps/backends/phpfpm.py
+++ b/orchestra/apps/webapps/backends/phpfpm.py
@@ -36,8 +36,8 @@ class PHPFPMBackend(WebAppServiceMixin, ServiceController):
})
context['fpm_listen'] = settings.WEBAPPS_FPM_LISTEN % context
fpm_config = Template(
- "[{{ user }}]\n"
";; {{ banner }}\n"
+ "[{{ user }}]\n"
"user = {{ user }}\n"
"group = {{ group }}\n\n"
"listen = {{ fpm_listen | safe }}\n"
diff --git a/orchestra/apps/webapps/models.py b/orchestra/apps/webapps/models.py
index 30c5eaa0..635e8c01 100644
--- a/orchestra/apps/webapps/models.py
+++ b/orchestra/apps/webapps/models.py
@@ -57,7 +57,7 @@ class WebApp(models.Model):
return init_vars
def get_fpm_port(self):
- return settings.WEBAPPS_FPM_START_PORT + self.account.user.pk
+ return settings.WEBAPPS_FPM_START_PORT + self.account.pk
def get_method(self):
method = settings.WEBAPPS_TYPES[self.type]
@@ -66,7 +66,7 @@ class WebApp(models.Model):
def get_path(self):
context = {
- 'user': self.account.user,
+ 'user': self.account.username,
'app_name': self.name,
}
return settings.WEBAPPS_BASE_ROOT % context
diff --git a/orchestra/apps/webapps/tests/__init__.py b/orchestra/apps/webapps/tests/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/orchestra/apps/webapps/tests/functional_tests/__init__.py b/orchestra/apps/webapps/tests/functional_tests/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/orchestra/apps/webapps/tests/functional_tests/tests.py b/orchestra/apps/webapps/tests/functional_tests/tests.py
new file mode 100644
index 00000000..7349a29e
--- /dev/null
+++ b/orchestra/apps/webapps/tests/functional_tests/tests.py
@@ -0,0 +1,181 @@
+import ftplib
+import os
+import time
+import textwrap
+from StringIO import StringIO
+
+from django.conf import settings as djsettings
+from django.contrib.contenttypes.models import ContentType
+from django.core.management.base import CommandError
+from django.core.urlresolvers import reverse
+from selenium.webdriver.support.select import Select
+
+from orchestra.apps.accounts.models import Account
+from orchestra.apps.domains.models import Domain
+from orchestra.apps.orchestration.models import Server, Route
+from orchestra.apps.resources.models import Resource
+from orchestra.apps.systemusers.backends import SystemUserBackend
+from orchestra.utils.system import run, sshrun
+from orchestra.utils.tests import BaseLiveServerTestCase, random_ascii, snapshot_on_error, save_response_on_error
+
+from ... import backends, settings
+from ...models import WebApp
+
+
+class WebAppMixin(object):
+ MASTER_SERVER = os.environ.get('ORCHESTRA_MASTER_SERVER', 'localhost')
+ DEPENDENCIES = (
+ 'orchestra.apps.orchestration',
+ 'orchestra.apps.systemusers',
+ 'orchestra.apps.webapps',
+ )
+
+ def setUp(self):
+ super(WebAppMixin, self).setUp()
+ self.add_route()
+ djsettings.DEBUG = True
+
+ def add_route(self):
+# backends = [
+# # TODO MU apps on SaaS?
+# backends.awstats.AwstatsBackend,
+# backends.dokuwikimu.DokuWikiMuBackend,
+# backends.drupalmu.DrupalMuBackend,
+# backends.phpfcgid.PHPFcgidBackend,
+# backends.phpfpm.PHPFPMBackend,
+# backends.static.StaticBackend,
+# backends.wordpressmu.WordpressMuBackend,
+# ]
+ server = Server.objects.create(name=self.MASTER_SERVER)
+ for backend in [SystemUserBackend, self.backend]:
+ backend = backend.get_name()
+ Route.objects.create(backend=backend, match=True, host=server)
+
+ def test_add(self):
+ name = '%s_%s_webapp' % (random_ascii(10), self.type_value)
+ self.add_webapp(name)
+ self.validate_add_webapp(name)
+# self.addCleanup(self.delete, username)
+
+
+class StaticWebAppMixin(object):
+ backend = backends.static.StaticBackend
+ type_value = 'static'
+ token = random_ascii(100)
+ page = (
+ 'index.html',
+ 'Hello World! %s \n' % token,
+ 'Hello World! %s \n' % token,
+ )
+
+ def validate_add_webapp(self, name):
+ try:
+ ftp = ftplib.FTP(self.MASTER_SERVER)
+ ftp.login(user=self.account.username, passwd=self.account_password)
+ ftp.cwd('webapps/%s' % name)
+ index = StringIO()
+ index.write(self.page[1])
+ index.seek(0)
+ ftp.storbinary('STOR %s' % self.page[0], index)
+ index.close()
+ finally:
+ ftp.close()
+
+
+class PHPFcidWebAppMixin(StaticWebAppMixin):
+ backend = backends.phpfcgid.PHPFcgidBackend
+ type_value = 'php5'
+ token = random_ascii(100)
+ page = (
+ 'index.php',
+ '\n' % token,
+ 'Hello World! %s' % token,
+ )
+
+
+class PHPFPMWebAppMixin(StaticWebAppMixin):
+ backend = backends.phpfpm.PHPFPMBackend
+ type_value = 'php5.5'
+ token = random_ascii(100)
+ page = (
+ 'index.php',
+ '\n' % token,
+ 'Hello World! %s' % token,
+ )
+
+
+class RESTWebAppMixin(object):
+ def setUp(self):
+ super(RESTWebAppMixin, self).setUp()
+ self.rest_login()
+ # create main user
+ self.save_systemuser()
+
+ @save_response_on_error
+ def save_systemuser(self):
+ self.rest.systemusers.retrieve().get().save()
+
+ @save_response_on_error
+ def add_webapp(self, name, options=[]):
+ self.rest.webapps.create(name=name, type=self.type_value)
+
+ @save_response_on_error
+ def delete_webapp(self, name):
+ list = self.rest.lists.retrieve(name=name).get()
+ list.delete()
+
+
+class AdminWebAppMixin(WebAppMixin):
+ def setUp(self):
+ super(AdminWebAppMixin, self).setUp()
+ self.admin_login()
+ # create main user
+ self.save_systemuser()
+ # TODO save_account()
+
+ @snapshot_on_error
+ def add(self, name, password, admin_email):
+ url = self.live_server_url + reverse('admin:mails_List_add')
+ self.selenium.get(url)
+
+ account_input = self.selenium.find_element_by_id('id_account')
+ account_select = Select(account_input)
+ account_select.select_by_value(str(self.account.pk))
+
+ name_field = self.selenium.find_element_by_id('id_name')
+ name_field.send_keys(username)
+
+ password_field = self.selenium.find_element_by_id('id_password1')
+ password_field.send_keys(password)
+ password_field = self.selenium.find_element_by_id('id_password2')
+ password_field.send_keys(password)
+
+ if quota is not None:
+ quota_id = 'id_resources-resourcedata-content_type-object_id-0-allocated'
+ quota_field = self.selenium.find_element_by_id(quota_id)
+ quota_field.clear()
+ quota_field.send_keys(quota)
+
+ if filtering is not None:
+ filtering_input = self.selenium.find_element_by_id('id_filtering')
+ filtering_select = Select(filtering_input)
+ filtering_select.select_by_value("CUSTOM")
+ filtering_inline = self.selenium.find_element_by_id('fieldsetcollapser0')
+ filtering_inline.click()
+ time.sleep(0.5)
+ filtering_field = self.selenium.find_element_by_id('id_custom_filtering')
+ filtering_field.send_keys(filtering)
+
+ name_field.submit()
+ self.assertNotEqual(url, self.selenium.current_url)
+
+
+class RESTWebAppTest(PHPFcidWebAppMixin, RESTWebAppMixin, WebAppMixin, BaseLiveServerTestCase):
+ pass
+
+
+#class AdminWebAppTest(AdminWebAppMixin, BaseLiveServerTestCase):
+# pass
+
+
+
diff --git a/orchestra/apps/websites/backends/apache.py b/orchestra/apps/websites/backends/apache.py
index eb252d5f..007c8cac 100644
--- a/orchestra/apps/websites/backends/apache.py
+++ b/orchestra/apps/websites/backends/apache.py
@@ -30,15 +30,13 @@ class Apache2Backend(ServiceController):
apache_conf = Template(textwrap.dedent("""\
# {{ banner }}
- ServerName {{ site.domains.all|first }}
+ ServerName {{ site.domains.all|first }}\
{% if site.domains.all|slice:"1:" %}
- ServerAlias {{ site.domains.all|slice:"1:"|join:' ' }}
- {% endif %}
+ ServerAlias {{ site.domains.all|slice:"1:"|join:' ' }}{% endif %}
CustomLog {{ logs }} common
- SuexecUserGroup {{ user }} {{ group }}
- {% for line in extra_conf.splitlines %}"
- {{ line | safe }}
- {% endfor %}
+ SuexecUserGroup {{ user }} {{ group }}\
+ {% for line in extra_conf.splitlines %}
+ {{ line | safe }}{% endfor %}
"""
))
apache_conf = apache_conf.render(Context(context))
@@ -83,13 +81,13 @@ class Apache2Backend(ServiceController):
context = self.get_content_context(content)
context['fcgid_path'] = fcgid_path % context
fcgid = self.get_alias_directives(content)
- fcgid += (
- "ProxyPass %(location)s !\n"
- "\n"
- " Options +ExecCGI\n"
- " AddHandler fcgid-script .php\n"
- " FcgidWrapper %(fcgid_path)s\n"
- ) % context
+ fcgid += textwrap.dedent("""\
+ ProxyPass %(location)s !
+
+ Options +ExecCGI
+ AddHandler fcgid-script .php
+ FcgidWrapper %(fcgid_path)s
+ """ % context)
for option in content.webapp.options.filter(name__startswith='Fcgid'):
fcgid += " %s %s\n" % (option.name, option.value)
fcgid += "\n"
@@ -100,11 +98,11 @@ class Apache2Backend(ServiceController):
custom_cert = site.options.filter(name='ssl')
if custom_cert:
cert = tuple(custom_cert[0].value.split())
- directives = (
- "SSLEngine on\n"
- "SSLCertificateFile %s\n"
- "SSLCertificateKeyFile %s\n"
- ) % cert
+ directives = textwrap.dedent("""\
+ SSLEngine on
+ SSLCertificateFile %s
+ SSLCertificateKeyFile %s""" % cert
+ )
return directives
def get_security(self, site):
@@ -129,17 +127,17 @@ class Apache2Backend(ServiceController):
path, name, passwd = re.match(regex, protection.value).groups()
path = os.path.join(context['root'], path)
passwd = os.path.join(self.USER_HOME % context, passwd)
- protections += ("\n"
- "\n"
- " AllowOverride All\n"
-# " AuthPAM_Enabled off\n"
- " AuthType Basic\n"
- " AuthName %s\n"
- " AuthUserFile %s\n"
- " \n"
- " require valid-user\n"
- " \n"
- "\n" % (path, name, passwd)
+ protections += textwrap.dedent("""
+
+ AllowOverride All
+ #AuthPAM_Enabled off
+ AuthType Basic
+ AuthName %s
+ AuthUserFile %s
+
+ require valid-user
+
+ """ % (path, name, passwd)
)
return protections
@@ -161,8 +159,8 @@ class Apache2Backend(ServiceController):
'site': site,
'site_name': site.name,
'site_unique_name': site.unique_name,
- 'user': site.account.user.username,
- 'group': site.account.user.username,
+ 'user': site.account.username,
+ 'group': site.account.username,
'sites_enabled': sites_enabled,
'sites_available': "%s.conf" % os.path.join(sites_available, site.unique_name),
'logs': os.path.join(settings.WEBSITES_BASE_APACHE_LOGS, site.unique_name),
@@ -190,7 +188,7 @@ class Apache2Traffic(ServiceMonitor):
def prepare(self):
current_date = timezone.localtime(self.current_date)
current_date = current_date.strftime("%Y%m%d%H%M%S")
- self.append(textwrap.dedent("""
+ self.append(textwrap.dedent("""\
function monitor () {
OBJECT_ID=$1
INI_DATE=$2
diff --git a/orchestra/apps/websites/tests/__init__.py b/orchestra/apps/websites/tests/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/orchestra/apps/websites/tests/functional_tests/__init__.py b/orchestra/apps/websites/tests/functional_tests/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/orchestra/apps/websites/tests/functional_tests/tests.py b/orchestra/apps/websites/tests/functional_tests/tests.py
new file mode 100644
index 00000000..7840a4a9
--- /dev/null
+++ b/orchestra/apps/websites/tests/functional_tests/tests.py
@@ -0,0 +1,91 @@
+import os
+import socket
+import time
+import textwrap
+
+from django.conf import settings as djsettings
+from django.contrib.contenttypes.models import ContentType
+from django.core.management.base import CommandError
+from django.core.urlresolvers import reverse
+import requests
+from selenium.webdriver.support.select import Select
+
+from orchestra.apps.accounts.models import Account
+from orchestra.apps.domains.models import Domain, Record
+from orchestra.apps.domains.backends import Bind9MasterDomainBackend
+from orchestra.apps.orchestration.models import Server, Route
+from orchestra.apps.resources.models import Resource
+from orchestra.apps.webapps.tests.functional_tests.tests import StaticWebAppMixin, RESTWebAppMixin, WebAppMixin, PHPFcidWebAppMixin, PHPFPMWebAppMixin
+from orchestra.utils.system import run, sshrun
+from orchestra.utils.tests import BaseLiveServerTestCase, random_ascii, snapshot_on_error, save_response_on_error
+
+from ... import backends, settings
+from ...models import Website
+
+
+class WebsiteMixin(WebAppMixin):
+ MASTER_SERVER = os.environ.get('ORCHESTRA_MASTER_SERVER', 'localhost')
+ MASTER_SERVER_ADDR = socket.gethostbyname(MASTER_SERVER)
+ DEPENDENCIES = (
+ 'orchestra.apps.orchestration',
+ 'orchestra.apps.domains',
+ 'orchestra.apps.websites',
+ 'orchestra.apps.webapps',
+ 'orchestra.apps.systemusers',
+ )
+
+ def add_route(self):
+ super(WebsiteMixin, self).add_route()
+ server = Server.objects.get()
+ backend = backends.apache.Apache2Backend.get_name()
+ Route.objects.create(backend=backend, match=True, host=server)
+ backend = Bind9MasterDomainBackend.get_name()
+ Route.objects.create(backend=backend, match=True, host=server)
+
+ def validate_add_website(self, name, domain):
+ url = 'http://%s/%s' % (domain.name, self.page[0])
+ self.assertEqual(self.page[2], requests.get(url).content)
+
+ def test_add(self):
+ # TODO domains with "_" bad name!
+ domain_name = '%sdomain.lan' % random_ascii(10)
+ domain = Domain.objects.create(name=domain_name, account=self.account)
+ domain.records.create(type=Record.A, value=self.MASTER_SERVER_ADDR)
+ self.save_domain(domain)
+ webapp = '%s_%s_webapp' % (random_ascii(10), self.type_value)
+ self.add_webapp(webapp)
+ self.validate_add_webapp(webapp)
+ website = '%s_website' % random_ascii(10)
+ self.add_website(website, domain, webapp)
+ self.validate_add_website(website, domain)
+
+
+class RESTWebsiteMixin(RESTWebAppMixin):
+ @save_response_on_error
+ def save_domain(self, domain):
+ self.rest.domains.retrieve().get().save()
+
+ def add_website(self, name, domain, webapp):
+ domain = self.rest.domains.retrieve().get()
+ webapp = self.rest.webapps.retrieve().get()
+ self.rest.websites.create(name=name, domains=[domain.url], contents=[{'webapp': webapp.url}])
+
+
+
+#class RESTWebsiteTest(RESTWebsiteMixin, StaticWebAppMixin, WebsiteMixin, BaseLiveServerTestCase):
+# pass
+
+
+PHPFPMWebAppMixin
+#class RESTWebsiteTest(RESTWebsiteMixin, PHPFcidWebAppMixin, WebsiteMixin, BaseLiveServerTestCase):
+# pass
+
+
+class RESTWebsiteTest(RESTWebsiteMixin, PHPFPMWebAppMixin, WebsiteMixin, BaseLiveServerTestCase):
+ pass
+
+#class AdminWebsiteTest(AdminWebsiteMixin, BaseLiveServerTestCase):
+# pass
+
+
+
diff --git a/orchestra/core/validators.py b/orchestra/core/validators.py
index 584387dd..aef8fc66 100644
--- a/orchestra/core/validators.py
+++ b/orchestra/core/validators.py
@@ -41,7 +41,7 @@ def validate_name(value):
"""
A single non-empty line of free-form text with no whitespace.
"""
- validators.RegexValidator('^\w+$',
+ validators.RegexValidator('^[\.\w]+$',
_("Enter a valid name (text without whitspaces)."), 'invalid')(value)
diff --git a/scripts/services/apache_full_stack.md b/scripts/services/apache_full_stack.md
index 229bd26a..57cb3b0a 100644
--- a/scripts/services/apache_full_stack.md
+++ b/scripts/services/apache_full_stack.md
@@ -28,12 +28,14 @@ The goal of this setup is having a high-performance state-of-the-art deployment
apt-get install apache2-mpm-event php5-fpm libapache2-mod-fcgid apache2-suexec-custom php5-cgi
```
+# TODO libapache2-mod-auth-pam is no longer part of the debian distribution,
+# replace with libapache2-mod-authnz-external pwauth
2. Enable some convinient Apache modules
```bash
a2enmod suexec
a2enmod ssl
- a2enmod auth_pam
+ #a2enmod auth_pam
a2enmod proxy_fcgi
a2emmod userdir
```
@@ -55,15 +57,18 @@ The goal of this setup is having a high-performance state-of-the-art deployment
```
+
+
+
5. Restart Apache
```bash
service apache2 restart
```
-* TODO
- libapache2-mod-auth-pam
- https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=710770
+
+
+
* ExecCGI
@@ -73,6 +78,11 @@ The goal of this setup is having a high-performance state-of-the-art deployment
```
+* Permissions
+
+ Require all granted
+
+
TODO CHRoot
https://andrewbevitt.com/tutorials/apache-varnish-chrooted-php-fpm-wordpress-virtual-host/
diff --git a/scripts/services/vsftpd.md b/scripts/services/vsftpd.md
index 0267c54e..91b0d0ca 100644
--- a/scripts/services/vsftpd.md
+++ b/scripts/services/vsftpd.md
@@ -10,10 +10,12 @@ VsFTPd with System Users
2. Make some configurations
```bash
- sed -i "s/anonymous_enable=YES/anonymous_enable=NO/" /etc/vsftpd.conf
- sed -i "s/#local_enable=YES/local_enable=YES/" /etc/vsftpd.conf
- sed -i "s/#write_enable=YES/write_enable=YES/" /etc/vsftpd.conf
- # sed -i "s/#chroot_local_user=YES/chroot_local_user=YES/" /etc/vsftpd.conf
+ sed -i "s/^anonymous_enable=YES/anonymous_enable=NO/" /etc/vsftpd.conf
+ sed -i "s/^#local_enable=YES/local_enable=YES/" /etc/vsftpd.conf
+ sed -i "s/^#write_enable=YES/write_enable=YES/" /etc/vsftpd.conf
+ # sed -i "s/^#chroot_local_user=YES/chroot_local_user=YES/" /etc/vsftpd.conf
+
+ sed -i "s/^#local_umask=022/local_umask=022/" /etc/vsftpd.conf
echo '/dev/null' >> /etc/shells
```