Wait; may it be that we are basically looking for Sockethub, which drives us back to the Unhosted / Offline apps movement again?
Although I like the concept of redecentralization, I set it up rather as a long-term target, rather than something we can reach with our immediate short needs. The idea of bringing Pierre here into the discussion was particularly to understand and plan together how:
- we can better adapt the existing infrastructure of Ecobytes (the 4 physical servers, which are currently undergoing revision/planning new setups) to use by the multiple services/applications the different collectives want to use
- find ways to articulate these different tools into one space with some form of integrated workflow for groups/organisations, particularly what we want to do with Drupal/Open Atrium 2 co-munity.net.
There is simply a huge need for such a thing and we have a lot of groups/users that are really waiting for the new co-munity to come into existance (many are already users of the old co-munity, which is far from being a decent set of tools, while others have used it/heard about and are waiting to bring their groups into the new tool). Not to mention that we need to have the new system
- ready, optimized and with redundancy for the participants area of the huge degrowth conference - 1000-2500 participants,
- interacting not only as general discussions,
- but also doing open peer review,
- collaborative document writing,
- ideally (but not priority) also uploading media content.
Would be therefore great if we could provide a show-off of all these amazing tools that are out there and that we can service, already at the conference or in a near future. I propose for this effect that we have an Ecobytes/Allmende/hackers space at the Degrowth fair that will happen at the conference everyday from 17h-19h - anyone volunteering )
My suggestions would be as follows:
- Use a OpenStack IaaS with a OpenShift PaaS instances to quickly run VPSs, Docker instances or anything PHP / Node / Ruby … distributed on the four servers.
- Use a central, API-aggregating iPaaS like bipio or Sockethub. Nicely integrated into Drupal.
Therefore following the argumentation from the Seafile maintainers:
About the plugin system: Seafile will not have a plugin system. Instead, we will try to integrate other services by API’s. For example, we will not have a plugin for features like syncing calendar and contacts, but we will try to work together with system like Radicale (https://github.com/Kozea/Radicale). Many systems are focus on their specific tasks and do much better. So instead of replicate their work, it is much better to work together with them.
I don’t believe in making OA2 an all-in-one tool, but in making it a flexible framework, where one can dock new Apps via their respective APIs.
My recommendations for the tasks are:
- OpenStack + OpenShift for redundancy and scalability
- Discourse for discussions.
- Peer Review : should be something workflow based like Penflip or Authorea. Or something where one can comment on PDFs. Anyone seen this out in the wild already?
- EtherPads + Fidus Writer for collaborative writing.
- Media Goblin for media content. But that could also been done by ownCloud (slow!) or Seafile.
Pushing a bit the discussion again.
OA2 not as all-in-one tool but flexible framework - yes, definitely, it would just be too heavy and unreliable to have everything working on a platform. Furthermore, it makes totally sense what Seafile maintainers write.
A major aspect I think needs to be addressed is how the fine group structure+access control that of OA2 provides be projected into the multiple apps.
A simple short-term solution for implementing some apps is, like we are doing now with an etherpad iframe integration in OA, to simply index the random token from the URL part within a group - and then providing a listing for all the etherpads in the group.
I know little IaaS, PaaS and iPaaS. At what level would they work and articulate with the platform?
What about LDAP as an underlying auth system, keeping track of groups and users, on which other apps could also rely?
On peer review I already commented on the Trello board. I didn’t find any PDF commenting possibilities on the open world.
ownCloud was what we were thinking to provide, Seafile is new to me, looks interesting.
But, generally, how do we move forward? In particular, some of these things (peer review, etherpad) should be working by the end of May, to be used by 2000 Degrowth conference participants in their OA2 participants area. @almereyda would you like to directly contribute here, helping to make a few first steps for this framework?
I have been thinking a lot about this lately. You can see that in my writings here and in the Trello boards. Now it would be nice to have some other Ecobyterians joining the discussion.
I will now say a little to your most important points
- Collaborative Writing
- Peer Review
which both work very reliably. Further on I will go into some detail regarding
- ** aaS*.
Following will be a disambiguation between Ecobytes, allmende.io and co-munity v2.
As you defined yourself, Etherpad Lite is the way to go. It’s a very easy installation process and a reliable service that can be extended by plugins. Which should be done for a professional Service like the one for Degrowth. I suppose the implementation time is around 2 to 4 hours.
If you want to have a more sophisticated approach that doesn’t lack a beautiful UI, please consider adding http://fiduswriter.org/ for real documents with annotations next to pad’ish scribbles.
As I have written in the co-munity v2 Board Peer Review Card Hypothes.is combines annotation with PDFs. I’ve now installed it and it works. Chrome users will have to install a custom-built plugin, that injects PDF.js into the browser, and/or the stock plugin (which is hard-coded at https://hypothes.is, which didn’t launch yet) + a bookmarklet. The bookmarklet works out-of-the-box in Firefox, as it’s using PDF.js internally already.
Coming with Open ID Connect is the tighter integration of authentication and authorization for different web services. These layers can be built connected to some LDAP service, some have it built in. Please refer to the Trello Card about SSO again, where I’ve collected some implementations. I suppose the best would be to rely on the Gluu stack, as it’s already available.
Again, one would need a middleware (thanks @daniel ) that propagates the desired restrictions or permissions to the applications. Also not all applications have access-controls, like Etherpad. So you will have to manage your private URIs from the OA2 frontend solely.
I’m not sure if OA/Drupal is capable of doing so. But if you hard-wire your OA2 instance to the self-deployed Open ID Connect provider only, then people wouldn’t need to provide different login credentials for any service in your grid.
This is the tricky question now, eh? Why I would also like to have Ecobyterians joining this discussion.
I have already laid out my strategy in multiple ways, but will try to resume again for easier comprehensibility. As I didn’t receive too much feedback on my work / comments / etc. on the Trello Boards, I have somewhat slowed down my productions. Otherwise you would now have access to more structural hypothesis images.
For detailed explanations, please also refer to your own research and as a starting point this Wikipedia article.
Infrastructure as a Service means you can spawn virtual machines of different size within a federated cluster of servers. Or move them. Or move storage independent of CPU. Or allocate RAM dynamically. Or create VLANs. Or run docker images.
We would need those for big applications like Groupserver, GitLab, Discourse, database servers or Seafile.
Platform as a Service means you can push application code (Node, Python, PHP, …) to your service and it will be a running program. This can be tightly integrated into Continuous Integration (CI) lifecycle processes.
This would include Etherpad, Fidus Writer, remoteStorage, Open Atrium, WordPress, …
Software as a Service is the tools we want to provide in the end.
Now the tricky part is if and how Ecobytes wants to re-setup the root servers.
Are these strong physical machines that can be transformed into an OpenStack + OpenShift cluster or do you want to keep the current systems (+ data!) and evolve a new system?
I’m here for help, practical work or guidance. But please not that I’m also an autodidact that can strongly fail.
The most interesting question for me is how you see co-munity inside this network of providers and services. Because I have the feeling that co-munity + Ecobytes is the same for you. For me it is not. Also allmende.io remains a research prototype that could slowly evolve, if MMM would have any resources left over or a big sponsorship.
If you want to have me working on this architecture and applications, please let’s go into in depth review of your concept, if there is any, and compare it with the one I’ve clicked together. Then we can say where we work together and where each one follows an own approach.
Like ownCloud, for example. I love to have multiple apps doing different things. ownCloud combines so many different ideas into one product, that it’s hard to say if it’s good or bad. Despite the fact that there are already real large scale deploys on top of this. Let’s take the calendar functionality: Seafile doesn’t have it, ownCloud does. But the Radicale server mentionned in the Seafile quote does have LDAP integration, so does Baikal. Seafile, too. It would be easy to connect CardDavMate and CalDavZap to an existing ecosystem. In a next step I could even imagine adding OpenID functionality to these lightweight web clients.
It always depends on what you want to do.
Thanks @almereyda for your detailed and comprehensive post. Somehow I don’t seem to be getting Trello notifications for this board (or I’ve overseen them - I confess I get easily lost with Trello threads). I have a troublesome week with a lot of paper reviews, ill daughter and funding applications, so I’ll try to reply shortly to provide some more disambiguation and bring others in.
Ecobytes, co-munity, Degrowth conference
The first disambiguation that I think needs to be made. All the three are different things, although the projects have aspects in common. Ecobytes is a tech collective that has been running since 2004 to support collectives, artists, etc… It has been trying to survive in the last few years, this year fortunately it is gaining new power after some new people have joined and the association was formed. We do both (paid and non-paid) hosting and (usually paid) web development. co-munity is simply a Drupal/Open Atrium platform that mainly I developed/am developing. It emerged with the Beyond Our Backyards project, but quickly attracted several groups to use it, which made me understand there was a potential usefulness in further developing this. The degrowth conference is a big event taking place in September in Leipzig (I know you know this, but to contextualize for others as well) and we want to setup a users area for it, with collaboration and peer review. The participants area should be launched between mid-May and June (with a beta version for testing 15 days before)
Due to the implementation time constraints for the Degrowth conference, I’m definitely headed for Etherpad. Many people are already used to it and the time-frame of collaboration is so short (it is intended for the working groups to use it during and probably after the conference to formulate proposals together), that we probably would not benefit from many additional features. Besides that, the etherpad will be framed withing What plugins did you have in mind?
Hypothes.is looking great! Unfortunately it seems not to have been released yet, which makes it difficult to get it on time for the conference. I did not find much about it, but since it’s built over Annotator, we can probably for now implement Annotator and hope an upgrade path is possible? (if not, we just discontinue it after the conference, no big issue I think). An important note on this: the peer review of the scientific papers will already have been passed. This will only have to do with the “stirring papers”, although it can also be used for an unofficial peer review of already accepted scientific papers. I tried to push open peer review to all the conference, but the hardcore academics are a bit difficult to convince, even in such a conference…
Open Atrium 2 supports LDAP, including spaces and group control (https://drupal.org/project/oa_ldap), which is one of the main reasons I am very excited about using it for access control to the multiple applications. I already talked with biko about this, he can probably comment better (I just invited him). One urgent thing we will need is to import all users that register on the degrowth conference to the OA platform users area. The registration tool outputs a csv table with all user information (inc. plain text password!), so it is a matter of us deciding where and how to import. I thought of importing everything into LDAP (inc. mapping them to the specific group/space of the conference), then OA2 can work as an OpenID provider (https://drupal.org/project/openid_provider) @pierre_ozoux can also probably provide some contribution in this field.
I’m relatively illiterate with cloud structures. I think @zeh is the right person to comment here, since he has started to deploy the new VMs. What I can say is that we cannot scrap at once all servers and go for a fully experiment. We have a lot of sites and users hosted that require stability. But we want - and need to - progressively move our servers to a new model/setup that provides more efficiency and flexibility (we still have a physical machine without any VMs!).
In general, we need manpower for new architecture developments and for the implementation of the applications. At the beginning without good perspectives of payment (well, apart from the revolutionary love), but with a scalable offer it can hopefully generate enough income to support further work on development, investment on new machines, etc… We have a good and diverse user database and a good perspective of expansion, that we can push as soon as we start providing the new generation of services
BTW, we have a permanent meeting room on IRC where you can always come to talk (well, not 24/7 of course, but we do chat occasionaly ;)) #ecobytes @ irc.indymedia.org
I will get back to the rest of questions soon, but for now:
That’s just not true. Have you had a look at https://github.com/hypothesis/h ? The latest release is one year old, but master is very up to date.
Great! Then lets go for it. Waiting for @zeh and @pierre_ozoux to get the new servers and VMs going, then we can start working on this. I have the hope that this will happen today, since both are meeting to work on it
@almereyda do I understand well that you want to take the setup of etherpad + hypothesis on your hand?
This was one of my main arguements for setting up a flexible system. Let me state it like this : I’d love to set those services up, if you in return gave me the chance to work with a OpenStack + OpenShift ecosystem.
I like the idea of openshift on top of Docker. But I believe OpenStack is overkilling for us right now (maybe in the future)
(And just to let you know that I’m quiet new in the boat ecobytes)
Sorry, I have been a bit away, mostly fundraising with two large projects where Ecobytes is partner. If we get them both we have over 100 k € for admin and dev of web services (CSAs and degrowth groups) in the next 3 years. Now I’m a bit more available again.
@pierre_ozoux, now that you have KVM setup (and Openshift? no idea what you have setup, btw we need documentation on our internal wiki on how you have set it up), could you provide @almereyda with 2 VMs so that he can move with the implementation of node.js apps and others we want to use? @almereyda what else would we need? We absolutely need to improve the communication channels. Maybe a meeting on IRC at some point would be good to define a common working procedure, because now we have our internal schleuder list where not all of you are, an internal wiki (ikiwiki) for documentation where I think none of you have access, this tool which most of the older Ecobytes members don’t use.
How does it look like for you a meeting on Sunday or Monday evening? (approx 22:00 CET)
Hi guys! Yes, we’ll meet with gualter tonight at 21h german time! See you there then?
Sorry for not having been with you. After Paris’ OuiShare Labs Camp + Fest and a very cold farm house the week before, I’m now scratching on a Bronchitis and trying not to loose all of my involvements.
I saw some changes on the Trello board, that indicated some movement. Despite the initial refusal of Hypothes.is I’d still be very happy to provide it, next to other lovely Node partners as Ghost and Etherpad, maybe even Wiki.
@pierre_ozoux and the Ecobytes elders ( @zeh I think ), *if I’m not idling on *IRC, you can also find me on Jabber / via XMPP as
email@example.com which should be a little more reliable.
If you grant me access to your documentation(s), I’d be happy to constructively criticize those representations by demanding neccessary information for my work.
@gualter Could you still offer a 2GB/2vCore VM for a Docker-Discourse-Setup? I could provide a little donation as of max. 5€/mth., if that kind of incentive already helps. @pierre_ozoux Docker’ed-OpenShift also sounds lovely. Any success?
Currently I’m in a SPAMergency move of
chn.io until the end of the week and will be available again from then on.
Me also away the whole week, was organising the GROWL course in South France and tomorrow I go back to Germany. I would like to move a lot forward next week with all this Ecobytes stuff. The 2 VMs should be no problem, 5 €/mth is probably ok, it’s actually our membership value (60 €/year) and you are anyway contributing to the ecosystem
Perfect. Let’s finally close this thread next week; and open many more!
I’ve greatly refreshed my knowledge on PaaS and Docker-based systems by reading a little again. There’s different possibilities:
Evaluation of Docker based FOSS PaaS
- First some general slides about Docker
- A beautiful, Bootstrap-based simple UI for managing containers.
- Maestro NG Orchestration
- It relies heavily on components to build a working system. These are:
- Heroku Buildpacks through buildstep
- and Plugins
- Supported platforms through Docker base images
- Cluster support
- Tsuru: plataforma de cloud computing open source should also have a Dashboard, but I didn’t find any screenshots.
Flynn + Deis
- Flynn should be similiar to Tsuru, but the commercial version is bound to commercial services, so I couldn’t try it out.
- Deis seems a very mature product that keeps somehow quite
non-competitive, but notable mentions
- Hipache: a distributed HTTP and websocket proxy
- Mesos, Cluster Computing done right.
- The russian cocaine infrastructure appears somehow different.
Some further readings
- StackOverflow How to scale Docker containers in production (StackOverflow).
- FutureOps with Immutable Infrastructures and Built-in Failure Recovery where I should also mention Serf somehow.
- A presentation how Packer works with Serf.
So I will be basically free to run the VMs as I like, right? Or do you have any constraints?
As long as Flynn is in development and Deis as a powerful orchestration engine, I have decided now we only need a leightweight Docker image management, as the Ecobytes infrastructure is not yet too big. On thise site before you find also another nice overview of Docker tools.
Flynn for example seems to become a somewhat reference implementation, as Dokku’s successor, for a Docker-based PaaS. And they also think of a transparent Database layer.
But I have decided to build on Shipyard first, keeping the two above and tsuru in the eye; literally, as I desperately want UI interfaces. Just for the sake of it.
Another nice read about DevOps Infrastructure can be found at The Twelve-Factor App.
Security is still a bit shaky apperently. But I’m sure it’ll be patch quickly. And we’ll not yet let our users run their own code on our containers, so it’s kind of ok.
Haha, ja, Docker and security; what a joke! You’re right; one has to take extra-special precautions with that. But I think Ecobytes nesting and NATing is already a strong wall.