[Feature request][Enhancement] Maintaining deployments

[ciucanu]

I'm using this playbook as a base for an offline and secured environment. From the way it works, it looks like it was designed only to deploy clusters. It's not doing anything to maintain them after that. Is there any chance to be developed that way in the future? I'm thinking about features like:
-maintaining service configuration
-creating and maintaining host templates
-detecting changes in the code/templates and applying them

I'm currently working in this direction, but I'm not too efficient due to the fact that I'm kind of new to Ansible and also Cloudera API to some extent. Thanks.

[dbeech]

Hi @ciucanu. That would be a really great feature. One challenge that any user of automation has is how to keep the scripts and templates up to date, making sure that any changes made manually through Cloudera Manager can be brought back into the playbook so that they are replicated whenever the environment is re-created or a new environment is stood up.

Let's discuss more here. Looking forward to seeing your ideas / contributions!

[enlightenalpha]

I have exactly the same need. We currently use Altus Director to deploy clusters and I wrote a Python script to maintain cluster non-default configs post-deployment. This has been serving us well, but it's becoming unwieldy as we deploy more clients and vary our cluster sizes. I could see having to deal with some of the same headaches using Ansible, but I feel the tooling would at least help some. Given that we manage 99% of our server configuration with Ansible, it just makes sense for me to include this into the fold. Would love to be included in the conversation.

[enlightenalpha]

It looks like all of the components are in place to allow for post deployment update of service configs, We would just need to introduce the ability to update / maintain existing cluster configurations via a separate task.

[ciucanu]

Hello,

After spending some time on it I managed to make it work quite nicely. I'll have to check if I can share the code (employer, legal dept, IP....).

In a few words what I have until now:

• three URI tasks for taking some YAML config and sending it to Cloudera API as JSON for CDH services, CM, and Management services
• a script which is doing the API calls to CM to get the same configs from Cloudera in JSON, convert it to YAML and add some info to make it easy in Ansible again.

In the end it works both ways:

• if you update configs from CM UI, you just have to run that script (mostly bash and python for the conversion) and you'll have it in a nice YAML ready for Ansible to parse it
• you can maintain the config in YAML files for ansible and push it to those three URI tasks.

Everything is working, but the code is ugly and it doesn't support multiple clusters.

I'll try to deal with the legal stuff and share it with you.

[dbeech]

Hi @ciucanu and @enlightenalpha. You might be interested to check out a branch I recently pushed: https://github.com/cloudera/cloudera-playbook/tree/director-style-config

I reworked a lot of the host template and configuration code and brought it into a single yaml file (cluster_definition.yml). Like before, this is just for initial cluster creation so it doesn't solve the requirement here totally but if you're familiar with Director then it might be nicer to work with.

@ciucanu - looking forward to seeing what you've done (assuming the legal stuff causes no trouble!)

[lhoss]

Hi @ciucanu and all,
great initiative (that I'm thinking about since early contact with Cloudera Manager.. and earlier with Ambari(API), but somehow had not posted a request)
I already know we will need such functionality as well, probably multiple customers, and I'ld be happy to contribute as well.

ps: Ironically I also need this around Ambari/HDP/HDF deployments (knowing that Ambari has 'no future' since the newgen CDP uses Cloudera Manager), for 1 big customer that needs fully automated deployments (including any Config changes, Service or Nodes additions) for HDP for >1 year (since they cannot update to CDP for some time). For that I should create the respective issue on https://github.com/hortonworks/ansible-hortonworks (I heavily contributed (also))

[ciucanu]

Hello, I'm currently still waiting for approval from my employer but if I don't get that soon I might start writting something similar on my personal cluster.

[serpro69]

Any updates @ciucanu? I was also looking for something similar to what you have described. Would be very interested to see what you have done so far.

[ciucanu]

Still waiting for approval. In essence, what I did were a few api calls using uri module for the following:

• remove obsolete templates because overwriting is not supported (cm_api_url / clusters / cluster_display_name / hostTemplates / hostTemplate_list
• creating the templates based on config files (YAML)
• create groups by host_template
• apply host template
• apply CM custom configs ( cm_api_url / cm / config )
• apply CDH services custom configs
• apply CM management custom configs

I created a shell script to get all the configs from the API for each ( CM, CDH services, CM management services) , and convert from JSON to YAML . With small adjustments, those files are pushed back using the last 3 calls presented here.

I’m writing it here because I don’t have time at this moment to reproduce it outside.

I’ll keep you posted if I get the approval

[serpro69]

Thanks for the reply @ciucanu. I was thinking of doing something similar (though I don't need the UI-to-yml part really as we are planning to prohibit making any changes from UI) using Ansible's URI module and Cloudera's APIs, but I'm not that efficient yet as I'm just getting to know Cloudera and Ansible as I keep ploughing my way through all the mess we have here with manual setup and converting it to automated. Will start working on implementing it from my end while keeping an eye out on this issue for any updates.

[ciucanu]

I think the UI to code export is really useful at least on the initial deployment, since it would take a lot of time to figure out which configs you want to define in the playbook variables/config. There are a lot of settings which are not documented and you cannot guess them until you check them using the API (JSON).

If you have time to start working on it, please let me know if you have any issue, I'll try to follow up and help.