Ansible and Pagerduty, the Good, the Bad, the Oh Dear Lord Why
How do you manually manage 100+ people, over a dozen teams, and who knows how many services and escalation policies in Pagerduty? Certainly not by hand, you need to automate that. The problem is that not many people have done it yet, or at least done it and open sourced the code.
My team inherited this type of a Pagerduty configuration with the mandate to make is stable with the ability to fork-lift it into another account if needed. This means we needed to automate only the Pagerduty users and other entities we were responsible for but leave everything else alone. Simple, right?
Ansible was the tool chosen for this but no capability existed yet to manage Pagerduty in this way so I set about writing a series of modules to do this. The lessons I learned and problems I had to solve in order to do this properly tell quite a story of how to scope a project effectively, how to handle management that wants a push button solution, how to write something for the community and your company that won't get you laughed at by the everyone, and most importantly...whitespace...lots and lots of whitespace.