USENIX LISA 2000 Birds of a Feather session: Documentation for Growing Sysadmin Teams

I led my first LISA BoF in 2000, in New Orleans.  This was my first time attending a LISA conference, at the recommendation of and with full support by my amazing mentor-manager at the time.

The following are notes from my informal talk and resulting discussion from the Birds of a Feather (BoF) session. Special thanks to Jessica Cole for typing notes during the BoF (while I projected writing with markers on actual plastic transparencies, my goodness how technology has improved!), and to Mike C for inspiration!

0) Overview of the Presentation

1. Background/Problem
2. Goals
3. Baby Steps
4. Brain Pages
5. Encourage to Write
6. Encourage to Use
7. Next Steps
8. More Ideas?

1) Background/Problem
Each SA is only or best expert for a set of services

• Hide and Seek
  • fixes are delayed while the rest of the team looks for the expert following an alarm or complaint
• Documentation by one, for one
  • email
  • file fragments
  • post-it notes
  • palms
  • comments in programs
• Once a set is accumulated, it persists
• The proverbial bus, or headhunter, or vacation

As the group grows, working together should make it more effective…

• Is the team greater or less than the sum of its parts?
• Does the team load-balance and fail-over for high-availability?
• Or does it crash?

2) Goals
Coordination

• Avoid stepping on toes
• Prevent falling through cracks
• Foster skill building
  • mentoring
  • peer training
  • justify conferences and classes – all internal training opportunities have
    already been fully utilized!

3) Baby steps
Creating and Fostering the Documentation Process

• The documentation process goes hand-in-hand with the operations process
• Using the documentation process *will* uncover problems with the operations
  process
• Documentation is one step before automation
• Get both teams and management buy-in
• See proceedings from related and helpful LISA 2000 Tutorials
  • S16
  • M11
  • M14
  • T11
  • T14
• Consider designating a documentation project lead (aka Doc Nag, or Documentation
  Specialist, or Documentation Evangelist)

A note to our international readers: it was brought to my attention that non-native English speakers don’t know what I meant by the word ‘Nag’, so here’s a definition from Merriam-Webster’s Collegiate Dictionary:
A nag…

noun
one who nags habitually

To nag…

verb
Inflected Form(s): nagged; nag·ging
Etymology: probably of Scandinavian origin; akin to Old Norse gnaga to gnaw; akin to Old English gnagan to gnaw
Intransitive Senses:
to find fault incessantly, COMPLAIN
to be a persistent source of annoyance or distraction
Transitive Senses:
to irritate by constant scolding or urging, BADGER, WORRY
– nag·ger noun
– nagging adjective
– nag·ging·ly /’na-gi[ng]-lE/ adverb

Project Leader Tasks

• Choose a format
  • HTML with example templates
  • Manually updated index file
  • Staff-only advertising / limit access
  • Full text search
  • Document rollout via one peer review
  • Brain page theme – humor, commonly played card game in our team’s spare time

4) Our First Attempt: Brain Pages

• Page Title
  • Alludes to scope and target audience… for instance, All About pages may include:
    • services overview
    • service level statement
    • alarms
    • policy
    • reasoning behind service decisions
  • Some example titles:
    • All About foo
    • Debugging foo
    • Adding/Editing/Upgrading foo
• Common Fields
  • author
  • maintainer
  • last edit date
  • table of contents
  • links to user docs and brain docs
  • procedure (explain everything, even down to permission details)
  • who can execute
  • when should execute
  • known problems
  • known exceptions
  • gaps in service or documentation (being partially done is no excuse not to document at all!)
  • revision history
  • contact email address for requests for changes/additions/deletions to document
  • “was this document helpful to you?” ask for feedback on the pages

5) Encourage to Write
How to get them written?

• Doc Nag as ghost writer
• Doc Nag as editor
  • offer to help collect data from post-its, email, code notes, etc to help take it off their hands
  • program called “script” to capture all procedures in a logfile for documentation
• Doc Nag as shepherd through peer review (example: take the reviewing peer out for coffee so they have time to read over the doc without interruptions)
• Try the procedure! Make sure it works!
• Hand of pieces of the documentation process to Jr SAs
  • watch and write and/or collect and edit material from Sr SAs
  • signoff by Sr SAs is required!
  • images/photos/diagrams
  • call it in-house training!

5) Encourage to Use

• Tie into alarms
  • trouble tickets link to documentation on how to fix the alarmed problem
• Peer review increases awareness of the documentation project
  • positive (note that it can help reduce their calls/emails for problems)
  • negative (one person hasn’t documented when everybody else has)
• Carrots from management
  • Ask for flex time to go away for documentation, or after finishing some docs
  • Ask to work from home or outside under a tree with a laptop for a few hours a week to write
  • Ask for either or both of those for your Jr SAs in training that are transcribing your illegible notes into documented procedures
  • When one service is rolled out and documentation finished, ask for a new and interesting project that you really want to do, now that you’ve handed maintenance of the service off to the team!
• Quantify vs. anecdotes (measure hits on website pages, measure tickets solved by docs)
  • financial or small awards as motivation; annual review, salary negotiations, flex-time

6) Next Steps

• More automation
• Better indexing/searching
• Revision control and history
• Contribute/share knowledge via the SAGE and LOPSA mailing lists

7) Additional ideas?
The items below came up in our 20 minute open discussion… and the list
will continue to grow as more people contribute, as least until I get around
to better categorizing it!

• When starting a doc project, keep the framework as simple as possible. Starting with a huge Oracle backend database and customizable submission tool is biting off too much at once!
• Use professional writing style…
• Don’t be cute at the expense of being informative.
• Use the SEE format: Statement, Explanation, Example, just as Michael said in his Tutorial session!
• One tangible metric for judging the effectiveness, and to quantify the benefits, of a documentation system is the rollout time for new sysadmins!
• Check out the Babble paper in the LISA 2000 Proceedings
• POD – Plain Old Documentation
• Test the usability of each document by having a Jr SA do it while the creator watches silently – make sure both understand expectations for this nerve-wracking exercise!
• The program called “script” to capture all procedures in a logfile for
  documentation is really useful
• Give power to people executing the script to change and/or recommend changes to the document (otherwise you will have an undocumented modification of docs culture among your Jr SAs)
• When the procedure does not work as documented, page and/or call the maintainer sysadmin, even if it’s 3:00 AM! That won’t have to happen many times before the document gets fixed!
• HTML cleanup tool for from Frontpage and Word – w3c.org and Dreamweaver
• w3m has an HTML to ASCII text plus color converter
• Set up a mail alias to send all requests for changes or additions to the documentation (and/or cc the email address to the docs project coordinator)
  for anything that would be helpful
• If you don’t have a better system within reach, set up an alias and a hypermail archive for procedures and fixes documentation
• Set a valid-until date on each document; renew it by peer review by maintainer and executors on some regular interval
• Also accept documentation requests from other groups/team members
• Consider web-based interface to submit to documentation
• FAQ-o-Matic
• WIKI or SWIKI
• PHP annotated manual pages
• Don’t force authors to deal with specialized authoring interface or tools if they don’t want to
• XML tools convert easily to other flavors
• Tie it more closely to the ticket system
• SDF (built on POD) Simple Document Format, builds a Table of Contents automatically
• Flat text files are a valid good start, and maybe the ultimately-portable format
• Avoid editor wars
• Realize there is a distinction between getting data and storing data in a documentation project
• cfengine or script must be able to recreate service – as condition of a service being “production” – has reduced the volume of documentation at one company
• ZOPE may be interesting and/or helpful