With USC CIO Ilee Rhimes, I co-presented Developing a Comprehensive Disaster-Recovery Plan, as an Association of Pacific Rim Universities (APRU) 2010 Videoconference Session.
USENIX LISA 2008 Birds of a Feather: Documentation and Project Management
I led my fifth LISA BoF in 2008, in San Diego.
While working at University of Southern California and after working at California Institute of Technology, I co-moderated my first system administration workshop at USENIX LISA 2008, with John “Rowan” Littell from California College of the Arts.
LISA 2008 Workshop: University Issues
The focus of this workshop is on issues peculiar to university and college computing shops. Schools vary greatly in their approach to running computing infrastructures. The differences can stem from the general culture of the school as well as upper management, or even from departmental versus institution-wide services.
Part of the goal of this workshop is to communicate what works and what does not work for your institution or your organization within the institution. Topics might include funding, student/faculty/staff needs, research, security, purchasing, staffing, training, working with students, working with research and instructional staff, and even the culture and campus integration of computing facilities and support.
To attend the workshop, please send email to firstname.lastname@example.org with a short paragraph describing your institution, the biggest issue you face today, or something about your institution that works particularly well or that others might want to consider for their own school. You can also include topics you would like to see on the workshop agenda.
I facilitated the Google Apps for Higher Education Roundtable at the EDUCAUSE Western Regional Conference 2008, in San Francisco.
USENIX LISA 2005 Birds of a Feather: Documentation and Project Management
I led my fourth LISA BoF in 2005, in San Diego.
USENIX LISA 2004 Birds of a Feather: Documentation and Project Management
I led my third LISA BoF in 2004, in Atlanta.
USENIX LISA 2001 Birds of a Feather session: Technical Documentation Evangelists
I led my second LISA BoF in 2001, in San Diego.
USENIX LISA 2000 Birds of a Feather session: Documentation for Growing Sysadmin Teams
I led my first LISA BoF in 2000, in New Orleans. This was my first time attending a LISA conference, at the recommendation of and with full support by my amazing mentor-manager at the time.
The following are notes from my informal talk and resulting discussion from the Birds of a Feather (BoF) session. Special thanks to Jessica Cole for typing notes during the BoF (while I projected writing with markers on actual plastic transparencies, my goodness how technology has improved!), and to Mike C for inspiration!
0) Overview of the Presentation
- Baby Steps
- Brain Pages
- Encourage to Write
- Encourage to Use
- Next Steps
- More Ideas?
Each SA is only or best expert for a set of services
- Hide and Seek
- fixes are delayed while the rest of the team looks for the expert following an alarm or complaint
- Documentation by one, for one
- file fragments
- post-it notes
- comments in programs
- Once a set is accumulated, it persists
- The proverbial bus, or headhunter, or vacation
As the group grows, working together should make it more effective…
- Is the team greater or less than the sum of its parts?
- Does the team load-balance and fail-over for high-availability?
- Or does it crash?
- Avoid stepping on toes
- Prevent falling through cracks
- Foster skill building
- peer training
- justify conferences and classes – all internal training opportunities have
already been fully utilized!
3) Baby steps
Creating and Fostering the Documentation Process
- The documentation process goes hand-in-hand with the operations process
- Using the documentation process *will* uncover problems with the operations
- Documentation is one step before automation
- Get both teams and management buy-in
- See proceedings from related and helpful LISA 2000 Tutorials
- Consider designating a documentation project lead (aka Doc Nag, or Documentation
Specialist, or Documentation Evangelist)
A note to our international readers: it was brought to my attention that non-native English speakers don’t know what I meant by the word ‘Nag’, so here’s a definition from Merriam-Webster’s Collegiate Dictionary:
one who nags habitually
Inflected Form(s): nagged; nag·ging
Etymology: probably of Scandinavian origin; akin to Old Norse gnaga to gnaw; akin to Old English gnagan to gnaw
to find fault incessantly, COMPLAIN
to be a persistent source of annoyance or distraction
to irritate by constant scolding or urging, BADGER, WORRY
– nag·ger noun
– nagging adjective
– nag·ging·ly /’na-gi[ng]-lE/ adverb
Project Leader Tasks
- Choose a format
- HTML with example templates
- Manually updated index file
- Staff-only advertising / limit access
- Full text search
- Document rollout via one peer review
- Brain page theme – humor, commonly played card game in our team’s spare time
4) Our First Attempt: Brain Pages
- Page Title
- Alludes to scope and target audience… for instance, All About pages may include:
- services overview
- service level statement
- reasoning behind service decisions
- Some example titles:
- All About foo
- Debugging foo
- Adding/Editing/Upgrading foo
- Alludes to scope and target audience… for instance, All About pages may include:
- Common Fields
- last edit date
- table of contents
- links to user docs and brain docs
- procedure (explain everything, even down to permission details)
- who can execute
- when should execute
- known problems
- known exceptions
- gaps in service or documentation (being partially done is no excuse not to document at all!)
- revision history
- contact email address for requests for changes/additions/deletions to document
- “was this document helpful to you?” ask for feedback on the pages
5) Encourage to Write
How to get them written?
- Doc Nag as ghost writer
- Doc Nag as editor
- offer to help collect data from post-its, email, code notes, etc to help take it off their hands
- program called “script” to capture all procedures in a logfile for documentation
- Doc Nag as shepherd through peer review (example: take the reviewing peer out for coffee so they have time to read over the doc without interruptions)
- Try the procedure! Make sure it works!
- Hand of pieces of the documentation process to Jr SAs
- watch and write and/or collect and edit material from Sr SAs
- signoff by Sr SAs is required!
- call it in-house training!
5) Encourage to Use
- Tie into alarms
- trouble tickets link to documentation on how to fix the alarmed problem
- Peer review increases awareness of the documentation project
- positive (note that it can help reduce their calls/emails for problems)
- negative (one person hasn’t documented when everybody else has)
- Carrots from management
- Ask for flex time to go away for documentation, or after finishing some docs
- Ask to work from home or outside under a tree with a laptop for a few hours a week to write
- Ask for either or both of those for your Jr SAs in training that are transcribing your illegible notes into documented procedures
- When one service is rolled out and documentation finished, ask for a new and interesting project that you really want to do, now that you’ve handed maintenance of the service off to the team!
- Quantify vs. anecdotes (measure hits on website pages, measure tickets solved by docs)
- financial or small awards as motivation; annual review, salary negotiations, flex-time
6) Next Steps
- More automation
- Better indexing/searching
- Revision control and history
- Contribute/share knowledge via the SAGE and LOPSA mailing lists
7) Additional ideas?
The items below came up in our 20 minute open discussion… and the list
will continue to grow as more people contribute, as least until I get around
to better categorizing it!
- When starting a doc project, keep the framework as simple as possible. Starting with a huge Oracle backend database and customizable submission tool is biting off too much at once!
- Use professional writing style…
- Don’t be cute at the expense of being informative.
- Use the SEE format: Statement, Explanation, Example, just as Michael said in his Tutorial session!
- One tangible metric for judging the effectiveness, and to quantify the benefits, of a documentation system is the rollout time for new sysadmins!
- Check out the Babble paper in the LISA 2000 Proceedings
- POD – Plain Old Documentation
- Test the usability of each document by having a Jr SA do it while the creator watches silently – make sure both understand expectations for this nerve-wracking exercise!
- The program called “script” to capture all procedures in a logfile for
documentation is really useful
- Give power to people executing the script to change and/or recommend changes to the document (otherwise you will have an undocumented modification of docs culture among your Jr SAs)
- When the procedure does not work as documented, page and/or call the maintainer sysadmin, even if it’s 3:00 AM! That won’t have to happen many times before the document gets fixed!
- HTML cleanup tool for from Frontpage and Word – w3c.org and Dreamweaver
- w3m has an HTML to ASCII text plus color converter
- Set up a mail alias to send all requests for changes or additions to the documentation (and/or cc the email address to the docs project coordinator)
for anything that would be helpful
- If you don’t have a better system within reach, set up an alias and a hypermail archive for procedures and fixes documentation
- Set a valid-until date on each document; renew it by peer review by maintainer and executors on some regular interval
- Also accept documentation requests from other groups/team members
- Consider web-based interface to submit to documentation
- WIKI or SWIKI
- PHP annotated manual pages
- Don’t force authors to deal with specialized authoring interface or tools if they don’t want to
- XML tools convert easily to other flavors
- Tie it more closely to the ticket system
- SDF (built on POD) Simple Document Format, builds a Table of Contents automatically
- Flat text files are a valid good start, and maybe the ultimately-portable format
- Avoid editor wars
- Realize there is a distinction between getting data and storing data in a documentation project
- cfengine or script must be able to recreate service – as condition of a service being “production” – has reduced the volume of documentation at one company
- ZOPE may be interesting and/or helpful
I presented “X-Ray Crystallographic Structure Determination of Hyperthermophilic Proteins II”, based on my Summer Undergraduate Research Fellowship in protein chemistry, at the Caltech SURF Seminar Day 1994.
The goal of my continuing research was to accomplish, via hanging-drop method, the growth of protein crystals suitable for x-ray diffraction characterization of its three-dimensional structure. I was working with a ferredoxin which is involved in electron transfer, isolated from the hyperthermophile Thermotoga maritima. I also grew crystals of a much more common mesophilic lysozyme protein which were used to help calibrate the x-ray diffraction equipment.
My research mentor was Douglas C. Rees at Caltech. Special thanks to the Rees Research Group at Caltech, Michael W. W. Adams at University of Georgia, the SURF Program, the Richter Foundation, and the Caltech Alumni Association.
Abstract and final paper available upon request. Abstract, final paper, and slides will be uploaded here as soon as I figure out a good way of digitizing actual physical transparency projector slides.