CENIC 2010 Conference Session: Developing a Comprehensive Disaster-Recovery Plan

With Senior Communications Engineer Ken Mills, I co-presented a Corporation for Education Network Initiatives in California (CENIC) 2010 Conference Session: Developing a Comprehensive Disaster-Recovery Plan.

Slides are available via the CENIC 2010 conference website.

LISA 2008 Workshop: University Issues

While working at University of Southern California and after working at California Institute of Technology, I co-moderated my first system administration workshop at USENIX LISA 2008, with John “Rowan” Littell from California College of the Arts.

LISA 2008 Workshop: University Issues

The focus of this workshop is on issues peculiar to university and college computing shops. Schools vary greatly in their approach to running computing infrastructures. The differences can stem from the general culture of the school as well as upper management, or even from departmental versus institution-wide services.

Part of the goal of this workshop is to communicate what works and what does not work for your institution or your organization within the institution. Topics might include funding, student/faculty/staff needs, research, security, purchasing, staffing, training, working with students, working with research and instructional staff, and even the culture and campus integration of computing facilities and support.

To attend the workshop, please send email to lisa08ws-univissues@usenix.org with a short paragraph describing your institution, the biggest issue you face today, or something about your institution that works particularly well or that others might want to consider for their own school. You can also include topics you would like to see on the workshop agenda.

LISA 2000 Birds of a Feather: Documentation for Growing Sysadmin Teams

USENIX LISA 2000 Birds of a Feather session: Documentation for Growing Sysadmin Teams

I led my first LISA BoF in 2000, in New Orleans.  This was my first time attending a LISA conference, at the recommendation of and with full support by my amazing mentor-manager at the time.

The following are notes from my informal talk and resulting discussion from the Birds of a Feather (BoF) session. Special thanks to Jessica Cole for typing notes during the BoF (while I projected writing with markers on actual plastic transparencies, my goodness how technology has improved!), and to Mike C for inspiration!

0) Overview of the Presentation

  1. Background/Problem
  2. Goals
  3. Baby Steps
  4. Brain Pages
  5. Encourage to Write
  6. Encourage to Use
  7. Next Steps
  8. More Ideas?

1) Background/Problem
Each SA is only or best expert for a set of services

  • Hide and Seek
    • fixes are delayed while the rest of the team looks for the expert following an alarm or complaint
  • Documentation by one, for one
    • email
    • file fragments
    • post-it notes
    • palms
    • comments in programs
  • Once a set is accumulated, it persists
  • The proverbial bus, or headhunter, or vacation

As the group grows, working together should make it more effective…

  • Is the team greater or less than the sum of its parts?
  • Does the team load-balance and fail-over for high-availability?
  • Or does it crash?

2) Goals

  • Avoid stepping on toes
  • Prevent falling through cracks
  • Foster skill building
    • mentoring
    • peer training
    • justify conferences and classes – all internal training opportunities have
      already been fully utilized!

3) Baby steps
Creating and Fostering the Documentation Process

  • The documentation process goes hand-in-hand with the operations process
  • Using the documentation process *will* uncover problems with the operations
  • Documentation is one step before automation
  • Get both teams and management buy-in
  • See proceedings from related and helpful LISA 2000 Tutorials
    • S16
    • M11
    • M14
    • T11
    • T14
  • Consider designating a documentation project lead (aka Doc Nag, or Documentation
    Specialist, or Documentation Evangelist)

A note to our international readers: it was brought to my attention that non-native English speakers don’t know what I meant by the word ‘Nag’, so here’s a definition from Merriam-Webster’s Collegiate Dictionary:
A nag…

one who nags habitually

To nag…

Inflected Form(s): nagged; nag·ging
Etymology: probably of Scandinavian origin; akin to Old Norse gnaga to gnaw; akin to Old English gnagan to gnaw
Intransitive Senses:
to find fault incessantly, COMPLAIN
to be a persistent source of annoyance or distraction
Transitive Senses:
to irritate by constant scolding or urging, BADGER, WORRY
– nag·ger noun
– nagging adjective
– nag·ging·ly /’na-gi[ng]-lE/ adverb

Project Leader Tasks

  • Choose a format
    • HTML with example templates
    • Manually updated index file
    • Staff-only advertising / limit access
    • Full text search
    • Document rollout via one peer review
    • Brain page theme – humor, commonly played card game in our team’s spare time

4) Our First Attempt: Brain Pages

  • Page Title
    • Alludes to scope and target audience… for instance, All About pages may include:
      • services overview
      • service level statement
      • alarms
      • policy
      • reasoning behind service decisions
    • Some example titles:
      • All About foo
      • Debugging foo
      • Adding/Editing/Upgrading foo
  • Common Fields
    • author
    • maintainer
    • last edit date
    • table of contents
    • links to user docs and brain docs
    • procedure (explain everything, even down to permission details)
    • who can execute
    • when should execute
    • known problems
    • known exceptions
    • gaps in service or documentation (being partially done is no excuse not to document at all!)
    • revision history
    • contact email address for requests for changes/additions/deletions to document
    • “was this document helpful to you?” ask for feedback on the pages

5) Encourage to Write
How to get them written?

  • Doc Nag as ghost writer
  • Doc Nag as editor
    • offer to help collect data from post-its, email, code notes, etc to help take it off their hands
    • program called “script” to capture all procedures in a logfile for documentation
  • Doc Nag as shepherd through peer review (example: take the reviewing peer out for coffee so they have time to read over the doc without interruptions)
  • Try the procedure! Make sure it works!
  • Hand of pieces of the documentation process to Jr SAs
    • watch and write and/or collect and edit material from Sr SAs
    • signoff by Sr SAs is required!
    • images/photos/diagrams
    • call it in-house training!

5) Encourage to Use

  • Tie into alarms
    • trouble tickets link to documentation on how to fix the alarmed problem
  • Peer review increases awareness of the documentation project
    • positive (note that it can help reduce their calls/emails for problems)
    • negative (one person hasn’t documented when everybody else has)
  • Carrots from management
    • Ask for flex time to go away for documentation, or after finishing some docs
    • Ask to work from home or outside under a tree with a laptop for a few hours a week to write
    • Ask for either or both of those for your Jr SAs in training that are transcribing your illegible notes into documented procedures
    • When one service is rolled out and documentation finished, ask for a new and interesting project that you really want to do, now that you’ve handed maintenance of the service off to the team!
  • Quantify vs. anecdotes (measure hits on website pages, measure tickets solved by docs)
    • financial or small awards as motivation; annual review, salary negotiations, flex-time

6) Next Steps

  • More automation
  • Better indexing/searching
  • Revision control and history
  • Contribute/share knowledge via the SAGE and LOPSA mailing lists

7) Additional ideas?
The items below came up in our 20 minute open discussion… and the list
will continue to grow as more people contribute, as least until I get around
to better categorizing it!

  • When starting a doc project, keep the framework as simple as possible. Starting with a huge Oracle backend database and customizable submission tool is biting off too much at once!
  • Use professional writing style…
  • Don’t be cute at the expense of being informative.
  • Use the SEE format: Statement, Explanation, Example, just as Michael said in his Tutorial session!
  • One tangible metric for judging the effectiveness, and to quantify the benefits, of a documentation system is the rollout time for new sysadmins!
  • Check out the Babble paper in the LISA 2000 Proceedings
  • POD – Plain Old Documentation
  • Test the usability of each document by having a Jr SA do it while the creator watches silently – make sure both understand expectations for this nerve-wracking exercise!
  • The program called “script” to capture all procedures in a logfile for
    documentation is really useful
  • Give power to people executing the script to change and/or recommend changes to the document (otherwise you will have an undocumented modification of docs culture among your Jr SAs)
  • When the procedure does not work as documented, page and/or call the maintainer sysadmin, even if it’s 3:00 AM! That won’t have to happen many times before the document gets fixed!
  • HTML cleanup tool for from Frontpage and Word – w3c.org and Dreamweaver
  • w3m has an HTML to ASCII text plus color converter
  • Set up a mail alias to send all requests for changes or additions to the documentation (and/or cc the email address to the docs project coordinator)
    for anything that would be helpful
  • If you don’t have a better system within reach, set up an alias and a hypermail archive for procedures and fixes documentation
  • Set a valid-until date on each document; renew it by peer review by maintainer and executors on some regular interval
  • Also accept documentation requests from other groups/team members
  • Consider web-based interface to submit to documentation
  • FAQ-o-Matic
  • PHP annotated manual pages
  • Don’t force authors to deal with specialized authoring interface or tools if they don’t want to
  • XML tools convert easily to other flavors
  • Tie it more closely to the ticket system
  • SDF (built on POD) Simple Document Format, builds a Table of Contents automatically
  • Flat text files are a valid good start, and maybe the ultimately-portable format
  • Avoid editor wars
  • Realize there is a distinction between getting data and storing data in a documentation project
  • cfengine or script must be able to recreate service – as condition of a service being “production” – has reduced the volume of documentation at one company
  • ZOPE may be interesting and/or helpful

SURF Seminar Day 1994: X-Ray Crystallographic Structure Determination of Hyperthermophilic Proteins II

I presented “X-Ray Crystallographic Structure Determination of Hyperthermophilic Proteins II”, based on my Summer Undergraduate Research Fellowship in protein chemistry, at the Caltech SURF Seminar Day 1994.

The goal of my continuing research was to accomplish, via hanging-drop method, the growth of protein crystals suitable for x-ray diffraction characterization of its three-dimensional structure. I was working with a ferredoxin which is involved in electron transfer, isolated from the hyperthermophile Thermotoga maritima. I also grew crystals of a much more common mesophilic lysozyme protein which were used to help calibrate the x-ray diffraction equipment.

My research mentor was Douglas C. Rees at Caltech.  Special thanks to the Rees Research Group at Caltech, Michael W. W. Adams at University of Georgia, the SURF Program, the Richter Foundation, and the Caltech Alumni Association.

Abstract and final paper available upon request.  Abstract, final paper, and slides will be uploaded here as soon as I figure out a good way of digitizing actual physical transparency projector slides.