Shrinking a Linux LVM on top of an md RAID1

If I don’t start blogging some of the stuff I’m doing, I’ll never remember how I did it in the first place!  So, time to try to come out from the darkness and see if I can keep discipline to post more.

Problem:  One of our servers has a Linux md RAID1 device (md1), which would keep falling out of sync, especially after a reboot.  Upon nearing the end of they re-sync, it would pause, fail, and start over.  Entries from /var/log/messages

Dec 22 11:56:52 xen-33-18-02 kernel: md: md1: sync done.
Dec 22 11:56:56 xen-33-18-02 kernel: ata1: soft resetting port
Dec 22 11:56:56 xen-33-18-02 kernel: ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Dec 22 11:56:56 xen-33-18-02 kernel: ata1.00: configured for UDMA/133
Dec 22 11:56:56 xen-33-18-02 kernel: ata1: EH complete
:
Dec 22 11:56:56 xen-33-18-02 kernel: SCSI device sda: 1465149168 512-byte hdwr sectors (750156 MB)
Dec 22 11:56:56 xen-33-18-02 kernel: sda: Write Protect is off
Dec 22 11:56:56 xen-33-18-02 kernel: SCSI device sda: drive cache: write back
Dec 22 11:57:04 xen-33-18-02 kernel: ata1.00: configured for UDMA/133
Dec 22 11:57:04 xen-33-18-02 kernel: sd 0:0:0:0: SCSI error: return code = 0x08000002
Dec 22 11:57:04 xen-33-18-02 kernel: sda: Current: sense key: Aborted Command
Dec 22 11:57:04 xen-33-18-02 kernel:     Additional sense: No additional sense information
Dec 22 11:57:04 xen-33-18-02 kernel: end_request: I/O error, dev sda, sector 1465143264
Dec 22 11:57:04 xen-33-18-02 kernel: sd 0:0:0:0: SCSI error: return code = 0x08000002
:
Dec 22 11:57:14 xen-33-18-02 kernel: sd 0:0:0:0: SCSI error: return code = 0x08000002
Dec 22 11:57:14 xen-33-18-02 kernel: sda: Current: sense key: Medium Error
Dec 22 11:57:14 xen-33-18-02 kernel:     Additional sense: Unrecovered read error - auto reallocate failed
:

Then, the RAID1 re-sync would start all over again … only to fail a few hours later … and again.

Since there appear to be some bad spots on sda at the end of the disk, the solution seemed to be to shrink the partition a bit to avoid the bad spots.  

Our md RAID1 (md1) consists of 2 x 694GB partitions on sda3 and sdb3.  On top of md1 lives an LVM PV, VG, and lots of LVs.  The PV is not 100% allocated, so we have some wiggle room to shrink everything.

Here’s the procedure we followed:

1. Shrink the LVM PV
   • “pvdisplay” showed the physical disk had 693.98 GB, so we just trimmed it down to an even 693GB.
   • use “pvresize” to shrink the PV

   # pvresize  -v --setphysicalvolumesize 693G /dev/md1      Using physical volume(s) on command line
       Archiving volume group "datavg" metadata (seqno 42).
       /dev/md1: Pretending size is 1453326336 not 1455376384 sectors.
       Resizing physical volume /dev/md1 from 177658 to 177407 extents.
       Resizing volume "/dev/md1" to 1453325952 sectors.
       Updating physical volume "/dev/md1"
       Creating volume group backup "/etc/lvm/backup/datavg" (seqno 43).
       Physical volume "/dev/md1" changed
       1 physical volume(s) resized / 0 physical volume(s) not resized

2. Shrink the md RAID1 device
   • If you do NOT do this and shrink the hard drive partitions beneath the md device … you loose your superblock!  Cool, eh?
   • Check the size of the existing md

   # mdadm --verbose --detail /dev/md1
   /dev/md1:
           Version : 00.90.03
     Creation Time : Mon Sep 29 23:41:30 2008
        Raid Level : raid1
        Array Size : 727688192 (693.98 GiB 745.15 GB)
     Used Dev Size : 727688192 (693.98 GiB 745.15 GB)
      Raid Devices : 2
     Total Devices : 1
   Preferred Minor : 1
       Persistence : Superblock is persistent
   
       Update Time : Mon Dec 22 22:57:31 2008
             State : clean, degraded
    Active Devices : 1
   Working Devices : 1
    Failed Devices : 0
     Spare Devices : 0
   
              UUID : d584a880:d77024a2:9ae023b4:27ec0db5
            Events : 0.5501090
   
       Number   Major   Minor   RaidDevice State
          0       8        3        0      active sync   /dev/sda3
          1       0        0        1      removed

   • Hey, what a coincidence … the Array Size is the same as the PV was!  Now, we need to calculate the new size.  The new size of the PV is 1453325952 sectors, which is 726662976 blocks (even *you* can divide by 2!!).
   • Shrink the md:

   # mdadm --verbose --grow /dev/md1 --size=726662976
   # mdadm --verbose --detail /dev/md1
   /dev/md1:
      :
        Array Size : 726662976 (693.00 GiB 744.10 GB)

   • Woot! Getting closer!
3. Now, technically, we should shrink the partition, too.  Here’s where I ran into a bit of trouble (ok, I’m too lazy to do the math).  ”fdisk” on linux doesn’t seem to want to let you specify a size in blocks or sectors, so you have to keep shrinking the ending cylinder number until you get in the range of the new block size.  I’ll leave this as an exercise for the reader, and feel free to post a comment with the actual procedure.  :)

After these steps were completed, we were able to perform the raid1 re-sync successfully.

Things Are Not Always What They Seem

OK, it’s been way too long since my last post … much going on, but this post will be short and about something else!

I was shopping at Sam’s Club the other day and in the freezer section, I came across a “seafood mixture” … ideal for Paella!  Yum!

So, what’s the problem?  GF Paella should be pretty easy, right (yellow rice + chicken (in our version) + seafood + some GF safe flavoring sauces) … nothing illegal there.  If you are doing CF, make sure you don’t add any butter or any other flavoring agent that would be illegal.

Here’s the problem – out of habit, I checked the ingredients list.  Yeah, the ingredients list on a bag of mixed seafood … how exciting could that be?  INCLUDES WHEAT!  What???  Yep, nearly made something deadly for my grandmother (86, frail, and a Celiac).  <sigh>

Anyway, today’s lesson is READ THE INGREDIENTS ON EVERYTHING.  No matter how trivial it may seem.

The Butterfly Pavilion

I found The Denver Digital Photography Meetup Group and went to the meetup at the Butterfly Pavilion and it was a nice trip. If there was any organized group stuff during or after (besides heading to the Rock Bottom Brewery after for lunch, but I don’t care for their food), I must have missed it. It would be nice to talk with other photographers and exchange tips and get some feedback on my techniques, etc. Maybe next meetup.

I was experimenting with a circular polarizer, but the side effect is I can’t see very well through the lens, so a lot of my shots were out of focus. I’ve also found that the autofocus doesn’t always catch what I want it to, so I tried manual mode … still not so great.

So, out of the 238 shots (that made it home), I have 15 below that were decent enough to look at again. More than I expected, actually, considering I was doing a *lot* of multiple shots and making minor tweaks between each shot (trying to get better with managing my exposure stops with the polarizer, etc).

Hopefully some of these look OK … feedback is always welcome.

752×500 (1/60) f/5.6 f(35)=300mm (flash)	752×500 (1/60) f/5.6 f(35)=300mm (flash)
752×500 (1/60) f/5.6 f(35)=300mm (flash)	752×500 (1/60) f/5.6 f(35)=300mm (flash)
752×500 (1/15) f/5.6 f(35)=300mm	752×500 (1/60) f/5.6 f(35)=142mm (flash)
752×500 (1/125) f/5.6 f(35)=300mm	752×500 (1/125) f/5.6 f(35)=157mm
752×500 (1/320) f/5.6 f(35)=300mm	752×500 (1/80) f/5.6 f(35)=300mm
752×500 (1/80) f/5.6 f(35)=142mm
500×752 (1/60) f/5.6 f(35)=187mm	500×752 (1/60) f/5.6 f(35)=142mm (flash)
500×752 (1/125) f/5.6 f(35)=187mm	500×752 (1/60) f/5.6 f(35)=142mm (flash)

These are all re-sized down from JPG-Basic (and then re-sized again for the thumbnails), and are untouched from the camera. I’ve not post-processed anything yet. I do have RAW (Nikon) versions of each of these as well.

Proof That Autism Can Be Defeated!

Ok, maybe not completely defeated, but the news we got today just reaffirms my belief that hard work, building a great support team, and aggressive therapy can make the difference. That said, did we do everything possible? I don’t think so, but then again, if you think you’ve done it all, maybe you are missing something else. To put it in a sports metaphor, go 110% and leave it all on the field … but know that there is always something more you can challenge yourself to do better next round.

We followed up with J’s developmental evaluation (from early December 2007) with the therapists today. The ladies at Developmental FX are just awesome, and they really care about what they do. They did his developmental evaluation back in August 2006, where they couldn’t even do age appropriate testing (he was physically 4 years old) and pegged his development at 22 to 29 months of age (mental, speech, physical, etc). Keep in mind that those numbers were similar to the speech eval that was done at University of New Mexico Hospital in March 2005, and he had made _progress_ in the 3 months with us prior to his 2006 eval. Scary, eh?

Well, we have the “final” report, although we would like to work in inputs from J’s biological parents (where he is now), but they still haven’t submitted their forms or made an appointment to meet with Developmental FX yet. In a nutshell, Developmental FX was very happy (and maybe a little surprised) at how much J has grown. They now estimate him to be around the 52 month (4 years, 4 months) developmental stage. At the time of the eval, that puts him only about a year behind!

Through hard work, a great support team (both professional and personal), LOTS of positive re-enforcement, education (for us and J), a (very) little medication, the GF/CF diet, and just plain challenging him to develop and grow, we managed to do over 2 years of developmental growth in about 15 months. Something like 176% growth and developmental rate? I’ll take that! I think his rapid ascent makes it pretty obvious that _he_ was not the problem, but rather it was his old environment and lack of a good support team. Reading his old reports, he had some pretty good early intervention support, but somehow, it just didn’t stick.

Could *we* have done better? Probably! We did have quite a few hiccups along the way, and other external events that made it difficult for him to focus and increased his overall anxiety. As we got closer to what became the transition back to his parents, his anxiety level was visibly high — huge increases in his door fixation, thumb sucking back in full force, even some behaviors that were completely gone came back (like the head shaking) and more frequent hand flapping. He even lost some of his language and looked “lost” more often. It’s really sad to see a child fall apart right in front of your eyes.

Naturally, this report is bittersweet. We are very proud of the work we have done, the progress we’ve made together, and the foundation we’ve helped him build. At the same time, we can only hope that he continues to get the same support and dedication that we’ve given him over the last 20 months. Now that we are no longer directly in the picture, we have to rely on information that trickles back from his support team as well as others who we have crossed paths with and have taken an interest in him (he’s a really likable kid). As much as we’d like to continue to be involved in his day-to-day, the emotional trauma and mental distress that bouncing between the households was causing would more than counteract any benefit we could add — and that’s something that we think and grieve about every day.

We still believe that J has the ability to be mainstreamed, but he still has a long way to go. We know that his parents are busy and work and all of that, but we hope they can find a way to keep giving him the attention he will need to succeed and have a quality life (meanwhile, not ignoring his little sister, A). If they don’t help him succeed, it will be everyone’s loss — he’s a bright kid and has a lot to offer.

The take away from this? If you have a child with autism, DO NOT GIVE UP! Anything can happen, and if you WORK HARD and be open to new ideas, talk to your community and support professionals, build a network (personal and professional), you have a great chance of giving your child the ultimate gift — a quality, and independent life. There are no “do-overs” or “I’ll do it tomorrow”, only do it now. To quote our Jedi master Yoda, “there is no try, only do”.

Seriously, though, you are your child’s only true advocate. You may build a fantastic support network, but they are only there to support you help you and your child. The effort you put in will bring you rewards that money cannot buy.

Whether it is or isn’t someone’s fault that your child has autism, or that $deity is to blame or whatever is all TOTALLY IRRELEVANT. The issue at hand is WHAT ARE YOU GOING TO DO NOW! And that can only be TAKE ACTION. Time is running out, so get to it.

We did it in a short time, with no forewarning, no training, heck, not even a clear timeline or future. If we can, so can you.

Love your children and work your a$$ off now … you can always sleep next year.