Shrinking a Linux LVM on top of an md RAID1

December 23, 2008

If I don’t start blogging some of the stuff I’m doing, I’ll never remember how I did it in the first place!  So, time to try to come out from the darkness and see if I can keep discipline to post more.

Problem:  One of our servers has a Linux md RAID1 device (md1), which would keep falling out of sync, especially after a reboot.  Upon nearing the end of they re-sync, it would pause, fail, and start over.  Entries from /var/log/messages

Dec 22 11:56:52 xen-33-18-02 kernel: md: md1: sync done.
Dec 22 11:56:56 xen-33-18-02 kernel: ata1: soft resetting port
Dec 22 11:56:56 xen-33-18-02 kernel: ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Dec 22 11:56:56 xen-33-18-02 kernel: ata1.00: configured for UDMA/133
Dec 22 11:56:56 xen-33-18-02 kernel: ata1: EH complete
Dec 22 11:56:56 xen-33-18-02 kernel: SCSI device sda: 1465149168 512-byte hdwr sectors (750156 MB)
Dec 22 11:56:56 xen-33-18-02 kernel: sda: Write Protect is off
Dec 22 11:56:56 xen-33-18-02 kernel: SCSI device sda: drive cache: write back
Dec 22 11:57:04 xen-33-18-02 kernel: ata1.00: configured for UDMA/133
Dec 22 11:57:04 xen-33-18-02 kernel: sd 0:0:0:0: SCSI error: return code = 0x08000002
Dec 22 11:57:04 xen-33-18-02 kernel: sda: Current: sense key: Aborted Command
Dec 22 11:57:04 xen-33-18-02 kernel:     Additional sense: No additional sense information
Dec 22 11:57:04 xen-33-18-02 kernel: end_request: I/O error, dev sda, sector 1465143264
Dec 22 11:57:04 xen-33-18-02 kernel: sd 0:0:0:0: SCSI error: return code = 0x08000002
Dec 22 11:57:14 xen-33-18-02 kernel: sd 0:0:0:0: SCSI error: return code = 0x08000002
Dec 22 11:57:14 xen-33-18-02 kernel: sda: Current: sense key: Medium Error
Dec 22 11:57:14 xen-33-18-02 kernel:     Additional sense: Unrecovered read error - auto reallocate failed

Then, the RAID1 re-sync would start all over again … only to fail a few hours later … and again.

Since there appear to be some bad spots on sda at the end of the disk, the solution seemed to be to shrink the partition a bit to avoid the bad spots.  

Our md RAID1 (md1) consists of 2 x 694GB partitions on sda3 and sdb3.  On top of md1 lives an LVM PV, VG, and lots of LVs.  The PV is not 100% allocated, so we have some wiggle room to shrink everything.

Here’s the procedure we followed:

  1. Shrink the LVM PV
    • “pvdisplay” showed the physical disk had 693.98 GB, so we just trimmed it down to an even 693GB.
    • use “pvresize” to shrink the PV
    • # pvresize  -v --setphysicalvolumesize 693G /dev/md1      Using physical volume(s) on command line
          Archiving volume group "datavg" metadata (seqno 42).
          /dev/md1: Pretending size is 1453326336 not 1455376384 sectors.
          Resizing physical volume /dev/md1 from 177658 to 177407 extents.
          Resizing volume "/dev/md1" to 1453325952 sectors.
          Updating physical volume "/dev/md1"
          Creating volume group backup "/etc/lvm/backup/datavg" (seqno 43).
          Physical volume "/dev/md1" changed
          1 physical volume(s) resized / 0 physical volume(s) not resized
  2. Shrink the md RAID1 device
    • If you do NOT do this and shrink the hard drive partitions beneath the md device … you loose your superblock!  Cool, eh?
    • Check the size of the existing md
    • # mdadm --verbose --detail /dev/md1
              Version : 00.90.03
        Creation Time : Mon Sep 29 23:41:30 2008
           Raid Level : raid1
           Array Size : 727688192 (693.98 GiB 745.15 GB)
        Used Dev Size : 727688192 (693.98 GiB 745.15 GB)
         Raid Devices : 2
        Total Devices : 1
      Preferred Minor : 1
          Persistence : Superblock is persistent
          Update Time : Mon Dec 22 22:57:31 2008
                State : clean, degraded
       Active Devices : 1
      Working Devices : 1
       Failed Devices : 0
        Spare Devices : 0
                 UUID : d584a880:d77024a2:9ae023b4:27ec0db5
               Events : 0.5501090
          Number   Major   Minor   RaidDevice State
             0       8        3        0      active sync   /dev/sda3
             1       0        0        1      removed
    • Hey, what a coincidence … the Array Size is the same as the PV was! 🙂  Now, we need to calculate the new size.  The new size of the PV is 1453325952 sectors, which is 726662976 blocks (even *you* can divide by 2!!).
    • Shrink the md:
    • # mdadm --verbose --grow /dev/md1 --size=726662976
      # mdadm --verbose --detail /dev/md1
           Array Size : 726662976 (693.00 GiB 744.10 GB)
    • Woot! Getting closer!
  3. Now, technically, we should shrink the partition, too.  Here’s where I ran into a bit of trouble (ok, I’m too lazy to do the math).  “fdisk” on linux doesn’t seem to want to let you specify a size in blocks or sectors, so you have to keep shrinking the ending cylinder number until you get in the range of the new block size.  I’ll leave this as an exercise for the reader, and feel free to post a comment with the actual procedure.  🙂

    After these steps were completed, we were able to perform the raid1 re-sync successfully.


      Things Are Not Always What They Seem

      April 8, 2008

      OK, it’s been way too long since my last post … much going on, but this post will be short and about something else!

      I was shopping at Sam’s Club the other day and in the freezer section, I came across a “seafood mixture” … ideal for Paella!  Yum!

      So, what’s the problem?  GF Paella should be pretty easy, right (yellow rice + chicken (in our version) + seafood + some GF safe flavoring sauces) … nothing illegal there.  If you are doing CF, make sure you don’t add any butter or any other flavoring agent that would be illegal.

      Here’s the problem – out of habit, I checked the ingredients list.  Yeah, the ingredients list on a bag of mixed seafood … how exciting could that be?  INCLUDES WHEAT!  What???  Yep, nearly made something deadly for my grandmother (86, frail, and a Celiac).  <sigh>

      Anyway, today’s lesson is READ THE INGREDIENTS ON EVERYTHING.  No matter how trivial it may seem.

      The Butterfly Pavilion

      January 22, 2008

      I found The Denver Digital Photography Meetup Group and went to the meetup at the Butterfly Pavilion and it was a nice trip. If there was any organized group stuff during or after (besides heading to the Rock Bottom Brewery after for lunch, but I don’t care for their food), I must have missed it. It would be nice to talk with other photographers and exchange tips and get some feedback on my techniques, etc. Maybe next meetup.

      I was experimenting with a circular polarizer, but the side effect is I can’t see very well through the lens, so a lot of my shots were out of focus. I’ve also found that the autofocus doesn’t always catch what I want it to, so I tried manual mode … still not so great. 🙂

      So, out of the 238 shots (that made it home), I have 15 below that were decent enough to look at again. More than I expected, actually, considering I was doing a *lot* of multiple shots and making minor tweaks between each shot (trying to get better with managing my exposure stops with the polarizer, etc).

      Hopefully some of these look OK … feedback is always welcome.

      Black and Red Butterfly 1 Black and Red Butterfly 2
      752×500 (1/60) f/5.6 f(35)=300mm (flash) 752×500 (1/60) f/5.6 f(35)=300mm (flash)
      Tiger butterfly Black and Red Butterfly
      752×500 (1/60) f/5.6 f(35)=300mm (flash) 752×500 (1/60) f/5.6 f(35)=300mm (flash)
      Brown Butterfly Two Butterflies
      752×500 (1/15) f/5.6 f(35)=300mm 752×500 (1/60) f/5.6 f(35)=142mm (flash)
      Hanging Fruit Hanging Flowers
      752×500 (1/125) f/5.6 f(35)=300mm 752×500 (1/125) f/5.6 f(35)=157mm
      Orange Flower Pink Flower
      752×500 (1/320) f/5.6 f(35)=300mm 752×500 (1/80) f/5.6 f(35)=300mm
      Horizontal Red Flower
      752×500 (1/80) f/5.6 f(35)=142mm
      Small Red Flowers Large Red Flower
      500×752 (1/60) f/5.6 f(35)=187mm 500×752 (1/60) f/5.6 f(35)=142mm (flash)
      Small Red Flowers 2 Big Flower
      500×752 (1/125) f/5.6 f(35)=187mm 500×752 (1/60) f/5.6 f(35)=142mm (flash)

      These are all re-sized down from JPG-Basic (and then re-sized again for the thumbnails), and are untouched from the camera. I’ve not post-processed anything yet. I do have RAW (Nikon) versions of each of these as well.

      Proof That Autism Can Be Defeated!

      January 16, 2008

      Ok, maybe not completely defeated, but the news we got today just reaffirms my belief that hard work, building a great support team, and aggressive therapy can make the difference. That said, did we do everything possible? I don’t think so, but then again, if you think you’ve done it all, maybe you are missing something else. To put it in a sports metaphor, go 110% and leave it all on the field … but know that there is always something more you can challenge yourself to do better next round.

      We followed up with J’s developmental evaluation (from early December 2007) with the therapists today. The ladies at Developmental FX are just awesome, and they really care about what they do. They did his developmental evaluation back in August 2006, where they couldn’t even do age appropriate testing (he was physically 4 years old) and pegged his development at 22 to 29 months of age (mental, speech, physical, etc). Keep in mind that those numbers were similar to the speech eval that was done at University of New Mexico Hospital in March 2005, and he had made _progress_ in the 3 months with us prior to his 2006 eval. Scary, eh?

      Well, we have the “final” report, although we would like to work in inputs from J’s biological parents (where he is now), but they still haven’t submitted their forms or made an appointment to meet with Developmental FX yet. In a nutshell, Developmental FX was very happy (and maybe a little surprised) at how much J has grown. They now estimate him to be around the 52 month (4 years, 4 months) developmental stage. At the time of the eval, that puts him only about a year behind!

      Through hard work, a great support team (both professional and personal), LOTS of positive re-enforcement, education (for us and J), a (very) little medication, the GF/CF diet, and just plain challenging him to develop and grow, we managed to do over 2 years of developmental growth in about 15 months. Something like 176% growth and developmental rate? I’ll take that! I think his rapid ascent makes it pretty obvious that _he_ was not the problem, but rather it was his old environment and lack of a good support team. Reading his old reports, he had some pretty good early intervention support, but somehow, it just didn’t stick.

      Could *we* have done better? Probably! We did have quite a few hiccups along the way, and other external events that made it difficult for him to focus and increased his overall anxiety. As we got closer to what became the transition back to his parents, his anxiety level was visibly high — huge increases in his door fixation, thumb sucking back in full force, even some behaviors that were completely gone came back (like the head shaking) and more frequent hand flapping. He even lost some of his language and looked “lost” more often. It’s really sad to see a child fall apart right in front of your eyes.

      Naturally, this report is bittersweet. We are very proud of the work we have done, the progress we’ve made together, and the foundation we’ve helped him build. At the same time, we can only hope that he continues to get the same support and dedication that we’ve given him over the last 20 months. Now that we are no longer directly in the picture, we have to rely on information that trickles back from his support team as well as others who we have crossed paths with and have taken an interest in him (he’s a really likable kid). As much as we’d like to continue to be involved in his day-to-day, the emotional trauma and mental distress that bouncing between the households was causing would more than counteract any benefit we could add — and that’s something that we think and grieve about every day.

      We still believe that J has the ability to be mainstreamed, but he still has a long way to go. We know that his parents are busy and work and all of that, but we hope they can find a way to keep giving him the attention he will need to succeed and have a quality life (meanwhile, not ignoring his little sister, A). If they don’t help him succeed, it will be everyone’s loss — he’s a bright kid and has a lot to offer.

      The take away from this? If you have a child with autism, DO NOT GIVE UP! Anything can happen, and if you WORK HARD and be open to new ideas, talk to your community and support professionals, build a network (personal and professional), you have a great chance of giving your child the ultimate gift — a quality, and independent life. There are no “do-overs” or “I’ll do it tomorrow”, only do it now. To quote our Jedi master Yoda, “there is no try, only do”.

      Seriously, though, you are your child’s only true advocate. You may build a fantastic support network, but they are only there to support you help you and your child. The effort you put in will bring you rewards that money cannot buy.

      Whether it is or isn’t someone’s fault that your child has autism, or that $deity is to blame or whatever is all TOTALLY IRRELEVANT. The issue at hand is WHAT ARE YOU GOING TO DO NOW! And that can only be TAKE ACTION. Time is running out, so get to it.

      We did it in a short time, with no forewarning, no training, heck, not even a clear timeline or future. If we can, so can you.

      Love your children and work your a$$ off now … you can always sleep next year. 🙂

      add to del.icio.usDigg itStumble It!Add to Blinkslistadd to furladd to ma.gnoliaadd to simpyseed the vineTailRank

      More Gluten Free / Casein Free (GF/CF) Links Added!

      January 14, 2008

      Thanks to a friend sending me a link to Hormel’s site, I finally got motivated to add a bunch more of my GF/CF links to my blogroll.

      I hope some of these help you find more resources out there, but BE CAREFUL — manufacturers can change ingredients, manufacturing facilities, and processes at any time. Also, the web is never the final authority — CALL or WRITE the manufacturer or distributor and get the bottom line (in writing, if possible) from them.

      Also, keep hounding your local grocery store to carry these products. Thanks to pressure from “the people”, Deby’s Gluten Free (soon to be renamed as “8 Free Foods”) will have a handful of products in our local King Sooper’s stores.

      King Sooper’s also carries Boars Head meats, most of which are GF/CF (according to the rep I talked to a few months ago). Unfortunately, the KS near us doesn’t stock Boars Head yet, but we’re pestering away.

      Also, in case I hadn’t mentioned it yet, Smuckers Jellies and Jams are supposedly all GF/CF (again, spoke to rep back in Sept 2007, so please re-confirm), as are Welsh(?) juices.

      Sadly, our involvement with the GF/CF world will be largely from the outside looking in now. J and A have been rather abruptly transitioned back to their biological parents (for various reasons, not the least of which was the emotional trauma and mental/developmental damage the kids were experiencing from the bouncing between very different households). The generally accepted presumption was the kids were going to transition back to the parents at some point, and due to a series of events, that transition had to be a little more quick and final than expected.

      To say we miss them dearly would be a gross understatement, and we’ll be cleaning and re-organizing the house for many weeks to come. They spent nearly 20 months with us, so they have left a very significant footprint in our home and hearts.

      This saga is not over yet, as we still have court assigned custody issue to work out (among other things), and I think the training and life experience we’ve had through this will keep us active in the Autism advocacy arena. But for right now, we need to spend some time healing before we can engage again.

      add to del.icio.usDigg itStumble It!Add to Blinkslistadd to furladd to ma.gnoliaadd to simpyseed the vineTailRank

      What Makes Someone “Interesting”?

      January 12, 2008

      That was the topic at tonight’s philosophical discussion group (not the Thursday Socrates Cafe meetup, but another organized by John).

      The discussion wandered around the concept of “what *is* ‘interesting’?” What makes something interesting, and what makes someone consider something interesting.

      I think most of us agreed that determining if something is “interesting” lies with the beholder. Just as everyone has different views and opinions on things, that naturally translates to a level of interest.

      One person suggested that he finds watching baseball uninteresting and wouldn’t do it by choice. He likes to *play* baseball, but from the observer point of view, it does nothing for him. He also mentioned that he doesn’t care for football, either, unless a team that he may have mild interest in is doing well and perhaps are in the playoffs.

      This led me to suggest that perhaps “interest” has at least two components: intellectual and emotional. Also, I offer that the combination of the two must aggregate such that the level of “interest” is sufficient to overcome the inherent inertia of inaction.

      This becomes apparent to me when you consider situations like the watching football only if the team is winning or is in a game of significance (i.e. a playoff game, the Superbowl, etc). Another example given was noticing a skydiver … it may be of mild interest to the observer, however, once the parachute opens, that interest may wain. But, what if the skydiver is a friend, the emotional component is likely increased sufficiently such that the observer’s interest level remains high enough to continue monitoring the activity.

      However, I do not believe having sufficient “interest” is the only reason to pursue something or participate in an activity. For example, some work tasks aren’t exactly something of great interest, or perhaps a school assignment, or just cleaning the kitty litter box.

      So, what makes someone “interesting”? Assuming that “interest” is from the observer’s point of view, a person or thing cannot be “interesting” in of itself, but rather it must satisfy a collection of intellectual and emotional stimuli of the observer in order to qualify. As these stimuli can expand and contract as the observer learns more about the subject, or as a result of time decay (or increased curiosity). “Loosing interest” could be a function of time decay or learning enough about the subject to realize that the emotional or intellectual stimulation is no longer prevalent enough to continue actively participating.

      We also explored the observation that the typical human interest has an odd attraction to negative or dark attributes. For example, casual observation of the major media outlets clearly shows they have a disproportionate amount of negative news to positive news. Some postulated reasons for the gravitation to such dark actions like murder, theft, or revenge include:

      • Determining the “why” a person performed such actions (i.e. understanding the human condition)
      • An inherent desire or temptation within most people to perform a similar action. Obviously most people are not murderers, but most people *have had* an occasion where they would have the desire or temptation to act out of anger or frustration — just not enough to overcome the barrier of common sense or morality and take action. Perhaps also described as an unconscious jealousy of the perpetrator’s ability to act?

      In the case of the person who proposed the topic, if he wanted to present himself as more “interesting” to other persons, he should probably find out what those persons found interesting and look inward to see if he had any overlap. If not, all is not lost, as he might just realize that something he hadn’t considered might actually be interesting to him after all, and that in itself could provide a basis for a conversation.

      add to del.icio.usDigg itStumble It!Add to Blinkslistadd to furladd to ma.gnoliaadd to simpyseed the vineTailRank

      Remove Tourists From Your Pictures … Legally!

      January 7, 2008

      Yep, it’s no longer necessary to pack your .45 to get rid of those pesky tourists when trying to take pictures while sightseeing. “Tourist Remover” by is a free tool that claims to allow you to take multiple shots from the same perspective (sounds easier with a tripod), and the “stuff” that is not-common across at least 2 shots will be removed from the composite image.

      This really interests me, so I’m going to sign up — it’s a web access only tool. I shoot a lot of multiple images, so I’m hoping I can find a couple of images of the same framing that I can get some shots uploaded and test. It looks like you get 100MB free storage and the web application has all of the expected refinement and effect/enhancement tools, too.

      In the meantime, they have 4 sample images to test the function with. You have to login to the the site and select the “Demo: Tourist Remover” folder, highlight all of the images, and hit the “Remove Tourists” function button (on the far right … on my window, I had to shift to the right to get to the button). Don’t forget to enable the site for a pop-up window! This brings you to a new page where you are recommended to order the images from most clear to least clear of your desired subject matter (left to right). Finally, hit the “Create” button to get your job submitted — my job was #333 in line.

      It ran pretty quick and my notification e-mail was in my inbox right away.

      The resulting image is placed back in your webspace on in a folder called “TouristRemover” (who wudda thunk?). Checking out the sample image, and it did indeed remove the people in it, but neither the source or final image is very sharp upon zoomed inspection.

      Final answer is that I really need to upload some higher quality images to see what it will do.

      add to del.icio.usDigg itStumble It!Add to Blinkslistadd to furladd to ma.gnoliaadd to simpyseed the vineTailRank