LinHES Forums
http://forums.linhes.org/

XFS Speed Hints-Measure % frag and defrag your XFS partition
http://forums.linhes.org/viewtopic.php?f=3&t=18595
Page 2 of 3

Author:  akulcsar [ Tue Sep 30, 2008 12:43 am ]
Post subject: 

Just throwing this one out there as a brainstorm:

Would it be possible to create a run level - like, say, 7 - have it run the defragmentation in an init script, and then when it is done to return to run level 5? Something like this:

  1. cron (intelligently, somehow) schedules a job to drop to init level X.
  2. The init scripts in level X mimic level 1.
  3. The last init script in level X is a custom script that runs the xfs defrag routine for a predetermined amount of time, and then switches back to run level 5 when done.


Does that sound...sane? :-)

Author:  opel70 [ Tue Oct 14, 2008 8:59 am ]
Post subject: 

graysky wrote:
Did you add the allocsize=512m to your /etc/fstab? It made a difference for my machine capturing SDTV.

Any reason not to use the max of allocsize-1024m? Even my small 1/2 hour shows are almost about 900MB, while the vast majority of what I record are multiple GB. I have been running with this for the past few months with no noticeable adverse affects, but am I missing some problem with this size?

Author:  graysky [ Tue Oct 14, 2008 12:57 pm ]
Post subject: 

opel70 wrote:
graysky wrote:
Did you add the allocsize=512m to your /etc/fstab? It made a difference for my machine capturing SDTV.

Any reason not to use the max of allocsize-1024m? Even my small 1/2 hour shows are almost about 900MB, while the vast majority of what I record are multiple GB. I have been running with this for the past few months with no noticeable adverse affects, but am I missing some problem with this size?


Dunno man, capping 4-6 gig SDTV files here. 512 does a great job for me. I just used it based on the suggestion on the wiki I cited.

Author:  RacerX [ Wed Oct 15, 2008 10:08 am ]
Post subject: 

Thanks Graysky for the tip. After I scaled out my setup with multiple drives and storage groups I didn't really realize how fragramented the drives really were. On my old box it was like 52% and 63% and the new box was 97%. I ran the tool for 1.5 hours and it went down to something like 10% or less on the old box and 1% on the new box.

NICE WORK!!! :wink:

Author:  graysky [ Wed Oct 15, 2008 3:13 pm ]
Post subject: 

Cool man, glad to hear people are getting mileage out of this thread.

Author:  tatuirovka [ Sat Oct 25, 2008 7:57 am ]
Post subject: 

Wondering if someone can help me with a little problem I'm having with the output of xfs_db after running xfs_fsr... for some reason xfs_db gives the same numbers before and after... if I reboot then they update.

Code:
# xfs_db -c frag -r /dev/sdb1
# actual 170, ideal 150, fragmentation factor 11.76%
# xfs_fsr -v -t 60
# xfs_db -c frag -r /dev/sdb1
# actual 170, ideal 150, fragmentation factor 11.76%
# reboot
# xfs_db -c frag -r /dev/sdb1
# actual 156, ideal 149, fragmentation factor 4.49%

Author:  graysky [ Sat Oct 25, 2008 8:15 am ]
Post subject: 

I've seen this before and can't explain it. You can reset the stats by umounting then remounting the filesystem, i.e.

Code:
# umount /myth
# mount /myth

Author:  marc.aronson [ Sat Oct 25, 2008 11:22 pm ]
Post subject: 

opel70 wrote:
graysky wrote:
Did you add the allocsize=512m to your /etc/fstab? It made a difference for my machine capturing SDTV.

Any reason not to use the max of allocsize-1024m? Even my small 1/2 hour shows are almost about 900MB, while the vast majority of what I record are multiple GB. I have been running with this for the past few months with no noticeable adverse affects, but am I missing some problem with this size?

Each recording also creates a ".png" file in /myth/tv -- this means that each of those files will use 1GB of disk. Not a show-stopper -- just something you should be aware of.

I have my machine defrag for 10 minutes once a day, just before it shuts down. My fragmentation has remained very low.

Marc

Author:  adoute [ Fri Oct 31, 2008 6:39 am ]
Post subject: 

I'm running R5F27, and recently replaced my media drives with larger ones. During the replacement, I also switched from ext3 to xfs.

I thought I'd see what my fragmentation levels were, and defrag if necessary. Both media drives are showing EXTREMELY high frag levels (98.80% and 92.39%), so I thought I'd defrag them while the box was going to be idle for several hours. Unfortunately, I do not have xfs_fsr on my system... The only xfs_* files available are:

Code:
xfs_admin     xfs_copy      xfs_growfs    xfs_logprint  xfs_quota
xfs_bmap      xfs_db        xfs_info      xfs_mkfile    xfs_repair
xfs_check     xfs_freeze    xfs_io        xfs_ncheck    xfs_rtcp


Can anyone offer any insight as to why I would have xfs_db and not xfs_fsr?

And more importantly, is there a way that I can get xfs_fsr so I can defrag my drives without making large changes to the core OS?

Thank you in advance for any advice you can offer.

Author:  marc.aronson [ Fri Oct 31, 2008 8:04 am ]
Post subject: 

adoute wrote:
And more importantly, is there a way that I can get xfs_fsr so I can defrag my drives without making large changes to the core OS?

Thank you in advance for any advice you can offer.


Look here

Marc

Author:  adoute [ Fri Oct 31, 2008 8:53 am ]
Post subject: 

Thank you for such a quick response. The required software installed perfectly. Now I just need to find a nice block of time to start defragging...

FWIW, I did try searching - I can't believe I missed the post that you linked to. Thanks again.

Author:  Grant_Edwards [ Fri Oct 31, 2008 10:41 am ]
Post subject: 

My XFS /myth partition was 99% fragmented. I never noticed any
performance issues, but after all the dire warnings in this
thread about fragmentation, I degragmented it anyway (which
took 15-20 hours). I also added the option "allocsize=256m" to
fstab and re-mounted the partition.

After doing the defrag and setting the allocsize, playback now
pauses and stutters when a recording is being deleted. It's
not as bad as my old machine that used ext3.

AFAICT, defragging and using a larger allocsize has caused this
stuttering as files are deleted. I don't suppose there's a
re-fragement utility I can use to get back to the previous
state? I guess if I decrease the allocsize it will eventually
return to "normal", right?

--
Grant

Author:  graysky [ Fri Oct 31, 2008 1:20 pm ]
Post subject: 

@grant - I never heard of that before.... I have been using my XFS system like this (with the 512meg thing in the fstab) and it's been just fine, in fact, it's noticibly faster when deleting 4-5 gig files (usually takes 1-2 sec). I wonder if you have an unrelated problem? Have you tried checking the consistency of the partition for troubles? See this thread for a step-by-step walk through.

Failing that, and if you suspect it's the allocsize statement, do this: take out the allocsize statement in the fstab, umount, remount, THEN try running the defrag util.

Let us know in either case :)

Author:  Grant_Edwards [ Sat Nov 01, 2008 2:56 pm ]
Post subject: 

graysky wrote:
@grant - I never heard of that before.... I have been using my
XFS system like this (with the 512meg thing in the fstab) and
it's been just fine, in fact, it's noticibly faster when
deleting 4-5 gig files (usually takes 1-2 sec). I wonder if
you have an unrelated problem?

It's possible it was just a coincidence -- I haven't really
done anything to troubleshoot it since it only stutters for a
second or two and then returns to normal. My ext3 based system
(a different box) used to stutter for 30+ seconds while a file
was being deleted (occasionally the player would even exit
spontaneously).
Quote:
Have you tried checking the consistency of the partition for
troubles?

I ran the XFS file system checker both before and after I
defragged the partition, but I haven't run in since then. I'll
run it again to make sure nothing's gone bad.
Quote:
Failing that, and if you suspect it's the allocsize statement,
do this: take out the allocsize statement in the fstab, umount,
remount, THEN try running the defrag util.

IIRC, I didn't add the allocsize statement until after I had
defragged the partition. I assumed the allocsize statement
would only affect new files that were being created.

Author:  knappster [ Wed Nov 12, 2008 12:15 am ]
Post subject: 

I've tried add a cron job for this and I have been unable to get xfs_fsr to output to a log file. xfs_db does no problem. The weird thing is that if I run the command as a cronjob it runs and the output ends up in /var/log/messages. If I copy and paste the same command into a shell, it runs and puts the output into the proper log file. Any thoughts on what I might be doing wrong?

Code:
02 00 * * 3 (/usr/local/bin/idle.sh && /usr/sbin/xfs_fsr -v -t 600) > /var/log/defrag.log 2>&1
as a cron job (root), results in:
Quote:
Checking what MythTV is doing now or plans within 20 minutes...

mythshutdown returned 0
schemalock 0
running jobs 0
inuse programs 0
potential recordings 0
planned recordings 0

System is idle
in /var/log/defrag.log
and
Quote:
Nov 12 00:02:02 mythtv fsr[9145]: Found 1 mounted, writable, XFS filesystems
Nov 12 00:02:02 mythtv fsr[9145]: xfs_fsr -m /etc/mtab -t 600 -f /var/tmp/.fsrlast_xfs ...
Nov 12 00:02:02 mythtv fsr[9145]: START: pass=0 ino=0 /dev/sda3 /myth
Nov 12 00:02:02 mythtv fsr[9146]: /myth start inode=0
Nov 12 00:02:02 mythtv fsr[9146]: ino=1124970593
Nov 12 00:02:02 mythtv fsr[9146]: No improvement will be made (skipping): ino=1124970593
Nov 12 00:02:02 mythtv fsr[9146]: ino=1124970594
Nov 12 00:02:02 mythtv fsr[9146]: No improvement will be made (skipping): ino=1124970594
Nov 12 00:02:02 mythtv fsr[9146]: ino=1124970596
Nov 12 00:02:02 mythtv fsr[9146]: No improvement will be made (skipping): ino=1124970596
Nov 12 00:02:02 mythtv fsr[9147]: /myth start inode=0
Nov 12 00:02:02 mythtv fsr[9147]: ino=1124970593
Nov 12 00:02:02 mythtv fsr[9147]: No improvement will be made (skipping): ino=1124970593
Nov 12 00:02:02 mythtv fsr[9147]: ino=1124970594
Nov 12 00:02:02 mythtv fsr[9147]: No improvement will be made (skipping): ino=1124970594
Nov 12 00:02:02 mythtv fsr[9147]: ino=1124970596
Nov 12 00:02:02 mythtv fsr[9147]: No improvement will be made (skipping): ino=1124970596
Nov 12 00:02:02 mythtv fsr[9148]: /myth start inode=0
Nov 12 00:02:02 mythtv fsr[9148]: ino=1124970593
Nov 12 00:02:02 mythtv fsr[9148]: No improvement will be made (skipping): ino=1124970593
Nov 12 00:02:02 mythtv fsr[9148]: ino=1124970594
Nov 12 00:02:02 mythtv fsr[9148]: No improvement will be made (skipping): ino=1124970594
Nov 12 00:02:02 mythtv fsr[9148]: ino=1124970596
Nov 12 00:02:02 mythtv fsr[9148]: No improvement will be made (skipping): ino=1124970596
Nov 12 00:02:02 mythtv fsr[9149]: /myth start inode=0
Nov 12 00:02:02 mythtv fsr[9149]: ino=1124970593
Nov 12 00:02:02 mythtv fsr[9149]: No improvement will be made (skipping): ino=1124970593
Nov 12 00:02:02 mythtv fsr[9149]: ino=1124970594
Nov 12 00:02:02 mythtv fsr[9149]: No improvement will be made (skipping): ino=1124970594
Nov 12 00:02:02 mythtv fsr[9149]: ino=1124970596
Nov 12 00:02:02 mythtv fsr[9149]: No improvement will be made (skipping): ino=1124970596
Nov 12 00:02:02 mythtv fsr[9150]: /myth start inode=0
Nov 12 00:02:02 mythtv fsr[9150]: ino=1124970593
Nov 12 00:02:02 mythtv fsr[9150]: No improvement will be made (skipping): ino=1124970593
Nov 12 00:02:02 mythtv fsr[9150]: ino=1124970594
Nov 12 00:02:02 mythtv fsr[9150]: No improvement will be made (skipping): ino=1124970594
Nov 12 00:02:02 mythtv fsr[9150]: ino=1124970596
Nov 12 00:02:02 mythtv fsr[9150]: No improvement will be made (skipping): ino=1124970596
Nov 12 00:02:02 mythtv fsr[9151]: /myth start inode=0
Nov 12 00:02:02 mythtv fsr[9151]: ino=1124970593
Nov 12 00:02:02 mythtv fsr[9151]: No improvement will be made (skipping): ino=1124970593
Nov 12 00:02:02 mythtv fsr[9151]: ino=1124970594
Nov 12 00:02:02 mythtv fsr[9151]: No improvement will be made (skipping): ino=1124970594
Nov 12 00:02:02 mythtv fsr[9151]: ino=1124970596
Nov 12 00:02:02 mythtv fsr[9151]: No improvement will be made (skipping): ino=1124970596
Nov 12 00:02:02 mythtv fsr[9152]: /myth start inode=0
Nov 12 00:02:02 mythtv fsr[9152]: ino=1124970593
Nov 12 00:02:02 mythtv fsr[9152]: No improvement will be made (skipping): ino=1124970593
Nov 12 00:02:02 mythtv fsr[9152]: ino=1124970594
Nov 12 00:02:02 mythtv fsr[9152]: No improvement will be made (skipping): ino=1124970594
Nov 12 00:02:02 mythtv fsr[9152]: ino=1124970596
Nov 12 00:02:02 mythtv fsr[9152]: No improvement will be made (skipping): ino=1124970596
Nov 12 00:02:02 mythtv fsr[9153]: /myth start inode=0
Nov 12 00:02:02 mythtv fsr[9153]: ino=1124970593
Nov 12 00:02:02 mythtv fsr[9153]: No improvement will be made (skipping): ino=1124970593
Nov 12 00:02:02 mythtv fsr[9153]: ino=1124970594
Nov 12 00:02:02 mythtv fsr[9153]: No improvement will be made (skipping): ino=1124970594
Nov 12 00:02:02 mythtv fsr[9153]: ino=1124970596
Nov 12 00:02:02 mythtv fsr[9153]: No improvement will be made (skipping): ino=1124970596
Nov 12 00:02:02 mythtv fsr[9154]: /myth start inode=0
Nov 12 00:02:02 mythtv fsr[9154]: ino=1124970593
Nov 12 00:02:03 mythtv fsr[9154]: No improvement will be made (skipping): ino=1124970593
Nov 12 00:02:03 mythtv fsr[9154]: ino=1124970594
Nov 12 00:02:03 mythtv fsr[9154]: No improvement will be made (skipping): ino=1124970594
Nov 12 00:02:03 mythtv fsr[9154]: ino=1124970596
Nov 12 00:02:03 mythtv fsr[9154]: No improvement will be made (skipping): ino=1124970596
Nov 12 00:02:03 mythtv fsr[9155]: /myth start inode=0
Nov 12 00:02:03 mythtv fsr[9155]: ino=1124970593
Nov 12 00:02:03 mythtv fsr[9155]: No improvement will be made (skipping): ino=1124970593
Nov 12 00:02:03 mythtv fsr[9155]: ino=1124970594
Nov 12 00:02:03 mythtv fsr[9155]: No improvement will be made (skipping): ino=1124970594
Nov 12 00:02:03 mythtv fsr[9155]: ino=1124970596
Nov 12 00:02:03 mythtv fsr[9155]: No improvement will be made (skipping): ino=1124970596
Nov 12 00:02:03 mythtv fsr[9145]: Completed all 10 passes
in /var/log/messages

If I run
Code:
(/usr/local/bin/idle.sh && /usr/sbin/xfs_fsr -v -t 600) > /var/log/defrag.log 2>&1
in a shell as root, I get:
Quote:
Checking what MythTV is doing now or plans within 20 minutes...

mythshutdown returned 0
schemalock 0
running jobs 0
inuse programs 0
potential recordings 0
planned recordings 0

System is idle
Found 1 mounted, writable, XFS filesystems
xfs_fsr -m /etc/mtab -t 600 -f /var/tmp/.fsrlast_xfs ...
START: pass=0 ino=0 /dev/sda3 /myth
/myth start inode=0
ino=1124970593
No improvement will be made (skipping): ino=1124970593
ino=1124970594
No improvement will be made (skipping): ino=1124970594
ino=1124970596
No improvement will be made (skipping): ino=1124970596
/myth start inode=0
ino=1124970593
No improvement will be made (skipping): ino=1124970593
ino=1124970594
No improvement will be made (skipping): ino=1124970594
ino=1124970596
No improvement will be made (skipping): ino=1124970596
/myth start inode=0
ino=1124970593
No improvement will be made (skipping): ino=1124970593
ino=1124970594
No improvement will be made (skipping): ino=1124970594
ino=1124970596
No improvement will be made (skipping): ino=1124970596
/myth start inode=0
ino=1124970593
No improvement will be made (skipping): ino=1124970593
ino=1124970594
No improvement will be made (skipping): ino=1124970594
ino=1124970596
No improvement will be made (skipping): ino=1124970596
/myth start inode=0
ino=1124970593
No improvement will be made (skipping): ino=1124970593
ino=1124970594
No improvement will be made (skipping): ino=1124970594
ino=1124970596
No improvement will be made (skipping): ino=1124970596
/myth start inode=0
ino=1124970593
No improvement will be made (skipping): ino=1124970593
ino=1124970594
No improvement will be made (skipping): ino=1124970594
ino=1124970596
No improvement will be made (skipping): ino=1124970596
/myth start inode=0
ino=1124970593
No improvement will be made (skipping): ino=1124970593
ino=1124970594
No improvement will be made (skipping): ino=1124970594
ino=1124970596
No improvement will be made (skipping): ino=1124970596
/myth start inode=0
ino=1124970593
No improvement will be made (skipping): ino=1124970593
ino=1124970594
No improvement will be made (skipping): ino=1124970594
ino=1124970596
No improvement will be made (skipping): ino=1124970596
/myth start inode=0
ino=1124970593
No improvement will be made (skipping): ino=1124970593
ino=1124970594
No improvement will be made (skipping): ino=1124970594
ino=1124970596
No improvement will be made (skipping): ino=1124970596
/myth start inode=0
ino=1124970593
No improvement will be made (skipping): ino=1124970593
ino=1124970594
No improvement will be made (skipping): ino=1124970594
ino=1124970596
No improvement will be made (skipping): ino=1124970596
Completed all 10 passes
with nothing new in /var/log/messages

What gives? Shouldn't I get the same log results whether a command is run in a shell or a cron job? If it makes any difference I am running R5F27.

Page 2 of 3 All times are UTC - 6 hours
Powered by phpBB® Forum Software © phpBB Group
http://www.phpbb.com/